BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards

# BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards
Source: [https://arxiv.org/html/2606.28707](https://arxiv.org/html/2606.28707)
Yupeng Chang1Yuan Wu1111Corresponding authorYi Chang1,2,3 1School of Artificial Intelligence, Jilin University 2Engineering Research Center of Knowledge\-Driven Human\-Machine Intelligence, MOE, China 3International Center of Future Science, Jilin University changyp23@mails\.jlu\.edu\.cn, \{yuanwu, yichang\}@jlu\.edu\.cn

###### Abstract

Critic\-free reinforcement learning with verifiable rewards \(RLVR\), exemplified by Group Relative Policy Optimization \(GRPO\), avoids training a value function \(critic\) and reduces memory and compute overhead relative to critic\-based PPO pipelines for aligning large language models\. However, GRPO\-style advantage estimation depends on prompt\-local \(within\-prompt\-group\) reward statistics and can be unstable\. In particular, when all rollouts in a prompt group receive identical rewards, the within\-group reward variance becomes zero, and group normalization yields*zero*advantages for that group, impeding learning in cold\-start regimes with binary verifiers\. We introduceBV\-Blend, a critic\-free framework that stabilizes advantage estimation by combining prompt\-local on\-policy statistics with semantic\-cluster\-conditioned historical moments\. BV\-Blend maintains EMA\-tracked reward moments for each cluster, derives a confidence weight from a standard error of the mean \(SEM\) proxy, and uses this weight to blend historical and prompt\-local baseline and variance statistics into a standardized advantage for PPO\-style clipped updates\. Experiments on verifiable reasoning benchmarks show that BV\-Blend improves training stability and performance, and remains robust in regimes where group\-normalized methods may stall\.

BV\-Blend: Uncertainty\-Weighted Historical Baselines for Stable Critic\-Free RL with Verifiable Rewards

Yupeng Chang1Yuan Wu1111Corresponding authorYi Chang1,2,31School of Artificial Intelligence, Jilin University2Engineering Research Center of Knowledge\-Driven Human\-Machine Intelligence, MOE, China3International Center of Future Science, Jilin Universitychangyp23@mails\.jlu\.edu\.cn, \{yuanwu, yichang\}@jlu\.edu\.cn

## 1Introduction

Reinforcement Learning with Verifiable Rewards \(RLVR\) has recently become a practical paradigm for aligning Large Language Models \(LLMs\) in domains with*objective*correctness signals, such as mathematics and code, where outputs can be automatically verified \(e\.g\., exact match against a reference answer, formal proof checking, or passing unit tests\)\(Yuet al\.,[2025b](https://arxiv.org/html/2606.28707#bib.bib110); Yanet al\.,[2025](https://arxiv.org/html/2606.28707#bib.bib67); Guiet al\.,[2024](https://arxiv.org/html/2606.28707#bib.bib111)\)\. Compared with process\-level supervision such as Process Reward Models \(PRMs\)\(Lightmanet al\.,[2023](https://arxiv.org/html/2606.28707#bib.bib26)\), RLVR optimizes outcome\-based rewards and does not require scoring intermediate reasoning steps\. When verifiers are reliable, collecting multiple rollouts per prompt naturally improves exploration and increases the chance of observing correct solutions, which has underpinned several recent reasoning\-focused RL systems\(Guoet al\.,[2025](https://arxiv.org/html/2606.28707#bib.bib71)\)\. Beyond structured domains, RLVR has also been extended to more diverse settings when reference signals or robust verifiers are available\(Suet al\.,[2025](https://arxiv.org/html/2606.28707#bib.bib28)\)\.

However, directly applying the de facto RLHF recipe—PPO with a learned critic\(Ziegleret al\.,[2019](https://arxiv.org/html/2606.28707#bib.bib57); Ouyanget al\.,[2022](https://arxiv.org/html/2606.28707#bib.bib30)\)—to RLVR exposes a practical tension\. With sparse, trajectory\-level verifiable rewards, learning an accurate value function can be challenging, while the additional critic introduces non\-trivial memory and compute overhead, as well as additional training complexity, at LLM scale\(Ouyanget al\.,[2022](https://arxiv.org/html/2606.28707#bib.bib30)\)\. These considerations have accelerated interest in critic\-free alignment\. Preference\-optimization methods such as DPO\(Rafailovet al\.,[2024](https://arxiv.org/html/2606.28707#bib.bib33)\)avoid explicit reward modeling and online RL, but are primarily studied under preference or utility supervision rather than settings where informative, programmatic outcome rewards are directly available\. We therefore focus on*direct\-reward, critic\-free*policy optimization, where GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2606.28707#bib.bib41)\)is a prominent baseline that derives advantages through prompt\-local reward normalization within each prompt group\.

Despite its simplicity, prompt\-local normalization can be unstable\. The resulting advantage estimator may depend strongly on transient within\-group statistics, leading to high variance and unstable learning dynamics, which has motivated a growing body of analyses and fixes\(Liang,[2025](https://arxiv.org/html/2606.28707#bib.bib122); Chenet al\.,[2025](https://arxiv.org/html/2606.28707#bib.bib123); Mroueh,[2025](https://arxiv.org/html/2606.28707#bib.bib124); Yuet al\.,[2025a](https://arxiv.org/html/2606.28707#bib.bib44)\)\. Moreover, RLVR commonly encounters a cold\-start regime in which the current policy produces predominantly incorrect solutions; under binary verifiers, many prompt groups become effectively deterministic \(e\.g\., all failures\), yielding \(near\-\)zero within\-group reward variance\. In this*zero\-variance*regime, standard within\-group normalization produces \(near\-\)zero advantages, removing the learning signal for those prompt groups\(Leet al\.,[2025](https://arxiv.org/html/2606.28707#bib.bib27)\)\.

To address this instability, we proposeBV\-Blend, a critic\-free framework that stabilizes advantage estimation by combining \(i\) an on\-policy prompt\-local signal with \(ii\) low\-variance historical moments aggregated over semantically similar prompts\. BV\-Blend computes an uncertainty\-aware confidence weight from a standard\-error\-of\-the\-mean \(SEM\) proxy of the historical statistics, downweighting unreliable historical information while relying more on it when the historical estimate is better supported\. As a result, BV\-Blend mitigates advantage collapse on zero\-variance prompt groups without training a critic and remains compatible with PPO\-style optimization\.

Our contributions are:

- •We analyze the*zero\-variance*failure mode of prompt\-local \(group\-normalized\) advantage estimation in direct\-reward, critic\-free RL, where the normalized learning signal vanishes when all rollouts in a prompt group receive identical rewards\.
- •We introduceBV\-Blend, an uncertainty\-aware historical blending mechanism that stabilizes prompt\-local advantage estimation by leveraging semantic\-cluster\-conditioned historical moments, without training a critic\.
- •We empirically demonstrate that BV\-Blend improves training stability and performance on verifiable reasoning benchmarks, and remains robust in regimes where standard group\-normalized methods may stall\.

## 2Related Work

##### Critic\-based and preference\-based alignment\.

Policy\-gradient methods typically reduce variance with a learned value\-function baseline; GAE\(Schulmanet al\.,[2015](https://arxiv.org/html/2606.28707#bib.bib114)\)\(commonly paired with PPO\(Schulmanet al\.,[2017](https://arxiv.org/html/2606.28707#bib.bib95)\)\) yields lower\-variance advantages than Monte Carlo estimators such as REINFORCE\(Williams,[1992](https://arxiv.org/html/2606.28707#bib.bib120)\)\. In RLHF\-style LLM alignment, PPO pipelines often train an auxiliary value head alongside the policy during RL fine\-tuning\(Ouyanget al\.,[2022](https://arxiv.org/html/2606.28707#bib.bib30)\)\. A separate line of work avoids online RL and critic training by directly optimizing objectives from preference or utility supervision, e\.g\., DPO\(Rafailovet al\.,[2024](https://arxiv.org/html/2606.28707#bib.bib33)\), KTO\(Ethayarajhet al\.,[2024](https://arxiv.org/html/2606.28707#bib.bib36)\), and SimPO\(Menget al\.,[2024](https://arxiv.org/html/2606.28707#bib.bib40)\)\. These methods are primarily studied under preference or utility supervision \(or implicit rewards\), rather than explicit outcome\-based verifiable rewards\.

##### Direct\-reward, critic\-free policy optimization\.

Our work builds on critic\-free policy optimization with verifiable rewards, exemplified by Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.,[2024](https://arxiv.org/html/2606.28707#bib.bib41)\)\. GRPO avoids a learned critic by normalizing rewards within each prompt group, but the learning signal can be sensitive to transient within\-group statistics and may collapse when within\-group reward variance is near zero\. Recent work has explored related issues and remedies, including correcting optimization bias in GRPO\-style objectives \(Dr\. GRPO\)\(Liuet al\.,[2025b](https://arxiv.org/html/2606.28707#bib.bib45)\), temporally smoothing baselines via lightweight Bayesian or Kalman\-style updates \(KRPO\)\(Wanget al\.,[2025](https://arxiv.org/html/2606.28707#bib.bib131)\), extending GRPO\-style optimization to multi\-turn tool\-use settings \(ARPO\)\(Donget al\.,[2025a](https://arxiv.org/html/2606.28707#bib.bib132)\), and extracting learning signals from zero\-variance prompts \(RL\-ZVP\)\(Leet al\.,[2025](https://arxiv.org/html/2606.28707#bib.bib27)\)\. Other work also mitigates zero\-variance or low\-signal training regimes through data\- or sampling\-level interventions, such as dynamic sampling\(Yuet al\.,[2025a](https://arxiv.org/html/2606.28707#bib.bib44)\)\.

Our method is most closely related to approaches that stabilize GRPO\-style advantage estimation without training a critic\. Compared with KRPO, which primarily smooths statistics across training steps, BV\-Blend maintains historical reward moments conditioned on semantic clusters of prompts\. Compared with RL\-ZVP, which specifically targets zero\-variance prompts, BV\-Blend uses uncertainty\-aware blending of prompt\-local and cluster\-conditioned historical statistics to form a unified standardized advantage for all prompt groups\. Compared with dynamic\-sampling\-based remedies, BV\-Blend operates at the level of the advantage estimator rather than the data\-selection policy\. Overall, BV\-Blend is a critic\-free method that modifies the advantage estimator while remaining compatible with standard PPO\-style training pipelines\.

![Refer to caption](https://arxiv.org/html/2606.28707v1/x1.png)Figure 1:BV\-Blend overview\.For each promptq\(m\)q^\{\(m\)\}, we sampleGGtrajectories\{τi\(m\)\}\\\{\\tau\_\{i\}^\{\(m\)\}\\\}with the behavior policy and obtain verifier rewards\{Ri\(m\)\}\\\{R\_\{i\}^\{\(m\)\}\\\}\. We compute prompt\-local statistics\(μ𝒢\(m\),σ𝒢\(m\)\)\(\\mu\_\{\\mathcal\{G\}\}^\{\(m\)\},\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}\), embedq\(m\)q^\{\(m\)\}, and assign it to a semantic clusterk\(m\)k^\{\(m\)\}\. Using*pre\-update*EMA moments\(μhist​\(k\),vhist​\(k\),Nkeff\)\(\\mu\_\{\\mathrm\{hist\}\}\(k\),v\_\{\\mathrm\{hist\}\}\(k\),N\_\{k\}^\{\\mathrm\{eff\}\}\), we compute the SEM\-based confidencewkw\_\{k\}\(Eq\. \([9](https://arxiv.org/html/2606.28707#S3.E9)\); cold start:wk=0w\_\{k\}\{=\}0for unseen clusters\), blend baseline and variance statistics to obtain\(b\(m\),s\(m\)\)\(b^\{\(m\)\},s^\{\(m\)\}\)\(Eq\. \([10](https://arxiv.org/html/2606.28707#S3.E10)\)\), and form advantagesAiBV,\(m\)A\_\{i\}^\{\\mathrm\{BV\},\(m\)\}\(Eq\. \([11](https://arxiv.org/html/2606.28707#S3.E11)\)\) for a PPO\-style update\. EMA moments are updated*post\-update*using the current batch\.

## 3Method

Critic\-free policy optimization for LLM alignment often relies on prompt\-local \(group\-dependent\) advantage normalization \(e\.g\., GRPO\-style estimators\), but when within\-group reward dispersion is small, prompt\-local standardization can yield near\-zero advantages and effectively remove the learning signal for that prompt group\. We proposeBV\-Blend, which constructs a*single*per\-trajectory advantage by combining prompt\-local statistics with semantic\-cluster\-conditioned historical moments, using a confidence weight derived from a standard\-error\-of\-the\-mean \(SEM\) proxy\. Concretely, BV\-Blend blends baseline and variance statistics, then standardizes once using the resulting scale\. BV\-Blend keeps the PPO\-style clipped objective unchanged\(Schulmanet al\.,[2017](https://arxiv.org/html/2606.28707#bib.bib95)\)and modifies only the advantage estimator: the SEM weight is computed from*pre\-update*EMA statistics, while EMA moments are updated*post\-update*using the current batch \(Fig\.[1](https://arxiv.org/html/2606.28707#S2.F1)\)\. As in other group\-based normalization methods, we stop gradients through all reward statistics and do not assume the resulting normalized estimator is unbiased\.

### 3\.1Background: prompt\-local normalization in critic\-free RL

A training batch containsMMprompts\{q\(m\)\}m=1M\\\{q^\{\(m\)\}\\\}\_\{m=1\}^\{M\}\. For each promptq\(m\)q^\{\(m\)\}, the behavior policyπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}samplesGGtrajectories𝒢​\(q\(m\)\)=\{τi\(m\)\}i=1G\\mathcal\{G\}\(q^\{\(m\)\}\)=\\\{\\tau^\{\(m\)\}\_\{i\}\\\}\_\{i=1\}^\{G\}\. Each trajectoryτi\(m\)=\(yi,1\(m\),…,yi,Ti\(m\)\(m\)\)\\tau^\{\(m\)\}\_\{i\}=\(y^\{\(m\)\}\_\{i,1\},\\dots,y^\{\(m\)\}\_\{i,T^\{\(m\)\}\_\{i\}\}\)receives a scalar trajectory\-level rewardRi\(m\)R^\{\(m\)\}\_\{i\}from an external verifier\.

For token positionttinτi\(m\)\\tau^\{\(m\)\}\_\{i\}, define the importance ratio

ri,t\(m\)​\(θ\)=πθ​\(yi,t\(m\)∣q\(m\),yi,<t\(m\)\)πθold​\(yi,t\(m\)∣q\(m\),yi,<t\(m\)\),r^\{\(m\)\}\_\{i,t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\\\!\\left\(y^\{\(m\)\}\_\{i,t\}\\mid q^\{\(m\)\},y^\{\(m\)\}\_\{i,<t\}\\right\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\\\!\\left\(y^\{\(m\)\}\_\{i,t\}\\mid q^\{\(m\)\},y^\{\(m\)\}\_\{i,<t\}\\right\)\},\(1\)and its clipped versionr~i,t\(m\)​\(θ\)=clip⁡\(ri,t\(m\)​\(θ\),1−ϵ,1\+ϵ\)\\tilde\{r\}^\{\(m\)\}\_\{i,t\}\(\\theta\)=\\operatorname\{clip\}\\\!\\bigl\(r^\{\(m\)\}\_\{i,t\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\\bigr\)\. Since rewards are trajectory\-level, we compute a single advantageAi\(m\)A^\{\(m\)\}\_\{i\}per trajectory and apply it to all completion tokens\. We restrict optimization to completion tokens using a maskmi,t\(m\)∈\{0,1\}m^\{\(m\)\}\_\{i,t\}\\in\\\{0,1\\\}and define masked token means as

𝔼t∼τi\(m\)​\[fi,t\]≜∑t=1Ti\(m\)mi,t\(m\)​fi,t∑t=1Ti\(m\)mi,t\(m\),\\mathbb\{E\}\_\{t\\sim\\tau^\{\(m\)\}\_\{i\}\}\[f\_\{i,t\}\]\\triangleq\\frac\{\\sum\_\{t=1\}^\{T\_\{i\}^\{\(m\)\}\}m^\{\(m\)\}\_\{i,t\}\\,f\_\{i,t\}\}\{\\sum\_\{t=1\}^\{T\_\{i\}^\{\(m\)\}\}m^\{\(m\)\}\_\{i,t\}\},\(2\)where each completion contains at least one generated token so the denominator is non\-zero\. Throughout,𝔼t∼τi\(m\)​\[⋅\]\\mathbb\{E\}\_\{t\\sim\\tau^\{\(m\)\}\_\{i\}\}\[\\cdot\]denotes a*mean*over completion tokens \(Eq\. \([2](https://arxiv.org/html/2606.28707#S3.E2)\)\) to avoid length\-dependent gradient scaling; concretely,mi,t\(m\)=1m^\{\(m\)\}\_\{i,t\}=1for generated completion tokens up to \(and including\) EOS, and0for prompt tokens and padding\. In implementation,Ai\(m\)A^\{\(m\)\}\_\{i\}is treated as a trajectory\-level constant \(no gradient through reward statistics\)\.

GRPO\-style training forms a prompt\-local standardized advantage\(Shaoet al\.,[2024](https://arxiv.org/html/2606.28707#bib.bib41)\):

AiGRPO,\(m\)=Ri\(m\)−μ𝒢\(m\)σ𝒢\(m\)\+δ,A^\{\\text\{GRPO\},\(m\)\}\_\{i\}=\\frac\{R^\{\(m\)\}\_\{i\}\-\\mu\_\{\\mathcal\{G\}\}^\{\(m\)\}\}\{\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}\+\\delta\},\(3\)whereμ𝒢\(m\)\\mu\_\{\\mathcal\{G\}\}^\{\(m\)\}andσ𝒢\(m\)\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}are the mean and standard deviation of\{Ri\(m\)\}i=1G\\\{R^\{\(m\)\}\_\{i\}\\\}\_\{i=1\}^\{G\}, andδ\>0\\delta\>0is a small constant\. IfG<2G<2, we setσ𝒢\(m\)=0\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}=0, yieldingA1GRPO,\(m\)=0A^\{\\text\{GRPO\},\(m\)\}\_\{1\}=0\. The clipped surrogate objective is

𝒥GRPO​\(θ\)=𝔼m\[1G∑i=1G𝔼t∼τi\(m\)\[min\(ri,t\(m\)\(θ\)AiGRPO,\(m\),r~i,t\(m\)\(θ\)AiGRPO,\(m\)\)\]\]\.\\displaystyle\\begin\{aligned\} \\mathcal\{J\}\_\{\\text\{GRPO\}\}\(\\theta\)&=\\mathbb\{E\}\_\{m\}\\\!\\Biggl\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\mathbb\{E\}\_\{t\\sim\\tau^\{\(m\)\}\_\{i\}\}\\\!\\Bigl\[\\min\\\!\\Bigl\(r^\{\(m\)\}\_\{i,t\}\(\\theta\)\\,A^\{\\text\{GRPO\},\(m\)\}\_\{i\},\\\\ &\\qquad\\qquad\\qquad\\qquad\\tilde\{r\}^\{\(m\)\}\_\{i,t\}\(\\theta\)\\,A^\{\\text\{GRPO\},\(m\)\}\_\{i\}\\Bigr\)\\Bigr\]\\Biggr\]\.\\end\{aligned\}

\(4\)
Whenσ𝒢\(m\)≈0\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}\\approx 0, Eq\. \([3](https://arxiv.org/html/2606.28707#S3.E3)\) collapses toward zero advantages, effectively removing the learning signal for that prompt group\.

### 3\.2BV\-Blend: SEM\-driven blending of historical and prompt\-local baselines

BV\-Blend computes a single standardized advantageAiBV,\(m\)A^\{\\mathrm\{BV\},\(m\)\}\_\{i\}by blending*baseline and variance statistics*and then standardizing once \(taking a square root to obtain the scale\)\. It follows the common formA=\(R−b\)/sA=\(R\-b\)/s: GRPO uses prompt\-local\(b,s\)=\(μ𝒢\(m\),σ𝒢\(m\)\)\(b,s\)=\(\\mu\_\{\\mathcal\{G\}\}^\{\(m\)\},\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}\), whereas BV\-Blend interpolates between prompt\-local statistics and*semantic\-cluster\-conditioned*EMA moments, with the interpolation controlled by a SEM\-based confidencewkw\_\{k\}\.

#### 3\.2\.1Prompt clustering

Each promptq\(m\)q^\{\(m\)\}is embedded by a frozen encoderE​\(⋅\)E\(\\cdot\)and assigned to a fixedKK\-means codebook\{cj\}j=1K\\\{c\_\{j\}\\\}\_\{j=1\}^\{K\}:

k\(m\)=arg⁡minj∈\{1,…,K\}⁡‖E​\(q\(m\)\)−cj‖22\.k^\{\(m\)\}=\\arg\\min\_\{j\\in\\\{1,\\dots,K\\\}\}\\ \\\|E\(q^\{\(m\)\}\)\-c\_\{j\}\\\|\_\{2\}^\{2\}\.\(5\)The codebook is trained offline on a representative prompt corpus and kept fixed during RL to avoid cluster\-identity drift\. We report the encoder choice,KK, codebook training corpus, andKK\-means settings \(implementation, seed, and iterations\) for reproducibility\.

#### 3\.2\.2Historical statistics with EMA moments

For each clusterkk, we maintain EMA moments: meanm1​\(k\)m\_\{1\}\(k\), second raw momentm2​\(k\)m\_\{2\}\(k\), and EMA massNkeffN^\{\\mathrm\{eff\}\}\_\{k\}\(herem1,m2m\_\{1\},m\_\{2\}denote moments, not the prompt indexmm\)\. Given batchℬ\\mathcal\{B\}, letℐℬ​\(k\)=\{\(m,i\):k\(m\)=k\}\\mathcal\{I\}\_\{\\mathcal\{B\}\}\(k\)=\\\{\(m,i\):k^\{\(m\)\}=k\\\}andNℬ​\(k\)=\|ℐℬ​\(k\)\|N\_\{\\mathcal\{B\}\}\(k\)=\|\\mathcal\{I\}\_\{\\mathcal\{B\}\}\(k\)\|\. Define sufficient statistics

S1,ℬ​\(k\)\\displaystyle S\_\{1,\\mathcal\{B\}\}\(k\)=∑\(m,i\)∈ℐℬ​\(k\)Ri\(m\),\\displaystyle=\\sum\_\{\(m,i\)\\in\\mathcal\{I\}\_\{\\mathcal\{B\}\}\(k\)\}R^\{\(m\)\}\_\{i\},S2,ℬ​\(k\)\\displaystyle S\_\{2,\\mathcal\{B\}\}\(k\)=∑\(m,i\)∈ℐℬ​\(k\)\(Ri\(m\)\)2\.\\displaystyle=\\sum\_\{\(m,i\)\\in\\mathcal\{I\}\_\{\\mathcal\{B\}\}\(k\)\}\\bigl\(R^\{\(m\)\}\_\{i\}\\bigr\)^\{2\}\.
In distributed training, we aggregate\{S1,ℬ​\(k\),S2,ℬ​\(k\),Nℬ​\(k\)\}\\\{S\_\{1,\\mathcal\{B\}\}\(k\),S\_\{2,\\mathcal\{B\}\}\(k\),N\_\{\\mathcal\{B\}\}\(k\)\\\}across workers and apply the EMA update*after*the policy optimization step\. IfNℬ​\(k\)\>0N\_\{\\mathcal\{B\}\}\(k\)\>0, define the batch mean and second raw moment

μℬ​\(k\)=S1,ℬ​\(k\)Nℬ​\(k\),μ2,ℬ​\(k\)=S2,ℬ​\(k\)Nℬ​\(k\)\.\\mu\_\{\\mathcal\{B\}\}\(k\)=\\frac\{S\_\{1,\\mathcal\{B\}\}\(k\)\}\{N\_\{\\mathcal\{B\}\}\(k\)\},\\qquad\\mu\_\{2,\\mathcal\{B\}\}\(k\)=\\frac\{S\_\{2,\\mathcal\{B\}\}\(k\)\}\{N\_\{\\mathcal\{B\}\}\(k\)\}\.With EMA rateγ∈\(0,1\]\\gamma\\in\(0,1\],

m1​\(k\)\\displaystyle m\_\{1\}\(k\)←\(1−γ\)​m1​\(k\)\+γ​μℬ​\(k\),\\displaystyle\\leftarrow\(1\-\\gamma\)m\_\{1\}\(k\)\+\\gamma\\,\\mu\_\{\\mathcal\{B\}\}\(k\),m2​\(k\)\\displaystyle m\_\{2\}\(k\)←\(1−γ\)​m2​\(k\)\+γ​μ2,ℬ​\(k\),\\displaystyle\\leftarrow\(1\-\\gamma\)m\_\{2\}\(k\)\+\\gamma\\,\\mu\_\{2,\\mathcal\{B\}\}\(k\),\(6\)Nkeff\\displaystyle N^\{\\mathrm\{eff\}\}\_\{k\}←\(1−γ\)​Nkeff\+γ​Nℬ​\(k\)\.\\displaystyle\\leftarrow\(1\-\\gamma\)N^\{\\mathrm\{eff\}\}\_\{k\}\+\\gamma\\,N\_\{\\mathcal\{B\}\}\(k\)\.We define

vhist​\(k\)\\displaystyle v\_\{\\mathrm\{hist\}\}\(k\)=max⁡\(m2​\(k\)−m1​\(k\)2,0\),\\displaystyle=\\max\\\!\\bigl\(m\_\{2\}\(k\)\-m\_\{1\}\(k\)^\{2\},0\\bigr\),\(7\)σhist​\(k\)\\displaystyle\\sigma\_\{\\mathrm\{hist\}\}\(k\)=vhist​\(k\),\\displaystyle=\\sqrt\{v\_\{\\mathrm\{hist\}\}\(k\)\},μhist​\(k\)\\displaystyle\\mu\_\{\\mathrm\{hist\}\}\(k\)=m1​\(k\)\.\\displaystyle=m\_\{1\}\(k\)\.
##### Initialization and cold start\.

On first observation of clusterkk, we initializeNkeff←N0,m1​\(k\)←μℬ​\(k\),m2​\(k\)←μℬ​\(k\)2\+Vprior,N^\{\\mathrm\{eff\}\}\_\{k\}\\\!\\leftarrow\\\!N\_\{0\},\\;m\_\{1\}\(k\)\\\!\\leftarrow\\\!\\mu\_\{\\mathcal\{B\}\}\(k\),\\;m\_\{2\}\(k\)\\\!\\leftarrow\\\!\\mu\_\{\\mathcal\{B\}\}\(k\)^\{2\}\+V\_\{\\mathrm\{prior\}\},withN0\>0N\_\{0\}\>0andVprior\>0V\_\{\\mathrm\{prior\}\}\>0\.

##### Update ordering\.

For each batch, we computewkw\_\{k\}using EMA statistics*before*incorporating the current batch\. If a clusterkkis first observed in the current batch \(i\.e\., no prior EMA state exists at advantage\-computation time\), we setwk=0w\_\{k\}=0for this batch \(pure prompt\-local normalization forABVA^\{\\mathrm\{BV\}\}\) and create its EMA state using the initialization above*after*the policy optimization step\. For previously seen clusters, EMA moments are updated*after*the policy optimization step using the aggregated sufficient statistics\.

#### 3\.2\.3Uncertainty\-to\-confidence mapping

We quantify historical uncertainty using a SEM\-style proxy

SEMhist​\(k\)=σhist​\(k\)Nkeff\+δN,\\mathrm\{SEM\}\_\{\\mathrm\{hist\}\}\(k\)=\\frac\{\\sigma\_\{\\mathrm\{hist\}\}\(k\)\}\{\\sqrt\{N^\{\\mathrm\{eff\}\}\_\{k\}\+\\delta\_\{N\}\}\},\(8\)withδN\>0\\delta\_\{N\}\>0\. We map uncertainty to a confidence weight

wk=exp⁡\(−SEMhist​\(k\)T\),w\_\{k\}=\\exp\\\!\\left\(\-\\frac\{\\mathrm\{SEM\}\_\{\\mathrm\{hist\}\}\(k\)\}\{T\}\\right\),\(9\)whereT\>0T\>0controls sensitivity to the reward scale\. For clusters with an EMA state,wk∈\(0,1\]w\_\{k\}\\in\(0,1\]: lower uncertainty yields largerwkw\_\{k\}\(more reliance on historical moments\), while higher uncertainty yields smallerwkw\_\{k\}\(more reliance on prompt\-local statistics\)\. We reportT,γ,N0,Vprior,δNT,\\gamma,N\_\{0\},V\_\{\\mathrm\{prior\}\},\\delta\_\{N\}in experiments\.

#### 3\.2\.4Baseline\-and\-scale blending and BV advantage

For each prompt groupmm, compute prompt\-local statisticsμ𝒢\(m\)\\mu\_\{\\mathcal\{G\}\}^\{\(m\)\}andσ𝒢\(m\)\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}over\{Ri\(m\)\}i=1G\\\{R^\{\(m\)\}\_\{i\}\\\}\_\{i=1\}^\{G\}\. Letw\(m\)=wk\(m\)w^\{\(m\)\}=w\_\{k^\{\(m\)\}\}\. We blend baseline and variance statistics, and take a square root to obtain the scale:

b\(m\)\\displaystyle b^\{\(m\)\}=w\(m\)​μhist​\(k\(m\)\)\+\(1−w\(m\)\)​μ𝒢\(m\),\\displaystyle=w^\{\(m\)\}\\,\\mu\_\{\\mathrm\{hist\}\}\(k^\{\(m\)\}\)\+\\bigl\(1\-w^\{\(m\)\}\\bigr\)\\,\\mu\_\{\\mathcal\{G\}\}^\{\(m\)\},\(10\)s\(m\)\\displaystyle s^\{\(m\)\}=w\(m\)​vhist​\(k\(m\)\)\+\(1−w\(m\)\)​\(σ𝒢\(m\)\)2,\\displaystyle=\\sqrt\{\\,w^\{\(m\)\}\\,v\_\{\\mathrm\{hist\}\}\(k^\{\(m\)\}\)\+\\bigl\(1\-w^\{\(m\)\}\\bigr\)\\bigl\(\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}\\bigr\)^\{2\}\\,\},and define the BV\-Blend advantage

AiBV,\(m\)=Ri\(m\)−b\(m\)s\(m\)\+δ\.A^\{\\mathrm\{BV\},\(m\)\}\_\{i\}=\\frac\{R^\{\(m\)\}\_\{i\}\-b^\{\(m\)\}\}\{s^\{\(m\)\}\+\\delta\}\.\(11\)Whenσ𝒢\(m\)≈0\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}\\approx 0, the blended scales\(m\)s^\{\(m\)\}remains non\-degenerate whenever the historical term provides non\-zero variance, thereby preventing collapse of the learning signal\.

## 4Experiments

##### Domain and Datasets\.

We focus on mathematical reasoning, where evaluation is rigorous and objective\. Compared with more subjective tasks, math offers \(i\) deterministic, programmatic verification against ground\-truth final answers, reducing reliance on preference labels and learned judges that may be biased or exploitable\(Huanget al\.,[2025](https://arxiv.org/html/2606.28707#bib.bib79); Zhenget al\.,[2025](https://arxiv.org/html/2606.28707#bib.bib80)\); \(ii\) scalable automated evaluation without costly human annotation; and \(iii\) diverse multi\-step problems with unambiguous correctness criteria\. Our training set contains 45,000 problems curated from the default 94k split of OpenR1\-Math\-220k\(Face,[2025](https://arxiv.org/html/2606.28707#bib.bib81); Yanet al\.,[2025](https://arxiv.org/html/2606.28707#bib.bib67)\)\. OpenR1\-Math\-220k is built from NuminaMath 1\.5 prompts and includes 2–4 reasoning traces generated by DeepSeek\-R1; most traces are verified byMath\-Verify\(with a small portion additionally judged by an LLM\), and each problem has at least one correct trace\(Face,[2025](https://arxiv.org/html/2606.28707#bib.bib81); LIet al\.,[2024](https://arxiv.org/html/2606.28707#bib.bib125); Guoet al\.,[2025](https://arxiv.org/html/2606.28707#bib.bib71)\)\. We useMath\-Verifyto remove instances with invalid or unverifiable final answers, and further filter samples whose traces exceed 8,192 tokens\. This curated set is used as \(1\) prompts for on\-policy rollouts, \(2\) ground\-truth answers for deterministic reward computation, and \(3\) a corpus of reasoning traces for training SFT baselines\.

Table 1:Main results on Qwen2\.5\-Math\-7B\. We compare BV\-Blend with baselines on mathematical reasoning and OOD generalization benchmarks\. Best and second\-best in each column are inboldandunderlined\. Math Reasoning Avg\. averages AIME 2024/2025, AMC, MATH\-500, Minerva, and Olympiad; Generalization Avg\. averages ARC\-C, GPQA\*, and MMLU\-Pro\.
##### Evaluation\.

We evaluate BV\-Blend on benchmarks probing two complementary aspects: \(i\)*in\-domain*mathematical reasoning across a broad difficulty range—from pre\-college competition problems \(AMC\(Heet al\.,[2024](https://arxiv.org/html/2606.28707#bib.bib84)\)\), through university\-level problem solving \(MATH\-500\(Hendryckset al\.,[2021](https://arxiv.org/html/2606.28707#bib.bib85)\), Minerva\(Lewkowyczet al\.,[2022](https://arxiv.org/html/2606.28707#bib.bib86)\)\), to elite competition\-level challenges \(AIME 2024/2025\(Liet al\.,[2024](https://arxiv.org/html/2606.28707#bib.bib90)\)\); and \(ii\)*out\-of\-distribution*\(OOD\) generalization to non\-mathematical benchmarks, including abstract reasoning \(ARC\-C\(Clarket al\.,[2018](https://arxiv.org/html/2606.28707#bib.bib87)\)\), expert\-level scientific knowledge \(GPQA\-diamond\(Reinet al\.,[2023](https://arxiv.org/html/2606.28707#bib.bib88)\)\), and multidisciplinary problem solving \(MMLU\-Pro\(Wanget al\.,[2024](https://arxiv.org/html/2606.28707#bib.bib89)\)\)\. Our main results use Qwen2\.5\-Math\-7B\(Yanget al\.,[2024c](https://arxiv.org/html/2606.28707#bib.bib82)\)as the backbone; additional backbones are reported in the*Implementation Details*paragraph below\. Unless otherwise specified, we decode for evaluation with temperature0\.60\.6\. For multiple\-choice benchmarks, we randomly permute answer options for each question \(with labels remapped accordingly\) to mitigate position bias\. We report pass@1 on large\-scale benchmarks \(MATH\-500, Minerva, and the OOD suite\)\. For smaller and more challenging test sets \(AIME and AMC\), we instead report avg@32, defined as the mean success rate over 32 independent samples per problem, which is more stable under stochastic decoding\.

##### Baselines\.

To contextualize BV\-Blend, we compare against baselines built on the same Qwen2\.5\-Math\-7B backbone \(Table[1](https://arxiv.org/html/2606.28707#S4.T1)\), grouped into two categories\.\(1\) On\-policy RLVR methodsstart from the base model and optimize using only on\-policy rollouts with verifiable rewards \(i\.e\., without SFT initialization or off\-policy demonstrations\)\. This category includes our replication of GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2606.28707#bib.bib41)\)and representative “Zero” RLVR systems trained under different reward designs and training recipes: SimpleRL\-Zero\(Zenget al\.,[2025](https://arxiv.org/html/2606.28707#bib.bib64)\), Open\-Reasoner\-Zero\(Huet al\.,[2025](https://arxiv.org/html/2606.28707#bib.bib77)\), PRIME\-Zero\(Cuiet al\.,[2025](https://arxiv.org/html/2606.28707#bib.bib76)\), and Oat\-Zero\(Liuet al\.,[2025a](https://arxiv.org/html/2606.28707#bib.bib96)\)\.\(2\) Hybrid & off\-policy methodsleverage additional external data beyond on\-policy rollouts, including SFT, a sequential SFT→\\rightarrowRL pipeline, ReLIFT\(Maet al\.,[2025](https://arxiv.org/html/2606.28707#bib.bib68)\), and LUFFY\(Yanet al\.,[2025](https://arxiv.org/html/2606.28707#bib.bib67)\)\. For LUFFY, we report both the standard and extended\-training \(†\\dagger\) variants when available\.

![Refer to caption](https://arxiv.org/html/2606.28707v1/x2.png)\(a\)Response length
![Refer to caption](https://arxiv.org/html/2606.28707v1/x3.png)\(b\)Training entropy
![Refer to caption](https://arxiv.org/html/2606.28707v1/x4.png)\(c\)Training reward
![Refer to caption](https://arxiv.org/html/2606.28707v1/x5.png)\(d\)Effective signal ratio

Figure 2:GRPO vs\. BV\-Blend\.We track \(a\) response length \(tokens\), \(b\) policy training entropy, \(c\) mean training reward \(verifier score\), and \(d\) the effective\-signal ratio: the fraction of prompts whose method\-specific normalization scale remains non\-degenerate during training \(GRPO:σ𝒢\(m\)\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}; BV\-Blend:s\(m\)s^\{\(m\)\}in Eq\. \([10](https://arxiv.org/html/2606.28707#S3.E10)\)\)\.![Refer to caption](https://arxiv.org/html/2606.28707v1/x6.png)Figure 3:Difficulty\-stratified BV\-Blend vs\. GRPO\.We partition prompts into four difficulty buckets \(Easy/Medium/Hard/Hardest\) using a fixed pre\-RL difficulty estimate \(Appendix[C\.4](https://arxiv.org/html/2606.28707#A3.SS4)\) shared across methods, and track*verifier accuracy*\(a,b; fraction of prompts with correct final answers\) and*average response length*\(c,d\) across training checkpoints\. The left pair \(a,c\) reports theStandardsubset and the right pair \(b,d\) reports theComplexsubset\. Response length is measured on completion tokens and plotted on a log scale\.
##### Implementation Details\.

We evaluate BV\-Blend across multiple backbones\. Our primary testbed is Qwen2\.5\-Math\-7B to align with prior RLVR work\. We additionally report results on Qwen2\.5\-Math\-1\.5B, Qwen2\.5\-7B\-Instruct, and Llama\-3\.1\-8B\-Instruct to assess robustness across model scales and instruction tuning\. Unless otherwise specified, we use a shared training setup across all runs\. We optimize with AdamW and a cosine learning\-rate schedule with linear warmup, with a peak learning rate of1×10−61\\times 10^\{\-6\}\. Each iteration collects a global rollout batch of 128 trajectories \(16 prompts×\\times8 rollouts\) with sampling temperature1\.01\.0, and performs policy updates with a trajectory minibatch size of 64\. Rewards are binary: we assign\+1\+1if the extracted final answer is verified as correct byMath\-Verify, and0otherwise, with no intermediate or format\-based shaping rewards\. We use a PPO\-style clipped objective shared by BV\-Blend and the PPO\-style baselines, with clippingϵ=0\.2\\epsilon=0\.2and an entropy bonus coefficient of0\.010\.01\. Unless otherwise stated, we include a KL\-to\-reference term in the objective but set its coefficient toβ=0\\beta=0in the main experiments, effectively disabling it to isolate the effect of different advantage estimators\. For BV\-Blend, we maintain cluster\-conditioned historical moments using EMA with update rateγ=0\.9\\gamma=0\.9, and useδ=10−8\\delta=10^\{\-8\}for numerical stability in advantage computation\. The remaining BV\-Blend\-specific hyperparameters and clustering settings are fixed across experiments unless otherwise stated\. We do not apply any additional global advantage normalization beyond the method\-specific estimator \(e\.g\., GRPO or BV\-Blend\)\. For purely on\-policy approaches \(including BV\-Blend and our GRPO replication\), all 8 rollouts per prompt are generated on\-policy from the current policy\. All models are trained for 500 iterations\. All experiments are conducted on 8 NVIDIA RTX PRO 6000 GPUs \(96 GB VRAM each\)\.

### 4\.1Main Results

Table[1](https://arxiv.org/html/2606.28707#S4.T1)summarizes results on the Qwen2\.5\-Math\-7B backbone\. Overall, BV\-Blend achieves the best average performance among the compared methods on both in\-domain mathematical reasoning and OOD benchmarks\. Notably, BV\-Blend is a purely on\-policy RLVR approach, yet it remains competitive with hybrid/off\-policy systems that additionally leverage external data\.

##### In\-Domain Mathematical Reasoning\.

We evaluate in\-domain performance on five math benchmarks, where AIME is reported for two years \(AIME 2024/2025\), yielding six scores in total for the Math Reasoning Avg\. AIME and AMC are reported as avg@32, while the remaining benchmarks use pass@1\. Across these six scores, BV\-Blend attains the highest average \(51\.7%\), outperforming the strongest on\-policy baseline Oat\-Zero \(43\.8%\) by 7\.9 points\. It also exceeds ReLIFT \(47\.8%\) and LUFFY \(50\.1%\), and is slightly above the extended\-training LUFFY†\\daggervariant \(51\.0%\)\. At the per\-benchmark level, BV\-Blend achieves the best reported results on AIME 2024 \(34\.2%\), AMC \(66\.5%\), MATH\-500 \(87\.9%\), and Olympiad \(57\.4%\)\. Meanwhile, it is marginally below the best baseline on Minerva \(40\.7% vs\. 41\.2%\) and on AIME 2025 \(23\.6% vs\. 25\.5%\), indicating broad but non\-uniform gains across evaluation settings\.

##### OOD Generalization\.

On the OOD suite \(ARC\-C, GPQA\*, and MMLU\-Pro\), BV\-Blend again achieves the best average score \(64\.1%\), improving over the next\-best baseline LUFFY†\\dagger\(61\.8%\) by 2\.3 points\. The largest gains appear on reasoning\-centric benchmarks: BV\-Blend achieves the best results on ARC\-C \(83\.1%\) and GPQA\* \(52\.6%\), exceeding LUFFY†\\daggerby 3\.6 points on GPQA\*\. On MMLU\-Pro, BV\-Blend is slightly below the best baseline \(56\.5% vs\. 58\.7%\), suggesting that improvements are more pronounced on benchmarks emphasizing multi\-step reasoning, while broad knowledge coverage may depend more on pretraining and instruction tuning\.

Taken together, Table[1](https://arxiv.org/html/2606.28707#S4.T1)shows that stabilizing the advantage signal can improve on\-policy RLVR training, yielding substantial gains without requiring additional supervision beyond verifiable rewards, while remaining competitive with hybrid/off\-policy approaches that leverage extra data\.

Table 2:Ablations of BV\-Blend on Qwen2\.5\-Math\-7B\.We ablate the SEM\-based confidence weighting by constructingwkw\_\{k\}from only the EMA effective countNkeffN\_\{k\}^\{\\text\{eff\}\}or only the historical scaleσhist​\(k\)\\sigma\_\{\\text\{hist\}\}\(k\)\. Avg\. is computed over six scores \(AIME 2024, AIME 2025, AMC, MATH\-500, Minerva, Olympiad\)\.

### 4\.2Analysis of Training Dynamics

To diagnose how BV\-Blend shapes optimization, Fig\.[2](https://arxiv.org/html/2606.28707#S4.F2)compares BV\-Blend with the on\-policy baseline GRPO across four training signals\. BV\-Blend consistently produces longer responses and reaches a higher, more stable length plateau \(Fig\.[2\(a\)](https://arxiv.org/html/2606.28707#S4.F2.sf1)\), which is consistent with sustaining multi\-step reasoning traces under the same rollout budget\. While entropy decreases for both methods early in training, BV\-Blend maintains a higher residual policy entropy throughout \(Fig\.[2\(b\)](https://arxiv.org/html/2606.28707#S4.F2.sf2)\), which is consistent with slower policy concentration and more persistent exploration\. This is accompanied by slightly higher and noticeably smoother training rewards \(Fig\.[2\(c\)](https://arxiv.org/html/2606.28707#S4.F2.sf3)\), with reduced volatility relative to GRPO\.

Most importantly, the effective\-signal ratio \(Fig\.[2\(d\)](https://arxiv.org/html/2606.28707#S4.F2.sf4)\) highlights a difference in the resulting learning signal\. We define the effective\-signal ratio as the fraction of prompts whose normalization scale is non\-degenerate under the corresponding estimator \(GRPO:σ𝒢\(m\)\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}; BV\-Blend:s\(m\)s^\{\(m\)\}\)\. With binary verifier rewards, prompt groups frequently become near\-deterministic \(all\-correct or all\-incorrect\), causingσ𝒢\(m\)\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}to approach zero and thereby collapsing GRPO advantages toward zero, which reduces the number of informative prompt groups available for learning\(Shaoet al\.,[2024](https://arxiv.org/html/2606.28707#bib.bib41)\)\. In contrast, BV\-Blend preserves a substantially higher effective\-signal ratio by stabilizing the baseline and scale via cluster\-conditioned historical moments when prompt\-local dispersion is small, thereby maintaining usable learning signals and yielding more stable optimization dynamics overall\.

### 4\.3Difficulty\-stratified dynamics

To better understand where BV\-Blend’s gains arise and whether they are accompanied by undesirable verbosity, we analyze learning dynamics under a fixed difficulty stratification\. Specifically, following the protocol described in Appendix[C\.4](https://arxiv.org/html/2606.28707#A3.SS4), we assign each prompt to one of four buckets \(Easy/Medium/Hard/Hardest\) using a pre\-RL difficulty estimate, and evaluate both BV\-Blend and the on\-policy baseline GRPO at the same checkpoints \(Steps0/30/60/90/1200/30/60/90/120\)\. Fig\.[3](https://arxiv.org/html/2606.28707#S4.F3)reports bucket\-wise average verifier accuracy and average completion length \(log scale\) on bothStandardandComplexsubsets\. Overall, BV\-Blend yields consistent accuracy improvements on harder buckets—most notably Hard and Hardest—while keeping response lengths comparable and avoiding systematic length blow\-up\. We observe mild non\-monotonic fluctuations across checkpoints, which is expected in on\-policy optimization due to sampling noise; importantly, BV\-Blend shows a more consistent upward trend on difficult prompts, consistent with the hypothesis that historically stabilized advantage estimation improves training stability without evidence of systematic length inflation\.

### 4\.4Ablation Study

Table[2](https://arxiv.org/html/2606.28707#S4.T2)ablates the core design of BV\-Blend on Qwen2\.5\-Math\-7B\. We compare GRPO, a naive fixed\-weight historical mixture \(w=0\.5w\{=\}0\.5\), two partial variants that compute the confidence weight using only one historical\-uncertainty ingredient, and the full BV\-Blend\. All BV\-Blend variants use the same baseline/scale blending in Eq\. \([10](https://arxiv.org/html/2606.28707#S3.E10)\) and differ only in howwkw\_\{k\}is computed from historical statistics\. Concretely, the full method computes the SEM\-style uncertaintySEMhist​\(k\)=σhist​\(k\)/Nkeff\+δN\\mathrm\{SEM\}\_\{\\mathrm\{hist\}\}\(k\)=\\sigma\_\{\\mathrm\{hist\}\}\(k\)/\\sqrt\{N\_\{k\}^\{\\mathrm\{eff\}\}\+\\delta\_\{N\}\}and maps it to confidence via Eq\. \([9](https://arxiv.org/html/2606.28707#S3.E9)\)\. TheNkeffN\_\{k\}^\{\\mathrm\{eff\}\}\-only variant drops the dependence onσhist​\(k\)\\sigma\_\{\\mathrm\{hist\}\}\(k\)\(i\.e\.,SEM​\(k\)∝1/Nkeff\+δN\\mathrm\{SEM\}\(k\)\\propto 1/\\sqrt\{N\_\{k\}^\{\\mathrm\{eff\}\}\+\\delta\_\{N\}\}\), while theσhist\\sigma\_\{\\mathrm\{hist\}\}\-only variant drops the dependence onNkeffN\_\{k\}^\{\\mathrm\{eff\}\}\(i\.e\.,SEM​\(k\)∝σhist​\(k\)\\mathrm\{SEM\}\(k\)\\propto\\sigma\_\{\\mathrm\{hist\}\}\(k\)\); in all cases,wkw\_\{k\}is a monotone function of the corresponding uncertainty proxy\.

Two observations emerge\. First, simply injecting historical information is insufficient: naive averaging drops from 45\.5 \(GRPO\) to 43\.7, indicating that historical moments should be used selectively rather than uniformly\. Second, either signal alone already improves over GRPO \(49\.7 and 49\.1\), and combining them through the SEM proxy performs best \(51\.7\), improving by 6\.2 points over GRPO and by 2\.0 points over the best single\-signal variant\. The gains are particularly noticeable on harder benchmarks, such as Olympiad and AIME, which is consistent with the role of SEM\-based confidence in stabilizing advantage estimation when prompt\-local reward dispersion is small\.

## 5Conclusion

We identify an instability in critic\-free RLVR: when prompt\-local reward dispersion is small, group\-normalized advantages can collapse and weaken the learning signal\. We propose BV\-Blend \(BV\-Blend\), which stabilizes advantage estimation by blending prompt\-local statistics with semantic\-cluster\-conditioned historical moments through an SEM\-based confidence weight, while leaving the PPO\-style clipped objective unchanged\. Across experiments, BV\-Blend improves training stability and performance on in\-domain, OOD, and cross\-backbone evaluations\. Future work will explore more effective mechanisms for sharing historical information across prompts, especially under distribution shift\.

## Limitations

BV\-Blend relies on a fixed prompt embedding and clustering pipeline; performance may depend on the encoder and codebook granularity, and may degrade under substantial distribution shift\. It also requires maintaining cluster\-conditioned EMA moments and aggregating per\-cluster statistics in distributed training, which introduces additional bookkeeping and hyperparameter tuning\. Finally, we primarily evaluate mathematical reasoning with verifiable outcome rewards and a limited set of backbones; broader domains and verifier/reward designs are needed to more fully assess generality\. In addition, under extremely sparse reward regimes, if both the current prompt group and its relevant historical cluster provide little or no reward variation, BV\-Blend may still offer limited learning signal\.

## Acknowledgments

This work is supported by the National Key Research and Development Program of China \(No\.2023YFF0905400\), the National Natural Science Foundation of China \(No\.U2341229\) and the Reform Commission Foundation of Jilin Province \(No\.2024C003\)\.

## References

- BA\-loRA: bias\-alleviating low\-rank adaptation to mitigate catastrophic inheritance in large language models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=q0X9SiXiRO)Cited by:[4th item](https://arxiv.org/html/2606.28707#A1.I3.i4.p1.1)\.
- Y\. Chang, C\. Guo, Y\. Chang, and Y\. Wu \(2025\)LoRA\-mgpo: mitigating double descent in low\-rank adaptation via momentum\-guided perturbation optimization\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 648–659\.Cited by:[4th item](https://arxiv.org/html/2606.28707#A1.I3.i4.p1.1)\.
- Y\. Chang, X\. Wang, J\. Wang, Y\. Wu, L\. Yang, K\. Zhu, H\. Chen, X\. Yi, C\. Wang, Y\. Wang,et al\.\(2024\)A survey on evaluation of large language models\.ACM transactions on intelligent systems and technology15\(3\),pp\. 1–45\.Cited by:[§A\.3](https://arxiv.org/html/2606.28707#A1.SS3.SSS0.Px1.p1.3)\.
- P\. Chen, X\. Li, Z\. Li, X\. Chen, and T\. Lin \(2025\)Spectral policy optimization: coloring your incorrect reasoning in grpo\.arXiv preprint arXiv:2505\.11595\.Cited by:[§1](https://arxiv.org/html/2606.28707#S1.p3.1)\.
- X\. Chen, W\. Xue, and Y\. Guo \(2026\)ActorMind: emulating human actor reasoning for speech role\-playing\.External Links:2604\.11103,[Link](https://arxiv.org/abs/2604.11103)Cited by:[§A\.3](https://arxiv.org/html/2606.28707#A1.SS3.SSS0.Px1.p1.3)\.
- X\. Chen and M\. Zeng \(2025\)Prototype conditioned generative replay for continual learning in NLP\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 12754–12770\.External Links:[Link](https://aclanthology.org/2025.naacl-long.636/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.636),ISBN 979\-8\-89176\-189\-6Cited by:[4th item](https://arxiv.org/html/2606.28707#A1.I3.i4.p1.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the AI2 reasoning challenge\.CoRRabs/1803\.05457\.External Links:[Link](http://arxiv.org/abs/1803.05457)Cited by:[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px2.p1.1)\.
- G\. Cui, L\. Yuan, Z\. Wang, H\. Wang, W\. Li, B\. He, Y\. Fan, T\. Yu, Q\. Xu, W\. Chen,et al\.\(2025\)Process reinforcement through implicit rewards\.arXiv preprint arXiv:2502\.01456\.Cited by:[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px3.p1.2)\.
- G\. Dong, H\. Mao, K\. Ma, L\. Bao, Y\. Chen, Z\. Wang, Z\. Chen, J\. Du, H\. Wang, F\. Zhang,et al\.\(2025a\)Agentic reinforced policy optimization\.arXiv preprint arXiv:2507\.19849\.Cited by:[§2](https://arxiv.org/html/2606.28707#S2.SS0.SSS0.Px2.p1.1)\.
- H\. Dong, K\. Jiang, H\. Ye, W\. Zhu, Z\. Kang, and G\. Song \(2026\)NeuReasoner: towards explainable, controllable, and unified reasoning via mixture\-of\-neurons\.External Links:2604\.02972,[Link](https://arxiv.org/abs/2604.02972)Cited by:[§A\.3](https://arxiv.org/html/2606.28707#A1.SS3.SSS0.Px1.p1.3)\.
- H\. Dong, W\. Zhu, G\. Song, and L\. Wang \(2025b\)AuroRA: breaking low\-rank bottleneck of loRA with nonlinear mapping\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=2hgHyoyVWj)Cited by:[4th item](https://arxiv.org/html/2606.28707#A1.I3.i4.p1.1)\.
- K\. Ethayarajh, W\. Xu, N\. Muennighoff, D\. Jurafsky, and D\. Kiela \(2024\)Kto: model alignment as prospect theoretic optimization\.arXiv preprint arXiv:2402\.01306\.Cited by:[§2](https://arxiv.org/html/2606.28707#S2.SS0.SSS0.Px1.p1.1)\.
- \[13\]H\. FaceChat templates\.Note:[https://huggingface\.co/docs/transformers/main/en/chat\_templating](https://huggingface.co/docs/transformers/main/en/chat_templating)Transformers documentation \(main\)\. Accessed: 2026\-01\-05\.Cited by:[3rd item](https://arxiv.org/html/2606.28707#A1.I3.i3.p1.1),[§B\.4](https://arxiv.org/html/2606.28707#A2.SS4.p1.1)\.
- H\. Face \(2025\)Open r1: a fully open reproduction of deepseek\-r1\.External Links:[Link](https://github.com/huggingface/open-r1)Cited by:[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px1.p1.1)\.
- J\. Gui, Y\. Liu, J\. Cheng, X\. Gu, X\. Liu, H\. Wang, Y\. Dong, J\. Tang, and M\. Huang \(2024\)Logicgame: benchmarking rule\-based reasoning abilities of large language models\.arXiv preprint arXiv:2408\.15778\.Cited by:[§1](https://arxiv.org/html/2606.28707#S1.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2606.28707#S1.p1.1),[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px1.p1.1)\.
- Z\. Hao, H\. Wang, H\. Liu, J\. Luo, J\. Yu, H\. Dong, Q\. Lin, C\. Wang, and J\. Chen \(2025\)Rethinking entropy interventions in rlvr: an entropy change perspective\.arXiv preprint arXiv:2510\.10150\.Cited by:[§A\.3](https://arxiv.org/html/2606.28707#A1.SS3.SSS0.Px1.p1.3)\.
- Z\. Hao, H\. Wang, J\. Luo, J\. Zhang, Y\. Zhou, Q\. Lin, C\. Wang, H\. Dong, and J\. Chen \(2026\)ReCreate: reasoning and creating domain agents driven by experience\.arXiv preprint arXiv:2601\.11100\.Cited by:[§A\.3](https://arxiv.org/html/2606.28707#A1.SS3.SSS0.Px1.p1.3)\.
- C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. L\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang, J\. Liu, L\. Qi, Z\. Liu, and M\. Sun \(2024\)OlympiadBench: A challenging benchmark for promoting AGI with olympiad\-level bilingual multimodal scientific problems\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2024, Bangkok, Thailand, August 11\-16, 2024,pp\. 3828–3850\.External Links:[Link](https://doi.org/10.18653/v1/2024.acl-long.211)Cited by:[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px2.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the MATH dataset\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual,External Links:[Link](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html)Cited by:[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px2.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.ICLR1\(2\),pp\. 3\.Cited by:[§B\.3](https://arxiv.org/html/2606.28707#A2.SS3.SSS0.Px1.p1.1)\.
- J\. Hu, Y\. Zhang, Q\. Han, D\. Jiang, X\. Zhang, and H\. Shum \(2025\)Open\-Reasoner\-Zero: an open source approach to scaling up reinforcement learning on the base model\.arXiv preprint arXiv:2503\.24290\.Cited by:[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px3.p1.2)\.
- Z\. Huang, Z\. Qiu, Z\. Wang, E\. M\. Ponti, and I\. Titov \(2025\)Post\-hoc reward calibration: A case study on length bias\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=Iu8RytBaji)Cited by:[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px1.p1.1)\.
- K\. Jiang, H\. Dong, Z\. Kang, Z\. Zhu, and G\. Song \(2026\)FoE: forest of errors makes the first solution the best in large reasoning models\.External Links:2604\.02967,[Link](https://arxiv.org/abs/2604.02967)Cited by:[§A\.3](https://arxiv.org/html/2606.28707#A1.SS3.SSS0.Px1.p1.3)\.
- H\. Kydlicek, A\. Lozovskaya, N\. Habib, and C\. Fourrier \(2025\)Fixing open llm leaderboard and introducing math\-verify\.Note:[https://huggingface\.co/blog/math\_verify\_leaderboard](https://huggingface.co/blog/math_verify_leaderboard)Hugging Face Blog\. Accessed: 2026\-01\-05\.Cited by:[§A\.3](https://arxiv.org/html/2606.28707#A1.SS3.SSS0.Px1.p1.2),[§A\.3](https://arxiv.org/html/2606.28707#A1.SS3.SSS0.Px1.p1.3)\.
- T\. V\. Le, M\. Jeon, K\. Vu, V\. Lai, and E\. Yang \(2025\)No prompt left behind: exploiting zero\-variance prompts in llm reinforcement learning via entropy\-guided advantage shaping\.arXiv preprint arXiv:2509\.21880\.Cited by:[§1](https://arxiv.org/html/2606.28707#S1.p3.1),[§2](https://arxiv.org/html/2606.28707#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Lewkowycz, A\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo, Y\. Wu, B\. Neyshabur, G\. Gur\-Ari, and V\. Misra \(2022\)Solving quantitative reasoning problems with language models\.InAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 \- December 9, 2022,External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2022/hash/18abbeef8cfe9203fdf9053c9c4fe191-Abstract-Conference.html)Cited by:[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px2.p1.1)\.
- J\. LI, E\. Beeching, L\. Tunstall, B\. Lipkin, R\. Soletskyi, S\. C\. Huang, K\. Rasul, L\. Yu, A\. Jiang, Z\. Shen, Z\. Qin, B\. Dong, L\. Zhou, Y\. Fleureau, G\. Lample, and S\. Polu \(2024\)NuminaMath\.Numina\.Cited by:[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px1.p1.1)\.
- J\. Li, E\. Beeching, L\. Tunstall, B\. Lipkin, R\. Soletskyi, S\. Huang, K\. Rasul, L\. Yu, A\. Q\. Jiang, Z\. Shen,et al\.\(2024\)Numinamath: the largest public dataset in ai4maths with 860k pairs of competition math problems and solutions\.Note:[https://huggingface\.co/datasets/Numinamath](https://huggingface.co/datasets/Numinamath)Hugging Face repository, 13:9Cited by:[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px2.p1.1)\.
- Z\. Li, Y\. Hu, Z\. Chen, Q\. Huang, G\. Qiu, Z\. Fu, and M\. Liu \(2026a\)ReTrack: evidence\-driven dual\-stream directional anchor calibration network for composed video retrieval\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 23373–23381\.Cited by:[§A\.3](https://arxiv.org/html/2606.28707#A1.SS3.SSS0.Px1.p1.3)\.
- Z\. Li, Y\. Hu, Z\. Chen, S\. Zhang, Q\. Huang, Z\. Fu, and Y\. Wei \(2026b\)HABIT: chrono\-synergia robust progressive learning framework for composed image retrieval\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 6762–6770\.Cited by:[§A\.3](https://arxiv.org/html/2606.28707#A1.SS3.SSS0.Px1.p1.3)\.
- X\. Liang \(2025\)Group relative policy optimization for image captioning\.arXiv preprint arXiv:2503\.01333\.Cited by:[§1](https://arxiv.org/html/2606.28707#S1.p3.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2023\)Let’s verify step by step\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.28707#S1.p1.1)\.
- Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. Lin \(2025a\)Understanding r1\-zero\-like training: a critical perspective\.arXiv preprint arXiv:2503\.20783\.Cited by:[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px3.p1.2)\.
- Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. Lin \(2025b\)Understanding r1\-zero\-like training: a critical perspective\.Note:[https://github\.com/sail\-sg/understand\-r1\-zero](https://github.com/sail-sg/understand-r1-zero)Cited by:[§2](https://arxiv.org/html/2606.28707#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Ma, H\. Liang, M\. Qiang, L\. Tang, X\. Ma, Z\. H\. Wong, J\. Niu, C\. Shen, R\. He, B\. Cui,et al\.\(2025\)Learning what reinforcement learning can’t: interleaved online fine\-tuning for hardest questions\.arXiv preprint arXiv:2506\.07527\.Cited by:[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px3.p1.2)\.
- Y\. Meng, M\. Xia, and D\. Chen \(2024\)Simpo: simple preference optimization with a reference\-free reward\.arXiv preprint arXiv:2405\.14734\.Cited by:[§2](https://arxiv.org/html/2606.28707#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Mroueh \(2025\)Reinforcement learning with verifiable rewards: grpo’s effective loss, dynamics, and success amplification\.arXiv preprint arXiv:2503\.06639\.Cited by:[§1](https://arxiv.org/html/2606.28707#S1.p3.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2606.28707#S1.p2.1),[§2](https://arxiv.org/html/2606.28707#S2.SS0.SSS0.Px1.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2024\)Direct preference optimization: your language model is secretly a reward model\.Advances in Neural Information Processing Systems36\.Cited by:[§1](https://arxiv.org/html/2606.28707#S1.p2.1),[§2](https://arxiv.org/html/2606.28707#S2.SS0.SSS0.Px1.p1.1)\.
- D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman \(2023\)GPQA: A graduate\-level google\-proof q&a benchmark\.CoRRabs/2311\.12022\.External Links:[Link](https://doi.org/10.48550/arXiv.2311.12022)Cited by:[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px2.p1.1)\.
- J\. Schulman, P\. Moritz, S\. Levine, M\. Jordan, and P\. Abbeel \(2015\)High\-dimensional continuous control using generalized advantage estimation\.arXiv preprint arXiv:1506\.02438\.Cited by:[§2](https://arxiv.org/html/2606.28707#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§B\.3](https://arxiv.org/html/2606.28707#A2.SS3.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2606.28707#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2606.28707#S3.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2606.28707#S1.p2.1),[§2](https://arxiv.org/html/2606.28707#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2606.28707#S3.SS1.p3.8),[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px3.p1.2),[§4\.2](https://arxiv.org/html/2606.28707#S4.SS2.p2.3)\.
- H\. Shen, T\. Ying, J\. Lu, and S\. Wang \(2026\)Preconditioned test\-time adaptation for out\-of\-distribution debiasing in narrative generation\.arXiv preprint arXiv:2603\.13683\.Cited by:[§A\.3](https://arxiv.org/html/2606.28707#A1.SS3.SSS0.Px1.p1.3)\.
- Y\. Su, D\. Yu, L\. Song, J\. Li, H\. Mi, Z\. Tu, M\. Zhang, and D\. Yu \(2025\)Crossing the reward bridge: expanding rl with verifiable rewards across diverse domains\.arXiv preprint arXiv:2503\.23829\.Cited by:[§1](https://arxiv.org/html/2606.28707#S1.p1.1)\.
- M\. Team \(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[2nd item](https://arxiv.org/html/2606.28707#A1.I3.i2.p1.1)\.
- K\. Tian, Y\. Tang, G\. Jiang, B\. Liu, Y\. Gao, X\. Su, L\. Qiao, and D\. Li \(2026\)GRASS: gradient\-based adaptive layer\-wise importance sampling for memory\-efficient large language model fine\-tuning\.External Links:2604\.07808,[Link](https://arxiv.org/abs/2604.07808)Cited by:[4th item](https://arxiv.org/html/2606.28707#A1.I3.i4.p1.1)\.
- H\. Wang, C\. Ma, I\. Reid, and M\. Yaqub \(2025\)Kalman filter enhanced grpo for reinforcement learning\-based language model reasoning\.arXiv preprint arXiv:2505\.07527\.Cited by:[§2](https://arxiv.org/html/2606.28707#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. Chen \(2024\)MMLU\-pro: A more robust and challenging multi\-task language understanding benchmark\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2024/hash/ad236edc564f3e3156e1b2feafb99a24-Abstract-Datasets%5C_and%5C_Benchmarks%5C_Track.html)Cited by:[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px2.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§B\.4](https://arxiv.org/html/2606.28707#A2.SS4.p1.1)\.
- R\. J\. Williams \(1992\)Simple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine learning8\(3\),pp\. 229–256\.Cited by:[§2](https://arxiv.org/html/2606.28707#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Wu, Z\. Li, J\. JieZhangChinaTele, Z\. He, J\. Yang, Y\. Zhao, R\. Fang, B\. Wang, H\. Xie, S\. Song,et al\.\(2025a\)UCS\-sql: uniting content and structure for enhanced semantic bridging in text\-to\-sql\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 8156–8168\.Cited by:[§A\.3](https://arxiv.org/html/2606.28707#A1.SS3.SSS0.Px1.p1.3)\.
- Z\. Wu, Z\. Li, M\. Li, J\. Zhang, Z\. He, J\. Yang, Y\. Zhao, R\. Fang, Y\. Li, Z\. Li,et al\.\(2025b\)MR\-sql: multi\-level retrieval enhances inference for llm in text\-to\-sql\.InInternational Conference on Database Systems for Advanced Applications,pp\. 403–413\.Cited by:[§A\.3](https://arxiv.org/html/2606.28707#A1.SS3.SSS0.Px1.p1.3)\.
- Z\. Wu, J\. Yang, J\. Liu, X\. Wu, C\. Pan, J\. Zhang, Y\. Zhao, S\. Song, Y\. Li, and Z\. Li \(2025c\)Table\-r1: region\-based reinforcement learning for table understanding\.arXiv preprint arXiv:2505\.12415\.Cited by:[§A\.3](https://arxiv.org/html/2606.28707#A1.SS3.SSS0.Px1.p1.3)\.
- J\. Yan, Y\. Li, Z\. Hu, Z\. Wang, G\. Cui, X\. Qu, Y\. Cheng, and Y\. Zhang \(2025\)Learning to reason under off\-policy guidance\.arXiv preprint arXiv:2504\.14945\.Cited by:[§1](https://arxiv.org/html/2606.28707#S1.p1.1),[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px3.p1.2)\.
- A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2024a\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[2nd item](https://arxiv.org/html/2606.28707#A1.I3.i2.p1.1)\.
- A\. Yang, B\. Zhang, B\. Hui, B\. Gao, B\. Yu, C\. Li, D\. Liu, J\. Tu, J\. Zhou, J\. Lin, K\. Lu, M\. Xue, R\. Lin, T\. Liu, X\. Ren, and Z\. Zhang \(2024b\)Qwen2\.5\-math technical report: toward mathematical expert model via self\-improvement\.External Links:2409\.12122,[Link](https://arxiv.org/abs/2409.12122)Cited by:[1st item](https://arxiv.org/html/2606.28707#A1.I3.i1.p1.1)\.
- A\. Yang, B\. Zhang, B\. Hui, B\. Gao, B\. Yu, C\. Li, D\. Liu, J\. Tu, J\. Zhou, J\. Lin,et al\.\(2024c\)Qwen2\. 5\-math technical report: toward mathematical expert model via self\-improvement\.arXiv preprint arXiv:2409\.12122\.Cited by:[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px2.p1.1)\.
- Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, T\. Fan, G\. Liu, L\. Liu, X\. Liu,et al\.\(2025a\)DAPO: an open\-source llm reinforcement learning system at scale\.arXiv preprint arXiv:2503\.14476\.Cited by:[§1](https://arxiv.org/html/2606.28707#S1.p3.1),[§2](https://arxiv.org/html/2606.28707#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Yu, R\. Peng, K\. Ding, Y\. Li, Z\. Peng, M\. Liu, Y\. Zhang, Z\. Yuan, H\. Xin, W\. Huang,et al\.\(2025b\)Formalmath: benchmarking formal mathematical reasoning of large language models\.arXiv preprint arXiv:2505\.02735\.Cited by:[§1](https://arxiv.org/html/2606.28707#S1.p1.1)\.
- W\. Zeng, Y\. Huang, Q\. Liu, W\. Liu, K\. He, Z\. Ma, and J\. He \(2025\)Simplerl\-zoo: investigating and taming zero reinforcement learning for open base models in the wild\.arXiv preprint arXiv:2503\.18892\.Cited by:[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px3.p1.2)\.
- X\. Zheng, T\. Pang, C\. Du, Q\. Liu, J\. Jiang, and M\. Lin \(2025\)Cheating automatic LLM benchmarks: null models achieve high win rates\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=syThiTmWWm)Cited by:[§4](https://arxiv.org/html/2606.28707#S4.SS0.SSS0.Px1.p1.1)\.
- Y\. Zhou, Y\. Li, D\. Cheng, H\. Fan, and Y\. Cheng \(2026\)Look inward to explore outward: learning temperature policy from llm internal states via hierarchical rl\.arXiv preprint arXiv:2602\.13035\.Cited by:[§A\.3](https://arxiv.org/html/2606.28707#A1.SS3.SSS0.Px1.p1.3)\.
- D\. M\. Ziegler, N\. Stiennon, J\. Wu, T\. B\. Brown, A\. Radford, D\. Amodei, P\. Christiano, and G\. Irving \(2019\)Fine\-tuning language models from human preferences\.arXiv preprint arXiv:1909\.08593\.Cited by:[§1](https://arxiv.org/html/2606.28707#S1.p2.1)\.

Appendix

## Contents

## Appendix AExperimental Setup

This appendix describes the experimental environment, task formulation, datasets, evaluation protocols, and base model configurations\. Our goal is to provide sufficient detail for reproducibility\.

### A\.1Problem Formulation as a Markov Decision Process \(MDP\)

We model autoregressive generation as a finite\-horizon episodic Markov Decision Process \(MDP\)\(𝒮,𝒜,P,r,H\)\(\\mathcal\{S\},\\mathcal\{A\},P,r,H\), whereHHis the maximum number of*generated completion tokens*\(including EOS when present\)\.

- •State Space \(𝒮\\mathcal\{S\}\)\.A state at stepttis the prompt concatenated with the generated prefix:st=\[q;y<t\]=\[q;y1;…;yt−1\]s\_\{t\}=\[q;y\_\{<t\}\]=\[q;y\_\{1\};\\dots;y\_\{t\-1\}\], with initial states1=\[q\]s\_\{1\}=\[q\]\.
- •Action Space \(𝒜\\mathcal\{A\}\)\.The action space is the model vocabulary\. At statests\_\{t\}, the policyπθ\(⋅∣st\)\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\)outputs a distribution over tokens, and we sample an actionat=yta\_\{t\}=y\_\{t\}\.
- •Transition Dynamics \(PP\)\.Transitions are deterministic: after taking actionat=yta\_\{t\}=y\_\{t\}, the next state isst\+1=\[q;y≤t\]=\[st;yt\]s\_\{t\+1\}=\[q;y\_\{\\leq t\}\]=\[s\_\{t\};y\_\{t\}\]\.
- •Reward \(rr\) and Return\.We use sparse, outcome\-based rewards\. A scalar reward is given only at episode termination and is binary: R​\(τ\)=\{1ifMath\-Verify​\(τ\)=True,0otherwise,R\(\\tau\)=\\begin\{cases\}1&\\text\{if \}\\textit\{Math\-Verify\}\(\\tau\)=\\text\{True\},\\\\ 0&\\text\{otherwise\},\\end\{cases\}whereτ=\(y1,…,yT\)\\tau=\(y\_\{1\},\\dots,y\_\{T\}\)denotes the generated completion andMath\-VerifyreturnsTrueif the extracted final answer is judged equivalent to the gold answer under its comparison rules\. No intermediate rewards are provided \(i\.e\.,rt=0r\_\{t\}=0fort<Tt<T\), and the terminal reward equalsrT=R​\(τ\)r\_\{T\}=R\(\\tau\)\. In optimization \(Sec\.[3](https://arxiv.org/html/2606.28707#S3)\), this trajectory\-level reward induces a single per\-trajectory advantage that is applied to completion tokens, with gradients stopped through all reward statistics\.
- •Termination and Horizon \(HH\)\.An episode terminates when either \(i\) the model emits an end\-of\-sequence token \(EOS\), or \(ii\) the number of generated completion tokens reaches the maximum length limitHH\. We setH=8192H=8192in all experiments\.

### A\.2Justification for the Mathematical Reasoning Domain

We select mathematical reasoning as our primary experimental domain for three reasons:

1. 1\.Objective, verifiable rewards\.Mathematical problems admit a well\-defined notion of correctness, enabling programmatic outcome verification against ground\-truth final answers \(e\.g\., by extracting the final answer, normalizing it, and checking equivalence\)\. This reduces reliance on subjective annotations or learned judges that may introduce noise or exploitable biases\.
2. 2\.A natural stress test for prompt\-local normalization under sparse binary rewards\.With sparse binary rewards, difficult prompts often yield rollout groups where most samples share the same outcome \(e\.g\., all incorrect\), making within\-prompt reward dispersion small\. This setting makes it possible to directly study failure cases of prompt\-local normalization methods, including GRPO\-style estimators, where the normalization scale can become \(near\-\)degenerate and standardized advantages weaken the effective learning signal \(Sec\.[3](https://arxiv.org/html/2606.28707#S3)\)\.
3. 3\.Scalable and rigorous evaluation\.Automated verification supports efficient evaluation on large test sets at low cost, improving scalability, consistency, and reproducibility across methods and backbones\.

### A\.3Reward Computation and Visualized Example

##### Reward Computation\.

We compute sparse, outcome\-based rewards usingMath\-Verify\. Given a completed trajectoryτ\\tau,Math\-Verify\(i\) extracts a candidate final answer from the model output using a priority\-based extraction pipeline \(e\.g\., preferring content inside\\boxed\{⋅\\cdot\}when present\), \(ii\) normalizes the extracted text and parses it into a canonical symbolic representation \(e\.g\., via ANTLR\-based parsing andlatex2sympy2\_extended\), and \(iii\) checks equivalence against the gold answer using rule\-based and SymPy\-backed comparisons\(Kydliceket al\.,[2025](https://arxiv.org/html/2606.28707#bib.bib24)\)\. We assign a binary trajectory\-level reward

R​\(τ\)=\{1ifMath\-Verify​\(τ\)=True,0otherwise,R\(\\tau\)=\\begin\{cases\}1&\\text\{if \}\\textit\{Math\-Verify\}\(\\tau\)=\\text\{True\},\\\\ 0&\\text\{otherwise\},\\end\{cases\}where parsing failures or non\-equivalence return0\. Some answer representations receive specialized handling \(e\.g\., relations, sets/intervals\), followingMath\-Verify’s documented comparison rules\(Kydliceket al\.,[2025](https://arxiv.org/html/2606.28707#bib.bib24)\)\. This outcome\-based formulation is standard in settings with verifiable rewards\(Haoet al\.,[2025](https://arxiv.org/html/2606.28707#bib.bib2)\)and is broadly aligned with evaluation practices for large language models that emphasize reliable outcome assessment\(Changet al\.,[2024](https://arxiv.org/html/2606.28707#bib.bib46)\)\. It is also compatible with broader reasoning\-oriented frameworks that rely on trajectory\-level outcomes\(Haoet al\.,[2026](https://arxiv.org/html/2606.28707#bib.bib22); Chenet al\.,[2026](https://arxiv.org/html/2606.28707#bib.bib3)\)\. Similar considerations also arise in structured reasoning tasks such as table understanding and text\-to\-SQL, where intermediate reasoning must ultimately support correct final outputs\(Wuet al\.,[2025c](https://arxiv.org/html/2606.28707#bib.bib19),[a](https://arxiv.org/html/2606.28707#bib.bib20),[b](https://arxiv.org/html/2606.28707#bib.bib21)\)\. More broadly, related retrieval settings also emphasize the importance of aligning intermediate evidence with final decisions\(Liet al\.,[2026a](https://arxiv.org/html/2606.28707#bib.bib4),[b](https://arxiv.org/html/2606.28707#bib.bib16)\)\. Recent work on controllable and explainable reasoning, adaptive exploration, test\-time adaptation, and solution refinement further highlights the value of structured intermediate reasoning for reliable final outcomes\(Donget al\.,[2026](https://arxiv.org/html/2606.28707#bib.bib14); Zhouet al\.,[2026](https://arxiv.org/html/2606.28707#bib.bib13); Shenet al\.,[2026](https://arxiv.org/html/2606.28707#bib.bib6); Jianget al\.,[2026](https://arxiv.org/html/2606.28707#bib.bib15)\)\.

Figure 4:An illustrative example showing a prompt, a successful and a failed generation trajectory, and their corresponding rewards underMath\-Verify\.

### A\.4Base Models and Prompting Format

To support reproducibility, we list all base models and their sources\.

- •Primary Model\.Our main experiments and ablations useQwen2\.5\-Math\-7B\(Yanget al\.,[2024b](https://arxiv.org/html/2606.28707#bib.bib63)\), loaded from the Hugging Face Hub \(Qwen/Qwen2\.5\-Math\-7B\)\.
- •Robustness\-Test Models\.To test robustness across model scales and instruction tuning, we additionally evaluateQwen/Qwen2\.5\-Math\-1\.5B,Qwen/Qwen2\.5\-7B\-Instruct\(Yanget al\.,[2024a](https://arxiv.org/html/2606.28707#bib.bib126)\), andmeta\-llama/Meta\-Llama\-3\.1\-8B\-Instruct\(Team,[2024](https://arxiv.org/html/2606.28707#bib.bib127)\)\. All checkpoints are obtained from the Hugging Face Hub\.
- •Prompt Format\.We use each checkpoint’s tokenizer\-provided chat template to render prompts, viatokenizer\.apply\_chat\_template\(…, add\_generation\_prompt=True\)for both training and evaluation\([Face,](https://arxiv.org/html/2606.28707#bib.bib23)\)\. A typical rendered prompt \(shown here for clarity\) has the following structure: ``` <|im_start|>system You are a helpful assistant.<|im_end|> <|im_start|>user {problem_description}<|im_end|> <|im_start|>assistant ``` where\{problem\_description\}is the problem text from the dataset\.
- •Implementation Note\.Although our experiments use full\-parameter fine\-tuning rather than PEFT, they are motivated by the broader goal of robust adaptation for large models\. This perspective is related to prior work on reducing adaptation overhead in large models\(Tianet al\.,[2026](https://arxiv.org/html/2606.28707#bib.bib18); Donget al\.,[2025b](https://arxiv.org/html/2606.28707#bib.bib5); Changet al\.,[2026](https://arxiv.org/html/2606.28707#bib.bib7),[2025](https://arxiv.org/html/2606.28707#bib.bib8)\)and to continual\-adaptation strategies that aim to preserve useful knowledge under iterative updates\(Chen and Zeng,[2025](https://arxiv.org/html/2606.28707#bib.bib17)\)\.

## Appendix BImplementation Details and Pseudocode

This section describes the implementation of BV\-Blend \(BV\-Blend\), including the historical\-moments buffer, update ordering, and simplified pseudocode for advantage computation and training\.

### B\.1Historical Moments Buffer

BV\-Blend maintains*semantic\-cluster\-conditioned*historical reward moments, rather than per\-prompt \(UID\-level\) statistics\. Let the clustering codebook haveKKclusters \(Sec\.[3\.2\.1](https://arxiv.org/html/2606.28707#S3.SS2.SSS1)\)\. For each clusterk∈\{1,…,K\}k\\in\\\{1,\\dots,K\\\}, we store three EMA statistics: \(i\) the EMA meanm1​\(k\)m\_\{1\}\(k\), \(ii\) the EMA second raw momentm2​\(k\)m\_\{2\}\(k\), and \(iii\) the EMA effective massNkeffN^\{\\mathrm\{eff\}\}\_\{k\}\. In practice, we implement these as dense length\-KKbuffersm1\[K\],m2\[K\],n\_eff\[K\], together with a booleanseen\[K\]flag, which simplifies distributed aggregation and implementation\.

##### Cold\-start initialization and update ordering\.

For each batch, BV\-Blend computes the confidence weightwkw\_\{k\}from the*pre\-update*EMA state \(i\.e\., before incorporating the current batch\)\. If a clusterkkhas not been observed previously \(seen\[k\]=False\), we treat it as cold start: we setwk=0w\_\{k\}=0for this batch \(pure prompt\-local normalization\), and initialize its EMA state*after*the policy update using batch statistics:

Nkeff\\displaystyle N^\{\\mathrm\{eff\}\}\_\{k\}←N0,\\displaystyle\\leftarrow N\_\{0\},\(12\)m1​\(k\)\\displaystyle m\_\{1\}\(k\)←μℬ​\(k\),\\displaystyle\\leftarrow\\mu\_\{\\mathcal\{B\}\}\(k\),m2​\(k\)\\displaystyle m\_\{2\}\(k\)←μℬ​\(k\)2\+Vprior\.\\displaystyle\\leftarrow\\mu\_\{\\mathcal\{B\}\}\(k\)^\{2\}\+V\_\{\\mathrm\{prior\}\}\.For clusters with an existing EMA state, we update moments*post\-update*using aggregated sufficient statistics from the current batch \(Eq\. \([6](https://arxiv.org/html/2606.28707#S3.E6)\)\)\.

##### Distributed aggregation\.

Given batchℬ\\mathcal\{B\}, we compute per\-cluster sufficient statisticsS1,ℬ​\(k\)=∑\(m,i\)∈ℐℬ​\(k\)Ri\(m\)S\_\{1,\\mathcal\{B\}\}\(k\)=\\sum\_\{\(m,i\)\\in\\mathcal\{I\}\_\{\\mathcal\{B\}\}\(k\)\}R\_\{i\}^\{\(m\)\},S2,ℬ​\(k\)=∑\(m,i\)∈ℐℬ​\(k\)\(Ri\(m\)\)2S\_\{2,\\mathcal\{B\}\}\(k\)=\\sum\_\{\(m,i\)\\in\\mathcal\{I\}\_\{\\mathcal\{B\}\}\(k\)\}\(R\_\{i\}^\{\(m\)\}\)^\{2\}, andNℬ​\(k\)=\|ℐℬ​\(k\)\|N\_\{\\mathcal\{B\}\}\(k\)=\|\\mathcal\{I\}\_\{\\mathcal\{B\}\}\(k\)\|\. In distributed training, we all\-reduce \(sum\) these dense buffers across workers before applying EMA updates\.

### B\.2BV\-Blend Training Loop and Advantage Computation

Algorithm[2](https://arxiv.org/html/2606.28707#alg2)shows the overall loop\. Algorithm[1](https://arxiv.org/html/2606.28707#alg1)details BV\-Blend advantage computation, which blends*baseline and variance statistics*and then standardizes once \(Eq\. \([10](https://arxiv.org/html/2606.28707#S3.E10)\)–\([11](https://arxiv.org/html/2606.28707#S3.E11)\)\)\. As in the main text, we stop gradients through all reward statistics, includingμ𝒢\(m\)\\mu\_\{\\mathcal\{G\}\}^\{\(m\)\},σ𝒢\(m\)\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}, and EMA moments\.

Algorithm 1Compute BV\-Blend AdvantagesABVA^\{\\mathrm\{BV\}\}\(Simplified\)0:Prompts

\{q\(m\)\}\\\{q^\{\(m\)\}\\\}, rewards

\{Ri\(m\)\}\\\{R\_\{i\}^\{\(m\)\}\\\}, EMA buffers

\(m1,m2,Neff\)\(m\_\{1\},m\_\{2\},N^\{\\mathrm\{eff\}\}\)and flagsseen, encoder

E​\(⋅\)E\(\\cdot\), codebook

\{cj\}\\\{c\_\{j\}\\\}\.

0:Advantages

\{AiBV,\(m\)\}\\\{A\_\{i\}^\{\\mathrm\{BV\},\(m\)\}\\\}\.

1:foreach prompt

mmdo

2:Compute prompt\-local mean/std

μ𝒢\(m\),σ𝒢\(m\)\\mu\_\{\\mathcal\{G\}\}^\{\(m\)\},\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}from

\{Ri\(m\)\}i=1G\\\{R\_\{i\}^\{\(m\)\}\\\}\_\{i=1\}^\{G\}\.

3:Assign cluster

k\(m\)←arg⁡minj⁡‖E​\(q\(m\)\)−cj‖22k^\{\(m\)\}\\leftarrow\\arg\\min\_\{j\}\\\|E\(q^\{\(m\)\}\)\-c\_\{j\}\\\|\_\{2\}^\{2\}\.

4:ifseen\[k\(m\)k^\{\(m\)\}\]=False\(unseen cluster, pre\-update\)then

5:Set

w\(m\)←0w^\{\(m\)\}\\leftarrow 0\.\(cold start; pure prompt\-local\)

6:Set

b\(m\)←μ𝒢\(m\)b^\{\(m\)\}\\leftarrow\\mu\_\{\\mathcal\{G\}\}^\{\(m\)\},

s\(m\)←σ𝒢\(m\)s^\{\(m\)\}\\leftarrow\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}\.

7:Mark

k\(m\)k^\{\(m\)\}as “needs init” for post\-update initialization\.

8:else

9:

μhist←m1​\(k\(m\)\)\\mu\_\{\\mathrm\{hist\}\}\\leftarrow m\_\{1\}\(k^\{\(m\)\}\);

vhist←max⁡\(m2​\(k\(m\)\)−m1​\(k\(m\)\)2,0\)v\_\{\\mathrm\{hist\}\}\\leftarrow\\max\(m\_\{2\}\(k^\{\(m\)\}\)\-m\_\{1\}\(k^\{\(m\)\}\)^\{2\},0\)\.

10:

SEMhist←vhist/Nk\(m\)eff\+δN\\mathrm\{SEM\}\_\{\\mathrm\{hist\}\}\\leftarrow\\sqrt\{v\_\{\\mathrm\{hist\}\}\}\\big/\\sqrt\{N^\{\\mathrm\{eff\}\}\_\{k^\{\(m\)\}\}\+\\delta\_\{N\}\}\.

11:

w\(m\)←exp⁡\(−SEMhist/T\)w^\{\(m\)\}\\leftarrow\\exp\(\-\\mathrm\{SEM\}\_\{\\mathrm\{hist\}\}/T\)\.

12:Set

b\(m\)\\displaystyle b^\{\(m\)\}←w\(m\)​μhist\+\(1−w\(m\)\)​μ𝒢\(m\),\\displaystyle\\leftarrow w^\{\(m\)\}\\mu\_\{\\mathrm\{hist\}\}\+\(1\-w^\{\(m\)\}\)\\mu\_\{\\mathcal\{G\}\}^\{\(m\)\},s\(m\)\\displaystyle s^\{\(m\)\}←w\(m\)​vhist\+\(1−w\(m\)\)​\(σ𝒢\(m\)\)2\.\\displaystyle\\leftarrow\\sqrt\{w^\{\(m\)\}v\_\{\\mathrm\{hist\}\}\+\(1\-w^\{\(m\)\}\)\(\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}\)^\{2\}\}\.
13:endif

14:foreach trajectory

i=1,…,Gi=1,\\dots,Gdo

15:

AiBV,\(m\)←\(Ri\(m\)−b\(m\)\)/\(s\(m\)\+δ\)A\_\{i\}^\{\\mathrm\{BV\},\(m\)\}\\leftarrow\(R\_\{i\}^\{\(m\)\}\-b^\{\(m\)\}\)/\(s^\{\(m\)\}\+\\delta\)\.

16:endfor

17:endfor

18:return

\{AiBV,\(m\)\}\\\{A\_\{i\}^\{\\mathrm\{BV\},\(m\)\}\\\}\.

Algorithm 2BV\-Blend \(BV\-Blend\) Training Loop \(Simplified\)1:Initializepolicy parameters

θ\\theta, reference policy

πref\\pi\_\{\\mathrm\{ref\}\}, and fixed clustering codebook

\{cj\}j=1K\\\{c\_\{j\}\\\}\_\{j=1\}^\{K\}\.

2:InitializeEMA buffers

m1\[K\],m2\[K\],n\_eff\[K\]\\texttt\{m1\[K\]\},\\texttt\{m2\[K\]\},\\texttt\{n\\\_eff\[K\]\}and flags

seen\[K\]←False\\texttt\{seen\[K\]\}\\leftarrow\\texttt\{False\}\.

3:foriteration

t=1,2,…t=1,2,\\dotsdo

4:Set

θold←θ\\theta\_\{\\mathrm\{old\}\}\\leftarrow\\theta\.\(behavior snapshot\)

5:Sample

MMprompts

\{q\(m\)\}m=1M\\\{q^\{\(m\)\}\\\}\_\{m=1\}^\{M\}; for each prompt, sample

GGtrajectories

\{τi\(m\)\}i=1G\\\{\\tau\_\{i\}^\{\(m\)\}\\\}\_\{i=1\}^\{G\}from

πθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\.

6:Compute verifier rewards

Ri\(m\)R\_\{i\}^\{\(m\)\}for all trajectories\.

7:Compute BV\-Blend advantages

AiBV,\(m\)A\_\{i\}^\{\\mathrm\{BV\},\(m\)\}using Algorithm[1](https://arxiv.org/html/2606.28707#alg1)\(pre\-update EMA\)\.

8:Update

θ\\thetaby optimizing the PPO\-style clipped objective using

AiBV,\(m\)A\_\{i\}^\{\\mathrm\{BV\},\(m\)\}\(and optional KL/entropy terms\)\.

9:Post\-update:aggregate per\-cluster sufficient statistics

\{S1,ℬ​\(k\),S2,ℬ​\(k\),Nℬ​\(k\)\}k=1K\\\{S\_\{1,\\mathcal\{B\}\}\(k\),S\_\{2,\\mathcal\{B\}\}\(k\),N\_\{\\mathcal\{B\}\}\(k\)\\\}\_\{k=1\}^\{K\}across workers\.

10:Post\-update:update EMA buffers for clusters with

Nℬ​\(k\)\>0N\_\{\\mathcal\{B\}\}\(k\)\>0using Eq\. \([6](https://arxiv.org/html/2606.28707#S3.E6)\); for previously unseen clusters, initialize using

\(N0,Vprior\)\(N\_\{0\},V\_\{\\mathrm\{prior\}\}\)and set

seen\[k\]←True\\texttt\{seen\[k\]\}\\leftarrow\\texttt\{True\}\.

11:endfor

### B\.3Model and Optimization

##### Model Architecture\.

We perform full\-parameter fine\-tuning on all base models\. We do not use parameter\-efficient fine\-tuning \(PEFT\) methods such as LoRA\(Huet al\.,[2022](https://arxiv.org/html/2606.28707#bib.bib129)\), and we do not add adapters or custom heads; the transformer architecture remains unchanged\.

##### Optimization Objective\.

During each policy update, we optimize a PPO\-style clipped surrogate objective\(Schulmanet al\.,[2017](https://arxiv.org/html/2606.28707#bib.bib95)\)using the BV\-Blend advantagesAiBV,\(m\)A\_\{i\}^\{\\mathrm\{BV\},\(m\)\}\(Sec\.[3](https://arxiv.org/html/2606.28707#S3)\)\. Concretely, we maximize

𝒥​\(θ\)\\displaystyle\\mathcal\{J\}\(\\theta\)=𝔼m\[1G∑i=1G𝔼t∼τi\(m\)\[min\(ri,t\(m\)\(θ\)AiBV,\(m\),\\displaystyle=\\mathbb\{E\}\_\{m\}\\\!\\Bigg\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\mathbb\{E\}\_\{t\\sim\\tau^\{\(m\)\}\_\{i\}\}\\\!\\Big\[\\min\\Big\(r^\{\(m\)\}\_\{i,t\}\(\\theta\)\\,A^\{\\mathrm\{BV\},\(m\)\}\_\{i\},r~i,t\(m\)\(θ\)AiBV,\(m\)\)\]\]\\displaystyle\\qquad\\qquad\\qquad\\qquad\\qquad\\tilde\{r\}^\{\(m\)\}\_\{i,t\}\(\\theta\)\\,A^\{\\mathrm\{BV\},\(m\)\}\_\{i\}\\Big\)\\Big\]\\Bigg\]−β𝔼m,i\[𝔼t∼τi\(m\)\[KL\(πθ\(⋅∣st\)∥\\displaystyle\\quad\-\\beta\\;\\mathbb\{E\}\_\{m,i\}\\\!\\Big\[\\mathbb\{E\}\_\{t\\sim\\tau^\{\(m\)\}\_\{i\}\}\\\!\\big\[\\mathrm\{KL\}\\\!\\Bigl\(\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\)\\,\\Big\\\|\\,πref\(⋅∣st\)\)\]\]\\displaystyle\\qquad\\qquad\\qquad\\qquad\\qquad\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid s\_\{t\}\)\\Bigr\)\\big\]\\Big\]\+λent𝔼m,i\[𝔼t∼τi\(m\)\[ℋ\(πθ\(⋅∣st\)\)\]\],\\displaystyle\\quad\+\\lambda\_\{\\mathrm\{ent\}\}\\;\\mathbb\{E\}\_\{m,i\}\\\!\\Big\[\\mathbb\{E\}\_\{t\\sim\\tau^\{\(m\)\}\_\{i\}\}\\\!\\big\[\\mathcal\{H\}\\\!\\left\(\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\)\\right\)\\big\]\\Big\],\(13\)whereri,t\(m\)​\(θ\)r^\{\(m\)\}\_\{i,t\}\(\\theta\)andr~i,t\(m\)​\(θ\)\\tilde\{r\}^\{\(m\)\}\_\{i,t\}\(\\theta\)are defined in Eq\. \([1](https://arxiv.org/html/2606.28707#S3.E1)\), and𝔼t∼τi\(m\)​\[⋅\]\\mathbb\{E\}\_\{t\\sim\\tau^\{\(m\)\}\_\{i\}\}\[\\cdot\]denotes the completion\-token mean with the same masking convention as Eq\. \([2](https://arxiv.org/html/2606.28707#S3.E2)\)\. We implement training by minimizing the lossℒ​\(θ\)=−𝒥​\(θ\)\\mathcal\{L\}\(\\theta\)=\-\\mathcal\{J\}\(\\theta\)\. Unless otherwise specified, we setλent=0\.01\\lambda\_\{\\mathrm\{ent\}\}=0\.01and use a fixed reference policyπref\\pi\_\{\\mathrm\{ref\}\}; in our main comparisons, we setβ=0\\beta=0\.

### B\.4Prompting Strategy

We format inputs using each model’s tokenizer\-provided chat template viatokenizer\.apply\_chat\_template\(…, add\_generation\_prompt=True\)for both training and evaluation, which improves reproducibility and avoids formatting mismatches across backbones\([Face,](https://arxiv.org/html/2606.28707#bib.bib23)\)\. Unless otherwise stated, we use a generic assistant system message and provide the problem statement as the user message\. We do not require additional task\-specific instructions for Qwen\-based math models; they typically produce multi\-step solutions by default\. For Llama\-3\.1\-Instruct in the robustness experiments, we optionally prepend a short reasoning cue \(e\.g\., “Let’s think step by step\.”\) to encourage step\-by\-step solutions\(Weiet al\.,[2022](https://arxiv.org/html/2606.28707#bib.bib130)\)\.

Rendered prompt for Qwen\-based chat models \(schematic\)``` <|im_start|>system You are a helpful assistant.<|im_end|> <|im_start|>user {problem_description}<|im_end|> <|im_start|>assistant ```

Rendered prompt for Llama\-3\.1\-Instruct \(schematic\)User:\{problem\_description\}Assistant:Let’s think step by step\.

##### Notation\.

For convenience, Table[3](https://arxiv.org/html/2606.28707#A2.T3)summarizes the key symbols used throughout the paper, consistent with Sec\.[3](https://arxiv.org/html/2606.28707#S3)\. We group notation into \(i\) trajectories and PPO\-style optimization, \(ii\) prompt\-local \(GRPO\-style\) reward statistics, and \(iii\) BV\-Blend’s semantic clustering, historical EMA moments, and SEM\-based confidence weighting\. Unless otherwise stated, prompts are indexed bymm, trajectories byii, completion\-token positions bytt, clusters bykk, and codebook entries byjj\. All reward/statistics terms used to form advantages \(e\.g\.,μ𝒢\(m\)\\mu\_\{\\mathcal\{G\}\}^\{\(m\)\},σ𝒢\(m\)\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}, and statistics derived from historical EMA moments\) are treated as stop\-gradient constants\.

Table 3:Summary of key symbols \(aligned with Sec\.[3](https://arxiv.org/html/2606.28707#S3)\)\.

## Appendix CExtended Experimental Results

This section presents supplementary quantitative results and ablations that complement the main paper\. Throughout, we compare against the on\-policy GRPO baseline under the same PPO\-style clipped objective and vary only the advantage estimator\.

### C\.1Further Ablation Studies

##### Confidence\-to\-weight mapping\.

We ablate the functional form that maps historical uncertainty to the confidence weightwkw\_\{k\}\. All variants compute the same SEM\-style uncertaintySEMhist​\(k\)\\mathrm\{SEM\}\_\{\\mathrm\{hist\}\}\(k\)from*pre\-update*EMA statistics \(Eq\. \([8](https://arxiv.org/html/2606.28707#S3.E8)\)\) and use the same baseline/variance blending \(Eq\. \([10](https://arxiv.org/html/2606.28707#S3.E10)\)\); they differ only in the mappingwk=g​\(uk\)w\_\{k\}=g\(u\_\{k\}\), whereuk≜SEMhist​\(k\)/Tu\_\{k\}\\triangleq\\mathrm\{SEM\}\_\{\\mathrm\{hist\}\}\(k\)/T\. Table[4](https://arxiv.org/html/2606.28707#A3.T4)reports results on AIME 2024 \(avg@32\)\. Among the tested monotone mappings, the exponential formwk=exp⁡\(−uk\)w\_\{k\}=\\exp\(\-u\_\{k\}\)performs best, which is consistent with the use of a smooth confidence schedule rather than a hard threshold\.

Table 4:Ablation on the confidence mappingwk=g​\(uk\)w\_\{k\}=g\(u\_\{k\}\)on AIME 2024 \(avg@32\), whereuk=SEMhist​\(k\)/Tu\_\{k\}=\\mathrm\{SEM\}\_\{\\mathrm\{hist\}\}\(k\)/T\.![Refer to caption](https://arxiv.org/html/2606.28707v1/x7.png)Figure 5:Cross\-backbone robustness\.Performance of BV\-Blend relative to baselines across diverse model backbones under the same RLVR setup\.

### C\.2Robustness Across Diverse Models

To assess robustness beyond our primary Qwen2\.5\-Math\-7B setting, we evaluate BV\-Blend on three additional backbones: Qwen2\.5\-Math\-1\.5B, Qwen2\.5\-7B\-Instruct, and LLaMA\-3\.1\-8B\. As shown in Fig\.[5](https://arxiv.org/html/2606.28707#A3.F5), BV\-Blend improves over the best baseline included in our comparison on all three settings: \+2\.1 points on Qwen2\.5\-Math\-1\.5B and \+1\.6 points on Qwen2\.5\-7B\-Instruct\. The most challenging case is LLaMA\-3\.1\-8B, where standard prompt\-local estimators exhibit pronounced instability under our RLVR setup and can substantially degrade final performance; in this regime, BV\-Blend reaches 19\.9, improving by 2\.8 points over the best\-performing baseline\. While absolute performance on LLaMA\-3\.1\-8B remains modest under this setup, the relative trend is still informative: BV\-Blend reduces degradation and remains more stable than prompt\-local normalization when within\-group reward dispersion is small\. Overall, these results suggest that BV\-Blend is a practical replacement for prompt\-local advantage normalization in on\-policy RLVR across heterogeneous backbones and training conditions\.

### C\.3Proof that GRPO Advantages Vanish Under Uniform Prompt\-Group Rewards

We show that the GRPO\-style prompt\-local standardized advantage is exactly zero when all rollouts within a prompt group receive the same reward\.

###### Theorem 1\(Prompt\-local collapse in GRPO\)\.

Fix a promptq\(m\)q^\{\(m\)\}with rollout set𝒢​\(q\(m\)\)=\{τi\(m\)\}i=1G\\mathcal\{G\}\(q^\{\(m\)\}\)=\\\{\\tau\_\{i\}^\{\(m\)\}\\\}\_\{i=1\}^\{G\}\. IfRi\(m\)=cR\_\{i\}^\{\(m\)\}=cfor alli∈\{1,…,G\}i\\in\\\{1,\\dots,G\\\}and some constantcc, thenAiGRPO,\(m\)=0A\_\{i\}^\{\\mathrm\{GRPO\},\(m\)\}=0for allii\.

###### Proof\.

By Eq\. \([3](https://arxiv.org/html/2606.28707#S3.E3)\),

AiGRPO,\(m\)=Ri\(m\)−μ𝒢\(m\)σ𝒢\(m\)\+δ,A\_\{i\}^\{\\mathrm\{GRPO\},\(m\)\}=\\frac\{R\_\{i\}^\{\(m\)\}\-\\mu\_\{\\mathcal\{G\}\}^\{\(m\)\}\}\{\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}\+\\delta\},whereμ𝒢\(m\)\\mu\_\{\\mathcal\{G\}\}^\{\(m\)\}andσ𝒢\(m\)\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}are the mean and standard deviation of\{Ri\(m\)\}i=1G\\\{R\_\{i\}^\{\(m\)\}\\\}\_\{i=1\}^\{G\}, andδ\>0\\delta\>0\. IfRi\(m\)=cR\_\{i\}^\{\(m\)\}=cfor allii, thenμ𝒢\(m\)=c\\mu\_\{\\mathcal\{G\}\}^\{\(m\)\}=candσ𝒢\(m\)=0\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}=0\. Substituting yields

AiGRPO,\(m\)=c−c0\+δ=0,A\_\{i\}^\{\\mathrm\{GRPO\},\(m\)\}=\\frac\{c\-c\}\{0\+\\delta\}=0,for allii\. ∎

### C\.4Difficulty Stratification and Subset Definition

For the analysis in Fig\. 3, we assign each prompt to one of four fixed difficulty buckets \(Easy, Medium, Hard, Hardest\) using a pre\-RL difficulty estimate computed before policy optimization\. The same bucket assignment is shared across all compared methods and kept fixed throughout training and evaluation\. We further partition the evaluation prompts intoStandardandComplexsubsets using the same precomputed protocol, and report checkpoint\-wise verifier accuracy and average completion length separately for the two subsets\. This analysis is intended only to provide a consistent difficulty\-stratified comparison between BV\-Blend and GRPO; it does not affect training, sampling, or model selection\.

## Appendix DBias–Variance Analysis of the BV\-Blend Advantage Estimator

This section analyzes the statistical behavior of BV\-Blend’s advantage construction\. Rather than mixing two advantage estimators directly, BV\-Blend blends a*baseline*and*variance statistics*\(which determine the scale\) and then performs a single standardization:

AiBV,\(m\)=Ri\(m\)−b\(m\)s\(m\)\+δ,A\_\{i\}^\{\\mathrm\{BV\},\(m\)\}\\;=\\;\\frac\{R\_\{i\}^\{\(m\)\}\-b^\{\(m\)\}\}\{s^\{\(m\)\}\+\\delta\},\(14\)where\(b\(m\),s\(m\)\)\(b^\{\(m\)\},s^\{\(m\)\}\)are defined by Eq\. \([10](https://arxiv.org/html/2606.28707#S3.E10)\)\. Throughout, we treat the confidence weightwkw\_\{k\}as fixed within the current batch because it is computed from*pre\-update*EMA moments \(Sec\.[3\.2\.3](https://arxiv.org/html/2606.28707#S3.SS2.SSS3)\) and gradients are stopped through all reward statistics\.

##### Conditional mean and variance \(givenb\(m\),s\(m\)b^\{\(m\)\},s^\{\(m\)\}\)\.

Conditioned on the \(stop\-gradient\) statistics\(b\(m\),s\(m\)\)\(b^\{\(m\)\},s^\{\(m\)\}\), BV\-Blend is an affine transform of the trajectory reward:

𝔼​\[AiBV,\(m\)∣b\(m\),s\(m\)\]\\displaystyle\\mathbb\{E\}\\\!\\left\[A\_\{i\}^\{\\mathrm\{BV\},\(m\)\}\\mid b^\{\(m\)\},s^\{\(m\)\}\\right\]=𝔼​\[Ri\(m\)\]−b\(m\)s\(m\)\+δ,\\displaystyle=\\frac\{\\mathbb\{E\}\[R\_\{i\}^\{\(m\)\}\]\-b^\{\(m\)\}\}\{s^\{\(m\)\}\+\\delta\},\(15\)Var​\(AiBV,\(m\)∣b\(m\),s\(m\)\)\\displaystyle\\mathrm\{Var\}\\\!\\left\(A\_\{i\}^\{\\mathrm\{BV\},\(m\)\}\\mid b^\{\(m\)\},s^\{\(m\)\}\\right\)=Var​\(Ri\(m\)\)\(s\(m\)\+δ\)2\.\\displaystyle=\\frac\{\\mathrm\{Var\}\(R\_\{i\}^\{\(m\)\}\)\}\{\(s^\{\(m\)\}\+\\delta\)^\{2\}\}\.\(16\)Thus,s\(m\)s^\{\(m\)\}directly controls the magnitude of the learning signal entering the PPO\-style surrogate objective\.

##### Why BV\-Blend can mitigate prompt\-local collapse\.

Under GRPO\-style prompt\-local normalization,s\(m\)=σ𝒢\(m\)s^\{\(m\)\}=\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}\. If rewards in a prompt group are constant \(all\-correct or all\-incorrect\), thenσ𝒢\(m\)=0\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}=0and the standardized advantages are exactly zero \(Theorem[1](https://arxiv.org/html/2606.28707#Thmtheorem1)\), eliminating the learning signal for that group\.

In BV\-Blend, the blended scale satisfies

\(s\(m\)\)2=w\(m\)​vhist​\(k\(m\)\)\+\(1−w\(m\)\)​\(σ𝒢\(m\)\)2\.\(s^\{\(m\)\}\)^\{2\}=w^\{\(m\)\}v\_\{\\mathrm\{hist\}\}\(k^\{\(m\)\}\)\+\(1\-w^\{\(m\)\}\)\(\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}\)^\{2\}\.Therefore, whenσ𝒢\(m\)≈0\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}\\approx 0, the scale remains non\-degenerate wheneverw\(m\)\>0w^\{\(m\)\}\>0andvhist​\(k\(m\)\)\>0v\_\{\\mathrm\{hist\}\}\(k^\{\(m\)\}\)\>0\. If a prompt group has constant rewardRi\(m\)=cR\_\{i\}^\{\(m\)\}=cfor allii, then

b\(m\)\\displaystyle b^\{\(m\)\}=\(1−w\(m\)\)​c\+w\(m\)​μhist​\(k\(m\)\),\\displaystyle=\(1\-w^\{\(m\)\}\)c\+w^\{\(m\)\}\\mu\_\{\\mathrm\{hist\}\}\(k^\{\(m\)\}\),s\(m\)\\displaystyle s^\{\(m\)\}=w\(m\)​vhist​\(k\(m\)\),\\displaystyle=\\sqrt\{w^\{\(m\)\}v\_\{\\mathrm\{hist\}\}\(k^\{\(m\)\}\)\},which yields

AiBV,\(m\)=w\(m\)​\(c−μhist​\(k\(m\)\)\)w\(m\)​vhist​\(k\(m\)\)\+δ\.A\_\{i\}^\{\\mathrm\{BV\},\(m\)\}=\\frac\{w^\{\(m\)\}\\bigl\(c\-\\mu\_\{\\mathrm\{hist\}\}\(k^\{\(m\)\}\)\\bigr\)\}\{\\sqrt\{w^\{\(m\)\}v\_\{\\mathrm\{hist\}\}\(k^\{\(m\)\}\)\}\+\\delta\}\.\(17\)Thus, BV\-Blend can retain a non\-zero advantage even when prompt\-local dispersion vanishes, providedw\(m\)\>0w^\{\(m\)\}\>0andvhist​\(k\(m\)\)\>0v\_\{\\mathrm\{hist\}\}\(k^\{\(m\)\}\)\>0; conversely, ifw\(m\)=0w^\{\(m\)\}=0\(e\.g\., an unseen cluster\) orvhist​\(k\(m\)\)=0v\_\{\\mathrm\{hist\}\}\(k^\{\(m\)\}\)=0, the advantage may still collapse for that batch\.

##### A bias–variance viewpoint via moment shrinkage\.

BV\-Blend can be viewed as constructing*shrunk*estimators of the reward mean and variance \(within a semantic cluster\), and then standardizing once\. Letμ⋆​\(k\)\\mu\_\{\\star\}\(k\)andv⋆​\(k\)v\_\{\\star\}\(k\)denote the \(hypothetical\) population moments for clusterkkunder the current policy, and letμ𝒢\(m\)\\mu\_\{\\mathcal\{G\}\}^\{\(m\)\}and\(σ𝒢\(m\)\)2\(\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}\)^\{2\}be the prompt\-local sample moments computed fromGGrollouts\. BV\-Blend uses

b\(m\)\\displaystyle b^\{\(m\)\}=w\(m\)​μhist​\(k\(m\)\)\+\(1−w\(m\)\)​μ𝒢\(m\),\\displaystyle=w^\{\(m\)\}\\mu\_\{\\mathrm\{hist\}\}\(k^\{\(m\)\}\)\+\(1\-w^\{\(m\)\}\)\\mu\_\{\\mathcal\{G\}\}^\{\(m\)\},\(18\)\(s\(m\)\)2\\displaystyle\(s^\{\(m\)\}\)^\{2\}=w\(m\)​vhist​\(k\(m\)\)\+\(1−w\(m\)\)​\(σ𝒢\(m\)\)2,\\displaystyle=w^\{\(m\)\}v\_\{\\mathrm\{hist\}\}\(k^\{\(m\)\}\)\+\(1\-w^\{\(m\)\}\)\(\\sigma\_\{\\mathcal\{G\}\}^\{\(m\)\}\)^\{2\},\(19\)i\.e\., convex combinations of a low\-variance historical estimate \(aggregating many past samples\) and a high\-variance prompt\-local estimate \(based onGGrollouts\)\. For intuition, under the approximation that \(i\)w\(m\)w^\{\(m\)\}is fixed within the batch and \(ii\) the prompt\-local moments are approximately unbiased for\(μ⋆,v⋆\)\(\\mu\_\{\\star\},v\_\{\\star\}\)up to standard finite\-sample effects, the bias ofb\(m\)b^\{\(m\)\}inherits the historical bias scaled byw\(m\)w^\{\(m\)\}:

Bias​\(b\(m\)\)≈w\(m\)​Bias​\(μhist​\(k\(m\)\)\)\.\\mathrm\{Bias\}\\\!\\left\(b^\{\(m\)\}\\right\)\\approx w^\{\(m\)\}\\,\\mathrm\{Bias\}\\\!\\left\(\\mu\_\{\\mathrm\{hist\}\}\(k^\{\(m\)\}\)\\right\)\.\(20\)Similarly, the variance ofb\(m\)b^\{\(m\)\}is reduced relative to the prompt\-local estimator when the historical estimate is substantially more certain:

Var​\(b\(m\)\)\\displaystyle\\mathrm\{Var\}\\\!\\left\(b^\{\(m\)\}\\right\)≈\(1−w\(m\)\)2​Var​\(μ𝒢\(m\)\)\\displaystyle\\approx\(1\-w^\{\(m\)\}\)^\{2\}\\mathrm\{Var\}\\\!\\left\(\\mu\_\{\\mathcal\{G\}\}^\{\(m\)\}\\right\)\(21\)\+\(w\(m\)\)2​Var​\(μhist​\(k\(m\)\)\),\\displaystyle\\quad\+\(w^\{\(m\)\}\)^\{2\}\\mathrm\{Var\}\\\!\\left\(\\mu\_\{\\mathrm\{hist\}\}\(k^\{\(m\)\}\)\\right\),where we typically expect the cross\-covariance to be small becauseμhist\\mu\_\{\\mathrm\{hist\}\}is computed from past batches\. These relations are intended as explanatory approximations; they ignore estimation error inw\(m\)w^\{\(m\)\}ands\(m\)s^\{\(m\)\}, as well as the additional nonlinearity introduced by the square root and the final normalization\.

##### Why SEM\-based weighting is a useful proxy\.

BV\-Blend setswkw\_\{k\}as a monotone function of the SEM\-style uncertainty proxySEMhist​\(k\)=σhist​\(k\)/Nkeff\+δN\\mathrm\{SEM\}\_\{\\mathrm\{hist\}\}\(k\)=\\sigma\_\{\\mathrm\{hist\}\}\(k\)/\\sqrt\{N\_\{k\}^\{\\mathrm\{eff\}\}\+\\delta\_\{N\}\}\(Eq\. \([8](https://arxiv.org/html/2606.28707#S3.E8)\)\), matching the classical scaling of the standard error of the sample mean with the standard deviation and sample size\. This makeswkw\_\{k\}large when historical estimates are both low\-variance and supported by a large effective count, and small otherwise, thereby adapting the shrinkage strength to estimated uncertainty\.

### D\.1Discussion of Key Assumptions

The analysis above is intended as an explanatory approximation and relies on standard assumptions:

- •Stop\-gradient statistics\.We treat\(b\(m\),s\(m\),wk\)\(b^\{\(m\)\},s^\{\(m\)\},w\_\{k\}\)as fixed w\.r\.t\. policy gradients, consistent with our implementation \(Sec\.[3](https://arxiv.org/html/2606.28707#S3)\)\.
- •Approximate independence across batches\.Historical EMA moments are computed from past batches, so their covariance with current prompt\-local moments is often small, though not exactly zero\.
- •Limited non\-stationarity\.Historical moments are most informative when the reward distribution within a semantic cluster does not shift arbitrarily fast; the SEM\-based weight is designed to reduce reliance on history when uncertainty is high\.
- •No strict unbiasedness claim\.Because BV\-Blend uses normalization and heuristic shrinkage \(and PPO further clips the objective\), we do not claim that the resulting standardized quantity is a strictly unbiased estimator of the true MDP advantage\.

## Appendix EReproducibility Details

All experiments were conducted on a single multi\-GPU node\. We report the main hardware and software stack to facilitate reproduction\.

##### Hardware\.

Experiments were conducted on one node equipped with 8 NVIDIA RTX PRO 6000 GPUs \(96 GB memory per GPU\)\. Training uses PyTorch Fully Sharded Data Parallel \(FSDP\) across all 8 GPUs\.

##### Software\.

Our implementation is based on PyTorch 2\.4\.0 and the Hugging Face Transformers/Accelerate stack\. For generation, we use vLLM \(v0\.6\.3\) with CUDA 12\.1\. We additionally use FlashAttention \(v2\.7\.3\), TensorDict \(v0\.5\.0\), and theverllibrary for RL orchestration\. All experiments are conducted in a containerized CUDA 12\.1 environment\.