GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models

# GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models
Source: [https://arxiv.org/html/2608.17411](https://arxiv.org/html/2608.17411)
Jianqi ZhangXingyu ZhangYun FanJiahuan ZhouChangwen ZhengWenwen Qiang\\corresponding

###### Abstract

Group Relative Policy Optimization \(GRPO\) has become a widely used approach for post\-training Large Language Models \(LLMs\) for reasoning\. In GRPO, the group gradients induced by different queries within the same mini\-batch are directly averaged to form the policy update\. However, these group gradients can point in conflicting directions\. Our empirical analysis suggests that group\-gradient conflicts tend to be associated with less effective policy updates, motivating the need for a reliable aggregated update direction under such conflicts\. Standard GRPO aggregation treats the realized group gradients as deterministic contributions and does not account for differences in their reliability during aggregation\. To address this issue, we propose Gradient Uncertainty\-Aware Policy Optimization \(GUPO\), which models each group gradient as a random variable under a Bayesian formulation and estimates its probability distribution\. GUPO then derives gradient uncertainty using a Dirichlet\-based formulation and uses it to calibrate the contribution of each group gradient during aggregation\. Extensive experiments on multiple benchmarks demonstrate the effectiveness of GUPO\.

## Introduction

Large language models \(LLMs\) have demonstrated strong capabilities in language understanding, text generation, and complex reasoning\([7](https://arxiv.org/html/2608.17411#bib.bib2);[9](https://arxiv.org/html/2608.17411#bib.bib3);[17](https://arxiv.org/html/2608.17411#bib.bib5);[29](https://arxiv.org/html/2608.17411#bib.bib4)\)\. Reinforcement\-learning\-based post\-training, particularly Group Relative Policy Optimization \(GRPO\)\([27](https://arxiv.org/html/2608.17411#bib.bib1)\), has become an important approach for further improving the reasoning capabilities of LLMs\. For each query, GRPO samples a group of responses, derives group\-relative advantages from their rewards, and uses these advantages to weight token\-level policy\-gradient terms\. The resulting gradient contributions are aggregated to update the policy parameters\.

![Refer to caption](https://arxiv.org/html/2608.17411v1/conflict_explanation-v5.png)Figure 1:Explanation of gradient uncertainty and conflicts\. Two\-dimensional example of two group gradients from the same GRPO mini\-batch\.![Refer to caption](https://arxiv.org/html/2608.17411v1/motivation_low_conflict_heatmap.png)\(a\)Low\-Conflict Mini\-batch
![Refer to caption](https://arxiv.org/html/2608.17411v1/motivation_high_conflict_heatmap.png)\(b\)High\-Conflict Mini\-batch
\(c\)Conflict Distribution\(d\)Conflict v\.s\. Performance
Figure 2:Empirical results of group\-gradient conflicts in GRPO\. \(a\) and \(b\) show the pairwise cosine\-similarity matrices of representative mini\-batches with low and high conflict levels\. \(c\) presents the distribution of pairwise cosine similarities among group gradients\. \(d\) compares the validationΔ​NLL\\Delta\\mathrm\{NLL\}across mini\-batches with low, medium, and high conflict rates\.In GRPO, for each query, the gradients induced by its sampled response group are aggregated into agroup gradient, which represents the contribution of that query group to the mini\-batch update\. The mini\-batch gradient is then obtained by averaging the group gradients across queries, as illustrated inFigure[3](https://arxiv.org/html/2608.17411#Sx1.F3)\([12](https://arxiv.org/html/2608.17411#bib.bib15);[23](https://arxiv.org/html/2608.17411#bib.bib16);[22](https://arxiv.org/html/2608.17411#bib.bib17);[5](https://arxiv.org/html/2608.17411#bib.bib18)\)\. Based on this update mechanism, we observe an interesting phenomenon: group gradients induced by different queries within the same mini\-batch may point in conflicting directions\. Specifically, for each mini\-batch, we compute the group gradients and quantify their pairwise directional conflicts\. We then analyze the association between the gradient conflict and the effectiveness of the corresponding policy update\. As shown inFigure[2](https://arxiv.org/html/2608.17411#Sx1.F2), group gradients are largely aligned in some mini\-batches, whereas clear directional conflicts emerge in others\. Mini batches with severe conflicts tend to exhibit limited validation update gains\. They motivate the central question of this work:when group gradients induced by different queries conflict within the same GRPO mini\-batch, how can we obtain a reliable aggregated update direction?

In this context, a reliable update direction can be understood as one that reasonably reflects the overall optimization tendency supported by the mini\-batch, rather than being overly influenced by a few conflicting gradients\. Gradient uncertainty provides a possible basis for assessing such reliability of different group gradients during aggregation\([13](https://arxiv.org/html/2608.17411#bib.bib32);[1](https://arxiv.org/html/2608.17411#bib.bib30)\)\. Specifically, group gradients can be treated as random variables and represented as probability distributions that describe the range of possible gradient values and their likelihoods\. If the distribution is concentrated, most of its probability mass lies within a narrow range around the mean, indicating that the gradient values supported by the distribution are similar\. Such limited variation among these values corresponds to lower gradient uncertainty and higher reliability\. If the distribution is more dispersed, its probability mass extends over a broader range of substantially different gradient values\. This larger variation among these values corresponds to higher gradient uncertainty and lower reliability\. Figure[1](https://arxiv.org/html/2608.17411#Sx1.F1)presents a two\-dimensional example for illustration\.𝐠1\\mathbf\{g\}\_\{1\}and𝐠2\\mathbf\{g\}\_\{2\}represent the mean gradients of two query groups, while the surrounding contours represent their probability distributions\. The distribution of𝐠1\\mathbf\{g\}\_\{1\}concentrates more probability mass near the mean gradient, corresponding to lower uncertainty and higher reliability\. By contrast, the distribution of𝐠2\\mathbf\{g\}\_\{2\}covers a broader range of gradient values, corresponding to higher uncertainty and lower reliability\. Direct aggregation assigns the two group gradients equal coefficients without accounting for this difference in reliability\. Consequently, the more uncertain group gradient may exert excessive influence on the directly aggregated direction𝐠da\\mathbf\{g\}\_\{\\mathrm\{da\}\}\. Based on this analysis, we propose to explicitly model group\-gradient uncertainty and incorporate it into gradient aggregation, emphasizing more reliable group gradients while attenuating the influence of less reliable ones, with the aim of obtaining a reliable update direction when gradient conflicts occur\.

To this end, we propose Gradient Uncertainty\-aware Policy Optimization \(GUPO\), explicitly modeling the uncertainty of group gradients within each mini\-batch to achieve reliable policy updates\. Rather than treating each group gradient solely as a deterministic vector, we model it as a random variable represented by a probability distribution\. Specifically, under a Bayesian formulation\([11](https://arxiv.org/html/2608.17411#bib.bib20);[6](https://arxiv.org/html/2608.17411#bib.bib19)\), we first estimate the probability distribution of each group gradient\. The precision of the resulting distribution is then mapped to gradient evidence\. Following the relationship among evidence, belief, and uncertainty in subjective logic\([16](https://arxiv.org/html/2608.17411#bib.bib21);[4](https://arxiv.org/html/2608.17411#bib.bib22);[26](https://arxiv.org/html/2608.17411#bib.bib23)\), we derive dimension\-wise beliefs and the overall uncertainty of each group gradient through a Dirichlet\-based evidential formulation\. The resulting group\-level uncertainty is incorporated into gradient aggregation, increasing the relative contribution of lower\-uncertainty group gradients while attenuating that of higher\-uncertainty ones\. In this way, GUPO aims to mitigate the excessive influence of highly uncertain gradients and obtain a more reliable aggregated update direction when group\-gradient conflicts occur\. Experiments across multiple benchmarks suggest the effectiveness of our method\.

The main contributions can be summarized as follows: \(i\) We identify directional conflicts among group gradients induced by different queries within the same GRPO mini\-batch, and empirically show that conflicts can be associated with less effective policy updates, motivating the problem of obtaining a reliable aggregated update direction\. \(ii\) We analyze this problem from the perspective of gradient uncertainty and propose GUPO, which models each group gradient as a probability distribution under a Bayesian formulation, derives its uncertainty through a Dirichlet\-based evidential formulation, and incorporates the resulting uncertainty into gradient aggregation\. \(iii\) Extensive experiments across multiple benchmarks demonstrate the effectiveness of GUPO, showing that incorporating group\-gradient uncertainty into policy optimization can improve GRPO\-based post\-training\.

![Refer to caption](https://arxiv.org/html/2608.17411v1/illustration-gradient-v1.png)Figure 3:Illustration of GRPO gradient optimization process\.
## Related Work

#### Gradient Conflict in LLM Post\-Training

Reinforcement learning has become an important approach for LLM post\-training, with GRPO\([27](https://arxiv.org/html/2608.17411#bib.bib1)\)being widely studied for improving reasoning capabilities\. However, different gradients produced during GRPO optimization may result in incompatible directions, thereby interfering with effective policy updates\. Recent studies investigate such gradient conflicts in GRPO\-based post\-training at different levels of the optimization process\. DaGRPO\([30](https://arxiv.org/html/2608.17411#bib.bib24)\)investigates conflicts within a rollout group, relating them to insufficient distinctiveness among sampled responses and applying sequence\-level gradient correction\. PCR\([24](https://arxiv.org/html/2608.17411#bib.bib26)\)studies conflicts between plasticity and stability gradients in GRPO and addresses them through probabilistic projection\. ResRL\([19](https://arxiv.org/html/2608.17411#bib.bib25)\)analyzes gradient interference between positive and negative responses and modulates negative gradients using projection residuals\. These studies indicate that gradient conflicts can arise from different sources and at different granularities\. Unlike prior works, we focus on conflicts among group gradients associated with different queries within the same GRPO mini\-batch, and analyze them from the perspective of gradient uncertainty\. We propose an optimization framework that explicitly characterizes the uncertainty of these query\-level group gradients and calibrates their contributions during mini\-batch gradient aggregation\.

#### Uncertainty Estimation in LLM Post\-Training

Uncertainty estimation provides a principled way to quantify the reliability of model outputs or evaluation signals and is widely applied to LLMs to assess the reliability of generated responses and alignment signals\. Uncertainty\-aware Reward Model\([21](https://arxiv.org/html/2608.17411#bib.bib28)\)quantifies uncertainty in learned reward\-model predictions by modeling stochastic human preferences and disagreement among reward models\. SEED\-GRPO\([8](https://arxiv.org/html/2608.17411#bib.bib27)\)estimates query\-level uncertainty from the semantic diversity of multiple sampled responses and uses it to modulate the corresponding advantages\. CAPO\([28](https://arxiv.org/html/2608.17411#bib.bib29)\)studies the overconfidence associated with uncertainty\-agnostic advantage estimation in GRPO and introduces uncertainty\-aware advantages to improve reasoning performance and confidence calibration\. Recent works show that uncertainty can serve as an explicit measure of how reliably a prediction or scalar evaluation should be trusted\. Unlike previous methods, we focus on conflicts among group gradients associated with different queries within the same GRPO mini\-batch and analyze them from the perspective of gradient uncertainty\. We model each group gradient as a random variable represented by a probability distribution under a Bayesian formulation, and derive an uncertainty estimation to reflect the reliability of its update signal\. The resulting uncertainty is further incorporated into gradient aggregation to mitigate the adverse effects of gradient conflicts and guide more effective policy updates\.

## Problem Settings and Analyses

#### Problem Settings

For a given queryq∼P⁡\(Q\)q\\sim P\(Q\), GRPO samples a group ofGGcandidate outputs\{oi\}i=1G\\\{o\_\{i\}\\\}\_\{i=1\}^\{G\}from the old policyπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\. Each outputoio\_\{i\}is generated autoregressively asoi=\{oi,1,oi,2,…,oi,Ti\}o\_\{i\}=\\\{o\_\{i,1\},o\_\{i,2\},\\dots,o\_\{i,T\_\{i\}\}\\\}, whereTiT\_\{i\}denotes the length of theii\-th output\. After generation, each output is evaluated by an outcome reward functionri=Rout​\(q,oi\)r\_\{i\}=R\_\{\\mathrm\{out\}\}\(q,o\_\{i\}\)\. In verifiable reasoning tasks,RoutR\_\{\\mathrm\{out\}\}is usually defined according to whether the final answer is correct\.

GRPO estimates the relative advantage of each output within the sampled group, encouraging the policy to increase the probability of outputs that obtain higher rewards\.

Ai=ri−mean⁡\(r1,r2,…,rG\)std⁡\(r1,r2,…,rG\)\.A\_\{i\}=\\frac\{r\_\{i\}\-\\mathrm\{mean\}\(r\_\{1\},r\_\{2\},\\dots,r\_\{G\}\)\}\{\\mathrm\{std\}\(r\_\{1\},r\_\{2\},\\dots,r\_\{G\}\)\}\.\(1\)
The GRPO objective can then be written as

𝒥GRPO​\(θ\)=𝔼q∼P⁡\(Q\),\{oi\}i=1G∼πθold\\displaystyle\\mathcal\{J\}\_\{\\mathrm\{GRPO\}\}\(\\theta\)=\\mathbb\{E\}\_\{q\\sim P\(Q\),\\\{o\_\{i\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\}\(2\)1G​∑i=1G1Ti​∑t=1Ti\[min⁡\(ρi,t​\(θ\)​Ai,clip⁡\(ρi,t​\(θ\),1−ε,1\+ε\)​Ai\)−β​DKLi,t\],\\displaystyle\\scalebox\{0\.78\}\{$\\displaystyle\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{T\_\{i\}\}\\sum\_\{t=1\}^\{T\_\{i\}\}\\bigg\[\\min\(\\rho\_\{i,t\}\(\\theta\)A\_\{i\},\\mathrm\{clip\}\(\\rho\_\{i,t\}\(\\theta\),1\-\\varepsilon,1\+\\varepsilon\)A\_\{i\}\)\-\\beta D\_\{\\mathrm\{KL\}\}^\{i,t\}\\bigg\]$\},whereε\\varepsilonis the clipping coefficient andβ\\betacontrols the strength of KL regularization\. GRPO uses the importance ratio between the current policyπθ\\pi\_\{\\theta\}and the old policyπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}:

ρi,t​\(θ\)=πθ​\(oi,t∣q,oi,<t\)πθold​\(oi,t∣q,oi,<t\)\.\\rho\_\{i,t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(o\_\{i,t\}\\mid q,o\_\{i,<t\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(o\_\{i,t\}\\mid q,o\_\{i,<t\}\)\}\.\(3\)
The KL regularization is computed with respect to a fixed reference policyπref\\pi\_\{\\mathrm\{ref\}\}, which can be estimated as

DKLi,t=πref​\(oi,t∣q,oi,<t\)πθ​\(oi,t∣q,oi,<t\)−log⁡πref​\(oi,t∣q,oi,<t\)πθ​\(oi,t∣q,oi,<t\)−1\.D\_\{\\mathrm\{KL\}\}^\{i,t\}=\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(o\_\{i,t\}\\mid q,o\_\{i,<t\}\)\}\{\\pi\_\{\\theta\}\(o\_\{i,t\}\\mid q,o\_\{i,<t\}\)\}\-\\log\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(o\_\{i,t\}\\mid q,o\_\{i,<t\}\)\}\{\\pi\_\{\\theta\}\(o\_\{i,t\}\\mid q,o\_\{i,<t\}\)\}\-1\.\(4\)
For an optimization mini\-batch containingBBqueries, we focus on the advantage\-weighted policy\-gradient signals that determine how the sampled responses contribute to the policy update\. The corresponding mini\-batch gradient is

∇θ𝒥ℬ​\(θ\)=1B​G\\displaystyle\\nabla\_\{\\theta\}\\mathcal\{J\}\_\{\\mathcal\{B\}\}\(\\theta\)=\\frac\{1\}\{BG\}∑b=1B∑i=1G1Tb,i∑t=1Tb,i\\displaystyle\\sum\_\{b=1\}^\{B\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{T\_\{b,i\}\}\\sum\_\{t=1\}^\{T\_\{b,i\}\}\(5\)Ab,i​ρb,i,t​\(θ\)​∇θ​log\\displaystyle A\_\{b,i\}\\rho\_\{b,i,t\}\(\\theta\)\\nabla\_\{\\theta\}\\logπθ​\(ob,i,t∣qb,ob,i,<t\)\.\\displaystyle\\pi\_\{\\theta\}\\left\(o\_\{b,i,t\}\\mid q\_\{b\},o\_\{b,i,<t\}\\right\)\.
This gradient can be decomposed according to the query groups within the mini\-batch\([12](https://arxiv.org/html/2608.17411#bib.bib15);[23](https://arxiv.org/html/2608.17411#bib.bib16);[22](https://arxiv.org/html/2608.17411#bib.bib17);[5](https://arxiv.org/html/2608.17411#bib.bib18)\)\. Specifically, the gradient contributed by thebb\-th query is defined as

𝐠b\(θ\)=1G∑i=1G\\displaystyle\\mathbf\{g\}\_\{b\}\(\\theta\)=\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}1Tb,i∑t=1Tb,i\\displaystyle\\frac\{1\}\{T\_\{b,i\}\}\\sum\_\{t=1\}^\{T\_\{b,i\}\}\(6\)Ab,i​ρb,i,t​\(θ\)​∇θ​log⁡πθ\\displaystyle A\_\{b,i\}\\rho\_\{b,i,t\}\(\\theta\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(ob,i,t∣qb,ob,i,<t\),\\displaystyle\\left\(o\_\{b,i,t\}\\mid q\_\{b\},o\_\{b,i,<t\}\\right\),which we term the*group gradient*\. Accordingly, the overall gradient can be expressed as

∇θ𝒥ℬ​\(θ\)=1B​∑b=1B𝐠b​\(θ\)\.\\nabla\_\{\\theta\}\\mathcal\{J\}\_\{\\mathcal\{B\}\}\(\\theta\)=\\frac\{1\}\{B\}\\sum\_\{b=1\}^\{B\}\\mathbf\{g\}\_\{b\}\(\\theta\)\.\(7\)
![Refer to caption](https://arxiv.org/html/2608.17411v1/overview-v9.png)Figure 4:Overview of our proposed method\.
#### Empirical Analysis

In GRPO, the policy gradient of an optimization mini\-batch can be decomposed as the aggregation of group gradients from different queries\([12](https://arxiv.org/html/2608.17411#bib.bib15);[23](https://arxiv.org/html/2608.17411#bib.bib16);[22](https://arxiv.org/html/2608.17411#bib.bib17);[5](https://arxiv.org/html/2608.17411#bib.bib18)\)\. As illustrated inFigure[3](https://arxiv.org/html/2608.17411#Sx1.F3)a, the token\-level gradients from theGGresponses for queryqbq\_\{b\}are aggregated into the group gradient𝐠b​\(θ\)\\mathbf\{g\}\_\{b\}\(\\theta\)\. For a mini\-batch containingBBqueries, the gradient is obtained by averaging these group gradients:∇θ𝒥ℬ​\(θ\)=1B​∑b=1B𝐠b​\(θ\)\\nabla\_\{\\theta\}\\mathcal\{J\}\_\{\\mathcal\{B\}\}\(\\theta\)=\\frac\{1\}\{B\}\\sum\_\{b=1\}^\{B\}\\mathbf\{g\}\_\{b\}\(\\theta\)\.

Based on this update mechanism, we conduct controlled experiments\. Starting from the same checkpoint, we sample different mini\-batches under identical settings, each containingBBquery groups withGGresponses per query\. For each group\-gradient pair, we compute the cosine similaritycb,b′=𝐠b⊤​𝐠b′∥𝐠b∥2​∥𝐠b′∥2c\_\{b,b^\{\\prime\}\}=\\frac\{\\mathbf\{g\}\_\{b\}^\{\\top\}\\mathbf\{g\}\_\{b^\{\\prime\}\}\}\{\\lVert\\mathbf\{g\}\_\{b\}\\rVert\_\{2\}\\lVert\\mathbf\{g\}\_\{b^\{\\prime\}\}\\rVert\_\{2\}\}, wherecb,b′<0c\_\{b,b^\{\\prime\}\}<0indicates a directional conflict\. The conflict level of mini\-batchℬ\\mathcal\{B\}is measured byCrate​\(ℬ\)=∑\(b,b′\)∈𝒫ℬ𝕀\[cb,b′<0\]\|𝒫ℬ\|C\_\{\\mathrm\{rate\}\}\(\\mathcal\{B\}\)=\\frac\{\\sum\_\{\(b,b^\{\\prime\}\)\\in\\mathcal\{P\}\_\{\\mathcal\{B\}\}\}\\mathbb\{I\}\[c\_\{b,b^\{\\prime\}\}<0\]\}\{\|\\mathcal\{P\}\_\{\\mathcal\{B\}\}\|\}, where𝒫ℬ\\mathcal\{P\}\_\{\\mathcal\{B\}\}is the set of group\-gradient pairs and𝕀⁡\[⋅\]\\mathbb\{I\}\[\\cdot\]is the indicator function\. To evaluate the corresponding policy update, we perform GRPO update using each mini\-batch and computeΔ​NLL=NLLbefore−NLLafter\\Delta\\mathrm\{NLL\}=\\mathrm\{NLL\}\_\{\\mathrm\{before\}\}\-\\mathrm\{NLL\}\_\{\\mathrm\{after\}\}on a fixed validation set, where a largerΔ​NLL\\Delta\\mathrm\{NLL\}indicates a more effective update\. More details are provided in theAppendix\.

Figure[2](https://arxiv.org/html/2608.17411#Sx1.F2)ashows a representative low\-conflict mini\-batch in which most cosine similarities are non\-negative, indicating the group gradients are largely aligned\. In contrast,Figure[2](https://arxiv.org/html/2608.17411#Sx1.F2)bpresents a high\-conflict mini\-batch containing several group\-gradient pairs with negative cosine similarities, showing that conflicts can arise within the same mini\-batch\.Figure[2](https://arxiv.org/html/2608.17411#Sx1.F2)cfurther summarizes the pairwise cosine similarities across all sampled mini\-batches: among563563group\-gradient pairs,262262pairs have negative cosine similarities, accounting for46\.5%46\.5\\%of the total\.Figure[2](https://arxiv.org/html/2608.17411#Sx1.F2)dcompares the validationΔ​NLL\\Delta\\mathrm\{NLL\}of mini\-batches with low, medium, and highCrateC\_\{\\mathrm\{rate\}\}, where the high\-conflict subset exhibits a relatively lower median and contains negative updates, suggesting that severe gradient conflicts can be associated with less effective policy updates\.

These findings suggest that group gradients induced by different queries within the same mini\-batch can exhibit conflicts\. Therefore, this leads to the central scientific question of this work:When group gradients induced by different queries conflict within the same GRPO mini\-batch, how can we obtain a reliable aggregated update direction?

#### Motivation Analysis

When group gradients point in conflicting directions, directly averaging them may not yield a reliable update direction\. Here, a reliable update direction refers to one that reasonably reflects the overall optimization tendency supported by the mini\-batch, rather than being dominated by a few conflicting group gradients\. Such a direction should preserve the optimization information jointly provided by the mini\-batch while limiting the disproportionate influence of particular conflicting signals\.

Gradient uncertainty provides a possible basis for assessing such reliability of different group gradients during aggregation\([13](https://arxiv.org/html/2608.17411#bib.bib32);[1](https://arxiv.org/html/2608.17411#bib.bib30);[3](https://arxiv.org/html/2608.17411#bib.bib31)\)\. A group gradient can be viewed as a random variable represented by a probability distribution, which describes not only the possible values of the group gradient, but also how likely these values are\. When most of the probability mass is concentrated within a narrow region around the mean, the distribution mainly supports gradient values that are similar to one another\. The limited variation among these values corresponds to lower gradient uncertainty and higher reliability\. Conversely, when the probability mass is spread over a broader region, the distribution supports a wider range of substantially different gradient values\. The larger variation among these values corresponds to higher gradient uncertainty and lower reliability\. This notion of reliability characterizes how consistently a group\-gradient direction is represented by its distribution, rather than whether that direction is necessarily correct\.Figure[1](https://arxiv.org/html/2608.17411#Sx1.F1)illustrates this perspective through a two\-dimensional example\. The arrows𝐠1\\mathbf\{g\}\_\{1\}and𝐠2\\mathbf\{g\}\_\{2\}denote the mean gradients of two query groups, while the surrounding contours depict their probability distributions\. These two mean gradients point in conflicting directions, and their distributions exhibit different degrees of concentration\. The compact contours around𝐠1\\mathbf\{g\}\_\{1\}correspond to lower uncertainty and higher reliability, whereas the broader contours around𝐠2\\mathbf\{g\}\_\{2\}correspond to higher uncertainty and lower reliability\. Direct aggregation assigns the two group gradients equal coefficients without accounting for this difference in reliability\. Consequently, the more uncertain group gradient may exert a disproportionate influence on the directly aggregated direction𝐠da\\mathbf\{g\}\_\{\\mathrm\{da\}\}\.

Motivated by the above, a natural way to address this problem is to account for group\-gradient uncertainty during aggregation, placing greater emphasis on more reliable group gradients while limiting the influence of less reliable ones to obtain a more reliable update direction under conflicts\.

## Methodology

Based on the above discussion, we propose Gradient Uncertainty\-aware Policy Optimization \(GUPO\), a framework that incorporates the uncertainty of group gradients into the gradient aggregation of GRPO\. Specifically, we first construct a probability distribution over the policy parameters under a Bayesian formulation and then estimate the induced distribution of each group gradient\. We then transform the precision of the resulting gradient distributions into gradient evidence, from which the dimension\-wise belief and group\-level uncertainty are derived\. Finally, we use the resulting uncertainty to calibrate the aggregation of group gradients within each mini\-batch, assigning greater influence to lower\-uncertainty gradients and attenuating the influence of higher\-uncertainty ones\. The overall framework is illustrated inFigure[4](https://arxiv.org/html/2608.17411#Sx3.F4), with the pseudo\-code provided in theAppendix\.

### Bayesian\-based Group Gradient Distribution

In this subsection, we formulate each query\-level group gradient as a random variable by propagating parameter uncertainty to the corresponding gradient\. Specifically, we construct a parameter distribution and use Monte Carlo sampling to estimate the induced distribution of each group gradient\.

Table 1:Pass@1 performance on mathematical reasoning datasets\. We compare base models trained with different approaches\. The best results under each base model are highlighted in bold\. More results are provided in theAppendix\.\(a\)Mean Performance Gain\(b\)Batch Performance Gain\(c\)High\-Conflict Batches\(d\)Conflict v\.s\. Performance
Figure 5:Effect of GUPO\. \(a\) and \(b\) report the mean and batch\-wise validation gains over GRPO\. \(c\) shows the performance comparison on high\-conflict mini\-batches\. \(d\) compares validationΔ​NLL\\Delta\\mathrm\{NLL\}produced by GUPO across different conflict levels\.#### Posterior Distribution Estimation of Policy Parameters

Consider an optimization mini\-batch containingBBqueries:ℬ=\{\(qb,𝒪b\)\}b=1B\\mathcal\{B\}=\\left\\\{\(q\_\{b\},\\mathcal\{O\}\_\{b\}\)\\right\\\}\_\{b=1\}^\{B\}, where𝒪b=\{ob,1,ob,2,…,ob,G\}\\mathcal\{O\}\_\{b\}=\\\{o\_\{b,1\},o\_\{b,2\},\\ldots,o\_\{b,G\}\\\}contains theGGresponses sampled for queryqbq\_\{b\}from the old policyπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\. Each responseob,i=\{ob,i,t\}t=1Tb,io\_\{b,i\}=\\\{o\_\{b,i,t\}\\\}\_\{t=1\}^\{T\_\{b,i\}\}receives an outcome rewardrb,ir\_\{b,i\}\. Its group\-relative advantage is defined asAb,i=rb,i−r¯bσr,bA\_\{b,i\}=\\frac\{r\_\{b,i\}\-\\bar\{r\}\_\{b\}\}\{\\sigma\_\{r,b\}\}, wherer¯b\\bar\{r\}\_\{b\}andσr,b\\sigma\_\{r,b\}denote the mean and standard deviation of the rewards within thebb\-th response group\. We use𝒟=\{\(qb,𝒪b\)\}b=1B\\mathcal\{D\}=\\left\\\{\(q\_\{b\},\\mathcal\{O\}\_\{b\}\)\\right\\\}\_\{b=1\}^\{B\}to denote the rollout sequences contained in the mini\-batch\.

Directly constructing a posterior distribution over all parameters of an LLM is computationally prohibitive\. We therefore adopt a Bayesian formulation for the parameter setΦ⊆θ\\Phi\\subseteq\\thetaof the final trainable layer of the policy model\. Given the rollout data𝒟\\mathcal\{D\}, the posterior distribution ofΦ\\Phiisp⁡\(Φ∣𝒟\)∝p⁡\(𝒟∣Φ\)​p​\(Φ\)p\(\\Phi\\mid\\mathcal\{D\}\)\\propto p\(\\mathcal\{D\}\\mid\\Phi\)p\(\\Phi\), wherep⁡\(Φ\)p\(\\Phi\)is the prior distribution andp⁡\(𝒟∣Φ\)p\(\\mathcal\{D\}\\mid\\Phi\)is the likelihood of the sampled responses\. The specific prior and likelihood are in theAppendix\.

The exact posterior is generally intractable because the likelihood depends nonlinearly on the policy parameters\. We therefore use its expansion aroundΦ0\\Phi\_\{0\}to construct a Gaussian approximation\. LetΔ​Φ=Φ−Φ0\\Delta\\Phi=\\Phi\-\\Phi\_\{0\}\. The second\-order Taylor expansion of the log\-posterior is

log⁡p⁡\(Φ\|𝒟\)≈c\+∇Φ​log​p​\(Φ\|𝒟\)\|Φ=Φ0⊤​Δ​Φ−12​Δ​Φ⊤​H​Δ​Φ,\\displaystyle\\scalebox\{0\.95\}\{$\\log p\(\\Phi\|\\mathcal\{D\}\)\\approx c\+\\nabla\_\{\\Phi\}\\log p\(\\Phi\|\\mathcal\{D\}\)\|\_\{\\Phi=\\Phi\_\{0\}\}^\{\\top\}\\Delta\\Phi\-\\frac\{1\}\{2\}\\Delta\\Phi^\{\\top\}H\\Delta\\Phi,$\}\(8\)wherecccontains the terms independent ofΦ\\Phi, andH=−∇Φ2​log⁡p⁡\(Φ\|𝒟\)\|Φ=Φ0H=\-\\nabla\_\{\\Phi\}^\{2\}\\log p\(\\Phi\|\\mathcal\{D\}\)\|\_\{\\Phi=\\Phi\_\{0\}\}is the negative Hessian matrix of the log\-posterior\. We omit the first\-order term and retain the quadratic term to construct the Gaussian approximationq⁡\(Φ∣𝒟\)=𝒩⁡\(Φ∣Φ0,H−1\)q\(\\Phi\\mid\\mathcal\{D\}\)=\\mathcal\{N\}\(\\Phi\\mid\\Phi\_\{0\},H^\{\-1\}\)centered at the current parameters \(seeAppendixfor further discussion\)\.

Directly computing the full Hessian matrix is infeasible for LLMs\. We therefore construct a tractable approximationH~\\widetilde\{H\}using empirical Fisher information\. Specifically, for each generated tokenob,i,to\_\{b,i,t\}, we compute the gradient of its log\-probability with respect to the trainable parameters:𝐬b,i,t=∇Φ​log​πΦ​\(ob,i,t\|qb,ob,i,<t\)\|Φ=Φ0\\mathbf\{s\}\_\{b,i,t\}=\\nabla\_\{\\Phi\}\\log\\pi\_\{\\Phi\}\(o\_\{b,i,t\}\|q\_\{b\},o\_\{b,i,<t\}\)\|\_\{\\Phi=\\Phi\_\{0\}\}, which describes how the generation probability of the token changes with the policy parameters\. The empirical Fisher matrix is then obtained:

FΦ=1Ntoken​∑b=1B∑i=1G∑t=1Tb,isb,i,t​sb,i,t⊤,F\_\{\\Phi\}=\\frac\{1\}\{N\_\{\\mathrm\{token\}\}\}\\sum\_\{b=1\}^\{B\}\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{T\_\{b,i\}\}s\_\{b,i,t\}s\_\{b,i,t\}^\{\\top\},\(9\)whereNtoken=∑b=1B∑i=1GTb,iN\_\{\\mathrm\{token\}\}=\\sum\_\{b=1\}^\{B\}\\sum\_\{i=1\}^\{G\}T\_\{b,i\}is the total number of tokens in the mini\-batch\. The negative Hessian is approximated asH≈H~=Diag⁡\(FΦ\)\+δ​I,δ\>0H\\approx\\widetilde\{H\}=\\operatorname\{Diag\}\(F\_\{\\Phi\}\)\+\\delta I,\\;\\delta\>0\. Accordingly, using this diagonal empirical Fisher approximation, we obtain

q⁡\(Φ∣𝒟\)=𝒩⁡\(Φ∣Φ0,H~−1\)\.q\(\\Phi\\mid\\mathcal\{D\}\)=\\mathcal\{N\}\(\\Phi\\mid\\Phi\_\{0\},\\widetilde\{H\}^\{\-1\}\)\.\(10\)

#### Group Gradient Distribution Estimation

Based on the approximate parameter distribution, we next estimate the distribution induced over each group gradient\. As defined inProblem Settings, the group gradient associated with queryqbq\_\{b\}is denoted by𝐠b​\(θ\)\\mathbf\{g\}\_\{b\}\(\\theta\)\. Because both the policy probabilities and their derivatives depend on the policy parameters, different values ofΦ\\Phican produce different group gradients\. Therefore, whenΦ∼q⁡\(Φ\|𝒟\)\\Phi\\sim q\(\\Phi\|\\mathcal\{D\}\), the corresponding group gradient can be regarded as a random variable\.

Since the distribution of𝐠b​\(Φ\)\\mathbf\{g\}\_\{b\}\(\\Phi\)is generally difficult to derive in closed form, we approximate it through Monte Carlo sampling\. Specifically, we drawMMparameter samples from the parameter distribution\{Φ\(m\)\}m=1M∼q⁡\(Φ\|𝒟\)\\\{\\Phi^\{\(m\)\}\\\}\_\{m=1\}^\{M\}\\sim q\(\\Phi\|\\mathcal\{D\}\)\. For each parameter sampleΦ\(m\)\\Phi^\{\(m\)\}, we construct the corresponding complete policy parameter setθ\(m\)=θ⁡\(Φ\(m\)\)\\theta^\{\(m\)\}=\\theta\\left\(\\Phi^\{\(m\)\}\\right\), while keeping the remaining policy parameters fixed\. We then compute thebb\-th group gradient𝐠b\(m\)=𝐠b​\(θ\(m\)\)\\mathbf\{g\}\_\{b\}^\{\(m\)\}=\\mathbf\{g\}\_\{b\}\(\\theta^\{\(m\)\}\)\. The mean and covariance of thebb\-th group gradient are defined as

𝝁b\\displaystyle\\boldsymbol\{\\mu\}\_\{b\}=1M​∑m=1M𝐠b\(m\),\\displaystyle=\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}\\mathbf\{g\}\_\{b\}^\{\(m\)\},\(11\)Σb\\displaystyle\\Sigma\_\{b\}=1M−1​∑m=1M\(𝐠b\(m\)−𝝁b\)​\(𝐠b\(m\)−𝝁b\)⊤\.\\displaystyle=\\frac\{1\}\{M\-1\}\\sum\_\{m=1\}^\{M\}\(\\mathbf\{g\}\_\{b\}^\{\(m\)\}\-\\boldsymbol\{\\mu\}\_\{b\}\)\(\\mathbf\{g\}\_\{b\}^\{\(m\)\}\-\\boldsymbol\{\\mu\}\_\{b\}\)^\{\\top\}\.
To avoid constructing a full covariance matrix in the high\-dimensional gradient space, we use the element\-wise variance vector𝝈b2\\boldsymbol\{\\sigma\}\_\{b\}^\{2\}instead of the full covariance matrix\. The probability distribution of𝐠b\\mathbf\{g\}\_\{b\}is then approximated as

q⁡\(𝐠b∣𝒟\)≈𝒩⁡\(𝐠b∣𝝁b,Diag⁡\(𝝈b2\)\)\.q\(\\mathbf\{g\}\_\{b\}\\mid\\mathcal\{D\}\)\\approx\\mathcal\{N\}\(\\mathbf\{g\}\_\{b\}\\mid\\boldsymbol\{\\mu\}\_\{b\},\\operatorname\{Diag\}\(\\boldsymbol\{\\sigma\}\_\{b\}^\{2\}\)\)\.\(12\)

### Gradient Uncertainty\-aware Policy Optimization

In this subsection, we incorporate gradient uncertainty into the aggregation of group gradients\. Specifically, we first derive the gradient precision and transform it into gradient evidence\. The evidence is then used to derive dimension\-wise belief and group\-level uncertainty\. The resulting group\-level uncertainty is subsequently used to calibrate the aggregation of group gradients within the same mini\-batch\.

For thebb\-th group, letσb,d2\\sigma\_\{b,d\}^\{2\}denote the variance of its gradient in thedd\-th dimension\. We first compute the corresponding gradient precisionλb,d=1σb,d2\\lambda\_\{b,d\}=\\frac\{1\}\{\\sigma\_\{b,d\}^\{2\}\}, which describes the concentration of the gradient distribution in that dimension\. A largerλb,d\\lambda\_\{b,d\}corresponds to a more concentrated gradient distribution in the corresponding dimension, indicating that the gradient varies less across the sampled parameters\.

We then map the gradient precision to non\-negative gradient evidenceeb,d=ϕ⁡\(λb,d\)=λb,dse\_\{b,d\}=\\phi\\left\(\\lambda\_\{b,d\}\\right\)=\\lambda\_\{b,d\}^\{\\,s\}, wheres\>0s\>0controls the sensitivity of the evidence to the gradient precision\. A larger gradient precision produces stronger evidence, while a smaller gradient precision produces weaker evidence\.

Inspired by subjective logic and evidence theory\([16](https://arxiv.org/html/2608.17411#bib.bib21);[26](https://arxiv.org/html/2608.17411#bib.bib23)\), the corresponding Dirichlet concentration parameter is defined asαb,d=eb,d\+1\\alpha\_\{b,d\}=e\_\{b,d\}\+1\. The dimension\-wise belief and group\-level uncertainty are then defined as

bb,d=eb,dSb,ub=KSb,Sb=∑d=1Kαb,d\.b\_\{b,d\}=\\frac\{e\_\{b,d\}\}\{S\_\{b\}\},\\quad u\_\{b\}=\\frac\{K\}\{S\_\{b\}\},\\quad S\_\{b\}=\\sum\_\{d=1\}^\{K\}\\alpha\_\{b,d\}\.\(13\)whereSbS\_\{b\}is the Dirichlet strength andKKdenotes the total number of gradient dimensions\.bb,db\_\{b,d\}represents the belief supported by the evidence in thedd\-th dimension, whileubu\_\{b\}summarizes the uncertainty of thebb\-th group gradient\.

Then, we use the group\-level uncertainty to adjust the aggregation of group gradients\. The uncertainty\-aware weight of thebb\-th group is defined asωb=1−ub∑ℓ=1B\(1−uℓ\)\\omega\_\{b\}=\\frac\{1\-u\_\{b\}\}\{\\sum\_\{\\ell=1\}^\{B\}\(1\-u\_\{\\ell\}\)\}\. To preserve the original aggregation structure of GRPO, we further combineωb\\omega\_\{b\}with the original gradient coefficient1B\\frac\{1\}\{B\}:ω~b=1−ηB\+η​ωb\\widetilde\{\\omega\}\_\{b\}=\\frac\{1\-\\eta\}\{B\}\+\\eta\\omega\_\{b\}, whereη∈\[0,1\]\\eta\\in\[0,1\]controls the degree to which group\-gradient uncertainty adjusts the original gradient aggregation \(see theAppendixfor more discussion\)\. The final GUPO gradient is

∇θ𝒥ℬGUPO​\(θ\)=∑b=1Bω~b​𝐠b​\(θ\)\.\\nabla\_\{\\theta\}\\mathcal\{J\}\_\{\\mathcal\{B\}\}^\{\\mathrm\{GUPO\}\}\(\\theta\)=\\sum\_\{b=1\}^\{B\}\\widetilde\{\\omega\}\_\{b\}\\mathbf\{g\}\_\{b\}\(\\theta\)\.\(14\)
Under this aggregation, group gradients with lower uncertainty receive greater contributions, whereas those with higher uncertainty receive smaller contributions\. Consequently, this aggregation aims to provide a more reliable update direction for policy optimization\.

## Experiments

In this section, we conduct experiments on various benchmarks to evaluate GUPO\. More details, experiments, and analysis are provided in theAppendix\.

### Experimental Settings

#### Benchmarks and Baselines

We compare our method with a series of representative post\-training baselines, including GRPO\([27](https://arxiv.org/html/2608.17411#bib.bib1)\), Length Penalty\([2](https://arxiv.org/html/2608.17411#bib.bib9)\), ReST\-MCTS\([31](https://arxiv.org/html/2608.17411#bib.bib10)\), GVPO\([32](https://arxiv.org/html/2608.17411#bib.bib11)\), Dr\. GRPO\([20](https://arxiv.org/html/2608.17411#bib.bib12)\), GCPO\([14](https://arxiv.org/html/2608.17411#bib.bib13)\), and MRT\([25](https://arxiv.org/html/2608.17411#bib.bib14)\)\. We evaluate our method on several widely used reasoning benchmarks: AIME 2024, AIME 2025, AMC 2023, MATH500\([15](https://arxiv.org/html/2608.17411#bib.bib7)\), MinervaMATH\([18](https://arxiv.org/html/2608.17411#bib.bib8)\), and GSM8K\([10](https://arxiv.org/html/2608.17411#bib.bib6)\)\. AIME and AMC evaluate competition\-level mathematical reasoning\. MATH500 and MinervaMATH cover diverse mathematical problem\-solving tasks\. GSM8K focuses on multi\-step grade\-school arithmetic reasoning\. We report Pass@1 accuracy on each benchmark and the average performance across all six benchmarks\.

#### Implementation Details

We conduct experiments on three base models at two scales: DeepScaleR\-1\.5B\-Preview, DeepSeek\-R1\-Distill\-Qwen\-1\.5B, and DeepSeek\-R1\-Distill\-Qwen\-7B\. We use a learning rate of1×10−61\\times 10^\{\-6\}, a weight decay of0\.010\.01, and a global batch size of256256\. All experiments are conducted on an NVIDIA H100 GPU cluster\. More implementation details are provided in theAppendix\.

\(a\)Parameterss\(b\)Parameterη\\eta
Figure 6:Parameter sensitivity\.

### Main Results

#### Performance Comparison

We compare GUPO with representative baselines across multiple benchmarks and base models\. As shown inTable[1](https://arxiv.org/html/2608.17411#Sx4.T1), the results demonstrate the effectiveness of GUPO, which incorporates group\-gradient uncertainty into gradient aggregation\.

#### Ablation Study about Parameter Sensitivity

We conduct analysis of evaluating evidence sensitivityssat\[0\.01,0\.05,0\.1,0\.3,0\.5,0\.7,0\.9\]\[0\.01,0\.05,0\.1,0\.3,0\.5,0\.7,0\.9\], and aggregation coefficientη\\etaover\[0,0\.5\]\[0,0\.5\]with an interval of0\.10\.1\.Figure[6](https://arxiv.org/html/2608.17411#Sx5.F6)shows the best performance is achieved ats=0\.5s=0\.5andη=0\.1\\eta=0\.1, which are our final setting\. More results are in theAppendix\.

#### Analysis

We further analyze the effect of GUPO under group\-gradient conflicts\. As shown inFigure[5](https://arxiv.org/html/2608.17411#Sx4.F5), GUPO can improve validation performance over GRPO across sampled mini\-batches, with the paired comparison on high\-conflict mini\-batches providing further evidence\. The results suggest that GUPO can mitigate the degradation in update effectiveness associated with severe group\-gradient conflicts\.

## Conclusion

In this work, we investigate conflicts among group gradients induced by different queries within the same GRPO mini\-batch and show that such conflicts can be associated with less effective policy updates\. To obtain a reliable update direction, we propose Gradient Uncertainty\-aware Policy Optimization \(GUPO\), which models each group gradient as a probability distribution under a Bayesian formulation, estimates its uncertainty, and incorporates the resulting uncertainty into gradient aggregation\. By enhancing the contribution of lower\-uncertainty group gradients and mitigating the influence of highly uncertain ones, GUPO can enable the aggregated gradient to better reflect the optimization tendency supported by the mini\-batch\. Experiments across multiple reasoning benchmarks demonstrate the effectiveness of GUPO\.

## References

- Akellaet al\.\(2021\)R\. T\. Akella, K\. Azizzadenesheli, M\. Ghavamzadeh, A\. Anandkumar, and Y\. YueDeep bayesian quadrature policy optimization\.InProceedings of the AAAI conference on artificial intelligence,Vol\.35,pp\. 6600–6608\.Cited by:[Introduction](https://arxiv.org/html/2608.17411#Sx1.p3.1),[Motivation Analysis](https://arxiv.org/html/2608.17411#Sx3.SS0.SSSx3.p2.1)\.
- Arora and Zanette \(2026\)D\. Arora and A\. ZanetteTraining language models to reason efficiently\.Advances in Neural Information Processing Systems38,pp\. 60770–60808\.Cited by:[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.13.1),[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.22.1),[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.4.1),[Benchmarks and Baselines](https://arxiv.org/html/2608.17411#Sx5.SSx1.SSSx1.p1.1)\.
- Balles and Hennig \(2018\)L\. Balles and P\. HennigDissecting adam: the sign, magnitude and variance of stochastic gradients\.InInternational Conference on Machine Learning,pp\. 404–413\.Cited by:[Motivation Analysis](https://arxiv.org/html/2608.17411#Sx3.SS0.SSSx3.p2.1)\.
- Baoet al\.\(2021\)W\. Bao, Q\. Yu, and Y\. KongEvidential deep learning for open set action recognition\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 13349–13358\.Cited by:[Introduction](https://arxiv.org/html/2608.17411#Sx1.p4.1)\.
- Barakatet al\.\(2026\)A\. Barakat, S\. Chakraborty, K\. Pahwa, and A\. S\. BediWhy pass@ k optimization can degrade pass@ 1: prompt interference in llm post\-training\.arXiv preprint arXiv:2602\.21189\.Cited by:[Introduction](https://arxiv.org/html/2608.17411#Sx1.p2.1),[Problem Settings](https://arxiv.org/html/2608.17411#Sx3.SS0.SSSx1.p6.2),[Empirical Analysis](https://arxiv.org/html/2608.17411#Sx3.SS0.SSSx2.p1.1)\.
- Box and Tiao \(2011\)G\. E\. Box and G\. C\. TiaoBayesian inference in statistical analysis\.John Wiley & Sons\.Cited by:[Introduction](https://arxiv.org/html/2608.17411#Sx1.p4.1)\.
- Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[Introduction](https://arxiv.org/html/2608.17411#Sx1.p1.1)\.
- Chenet al\.\(2025\)M\. Chen, G\. Chen, W\. Wang, and Y\. YangSeed\-grpo: semantic entropy enhanced grpo for uncertainty\-aware policy optimization\.arXiv preprint arXiv:2505\.12346\.Cited by:[Uncertainty Estimation in LLM Post\-Training](https://arxiv.org/html/2608.17411#Sx2.SS0.SSSx2.p1.1)\.
- Chowdheryet al\.\(2023\)A\. Chowdhery, S\. Narang, J\. Devlin, M\. Bosma, G\. Mishra, A\. Roberts, P\. Barham, H\. W\. Chung, C\. Sutton, S\. Gehrmann,et al\.Palm: scaling language modeling with pathways\.Journal of machine learning research24\(240\),pp\. 1–113\.Cited by:[Introduction](https://arxiv.org/html/2608.17411#Sx1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[Benchmarks and Baselines](https://arxiv.org/html/2608.17411#Sx5.SSx1.SSSx1.p1.1)\.
- Dempster \(1968\)A\. P\. DempsterA generalization of bayesian inference\.Journal of the Royal Statistical Society: Series B \(Methodological\)30\(2\),pp\. 205–232\.Cited by:[Introduction](https://arxiv.org/html/2608.17411#Sx1.p4.1)\.
- Geet al\.\(2026\)C\. Ge, C\. H\. Yin, H\. Liang, and J\. ZhangWhy grpo needs normalization: a local\-curvature perspective on adaptive gradients\.arXiv preprint arXiv:2601\.23135\.Cited by:[Introduction](https://arxiv.org/html/2608.17411#Sx1.p2.1),[Problem Settings](https://arxiv.org/html/2608.17411#Sx3.SS0.SSSx1.p6.2),[Empirical Analysis](https://arxiv.org/html/2608.17411#Sx3.SS0.SSSx2.p1.1)\.
- Ghavamzadehet al\.\(2016\)M\. Ghavamzadeh, Y\. Engel, and M\. ValkoBayesian policy gradient and actor\-critic algorithms\.Journal of Machine Learning Research17\(66\),pp\. 1–53\.Cited by:[Introduction](https://arxiv.org/html/2608.17411#Sx1.p3.1),[Motivation Analysis](https://arxiv.org/html/2608.17411#Sx3.SS0.SSSx3.p2.1)\.
- Guet al\.\(2026\)Z\. Gu, J\. Wang, R\. Zuo, C\. Sun, Z\. Song, C\. Zheng, and W\. QiangGroup causal policy optimization for post\-training large language models\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 30834–30842\.Cited by:[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.17.1),[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.26.1),[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.8.1),[Benchmarks and Baselines](https://arxiv.org/html/2608.17411#Sx5.SSx1.SSSx1.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the math dataset\.arXiv preprint arXiv:2103\.03874\.Cited by:[Benchmarks and Baselines](https://arxiv.org/html/2608.17411#Sx5.SSx1.SSSx1.p1.1)\.
- Jsang \(2018\)A\. JsangSubjective logic: a formalism for reasoning under uncertainty\.Springer Publishing Company, Incorporated\.Cited by:[Introduction](https://arxiv.org/html/2608.17411#Sx1.p4.1),[Gradient Uncertainty\-aware Policy Optimization](https://arxiv.org/html/2608.17411#Sx4.SSx2.p4.1)\.
- Kojimaet al\.\(2022\)T\. Kojima, S\. S\. Gu, M\. Reid, Y\. Matsuo, and Y\. IwasawaLarge language models are zero\-shot reasoners\.Advances in neural information processing systems35,pp\. 22199–22213\.Cited by:[Introduction](https://arxiv.org/html/2608.17411#Sx1.p1.1)\.
- Lewkowyczet al\.\(2022\)A\. Lewkowycz, A\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo,et al\.Solving quantitative reasoning problems with language models\.Advances in neural information processing systems35,pp\. 3843–3857\.Cited by:[Benchmarks and Baselines](https://arxiv.org/html/2608.17411#Sx5.SSx1.SSSx1.p1.1)\.
- Linet al\.\(2026\)Z\. Lin, X\. Wang, J\. Cao, J\. Chai, L\. Wang, X\. Lu, W\. Lin, R\. He, and G\. YinResrl: boosting llm reasoning via negative sample projection residual reinforcement learning\.arXiv preprint arXiv:2605\.00380\.Cited by:[Gradient Conflict in LLM Post\-Training](https://arxiv.org/html/2608.17411#Sx2.SS0.SSSx1.p1.1)\.
- Liuet al\.\(2025\)Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. LinUnderstanding r1\-zero\-like training: a critical perspective\.arXiv preprint arXiv:2503\.20783\.Cited by:[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.16.1),[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.25.1),[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.7.1),[Benchmarks and Baselines](https://arxiv.org/html/2608.17411#Sx5.SSx1.SSSx1.p1.1)\.
- Louet al\.\(2024\)X\. Lou, D\. Yan, W\. Shen, Y\. Yan, J\. Xie, and J\. ZhangUncertainty\-aware reward model: teaching reward models to know what is unknown\.arXiv preprint arXiv:2410\.00847\.Cited by:[Uncertainty Estimation in LLM Post\-Training](https://arxiv.org/html/2608.17411#Sx2.SS0.SSSx2.p1.1)\.
- Nguyenet al\.\(2026\)H\. T\. Nguyen, B\. Nguyen, W\. Ma, Y\. Zhao, R\. She, and V\. A\. NguyenAdaptive rollout allocation for online reinforcement learning with verifiable rewards\.arXiv preprint arXiv:2602\.01601\.Cited by:[Introduction](https://arxiv.org/html/2608.17411#Sx1.p2.1),[Problem Settings](https://arxiv.org/html/2608.17411#Sx3.SS0.SSSx1.p6.2),[Empirical Analysis](https://arxiv.org/html/2608.17411#Sx3.SS0.SSSx2.p1.1)\.
- Panagantiet al\.\(2026\)K\. Panaganti, Z\. Liang, W\. Yu, H\. Mi, and D\. YuGroup distributionally robust optimization\-driven reinforcement learning for llm reasoning\.arXiv preprint arXiv:2601\.19280\.Cited by:[Introduction](https://arxiv.org/html/2608.17411#Sx1.p2.1),[Problem Settings](https://arxiv.org/html/2608.17411#Sx3.SS0.SSSx1.p6.2),[Empirical Analysis](https://arxiv.org/html/2608.17411#Sx3.SS0.SSSx2.p1.1)\.
- Qianget al\.\(2026\)W\. Qiang, Z\. Gu, J\. Zhou, J\. Hu, J\. Wang, C\. Zheng, and H\. XiongOn the plasticity and stability for post\-training large language models\.arXiv preprint arXiv:2602\.06453\.Cited by:[Gradient Conflict in LLM Post\-Training](https://arxiv.org/html/2608.17411#Sx2.SS0.SSSx1.p1.1)\.
- Quet al\.\(2025\)Y\. Qu, M\. Y\. Yang, A\. Setlur, L\. Tunstall, E\. E\. Beeching, R\. Salakhutdinov, and A\. KumarOptimizing test\-time compute via meta reinforcement fine\-tuning\.arXiv preprint arXiv:2503\.07572\.Cited by:[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.18.1),[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.27.1),[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.9.1),[Benchmarks and Baselines](https://arxiv.org/html/2608.17411#Sx5.SSx1.SSSx1.p1.1)\.
- Sensoyet al\.\(2018\)M\. Sensoy, L\. Kaplan, and M\. KandemirEvidential deep learning to quantify classification uncertainty\.Advances in neural information processing systems31\.Cited by:[Introduction](https://arxiv.org/html/2608.17411#Sx1.p4.1),[Gradient Uncertainty\-aware Policy Optimization](https://arxiv.org/html/2608.17411#Sx4.SSx2.p4.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[Introduction](https://arxiv.org/html/2608.17411#Sx1.p1.1),[Gradient Conflict in LLM Post\-Training](https://arxiv.org/html/2608.17411#Sx2.SS0.SSSx1.p1.1),[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.12.1),[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.21.1),[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.3.1),[Benchmarks and Baselines](https://arxiv.org/html/2608.17411#Sx5.SSx1.SSSx1.p1.1)\.
- Wanget al\.\(2026\)Z\. Wang, X\. Lou, M\. Wu, Z\. Wen, and J\. ZhangCalibration\-aware policy optimization for reasoning llms\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 18375–18390\.Cited by:[Uncertainty Estimation in LLM Post\-Training](https://arxiv.org/html/2608.17411#Sx2.SS0.SSSx2.p1.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[Introduction](https://arxiv.org/html/2608.17411#Sx1.p1.1)\.
- Xieet al\.\(2025\)X\. Xie, X\. Wang, W\. Wang, S\. Chen, and W\. LinDagrpo: rectifying gradient conflict in reasoning via distinctiveness\-aware group relative policy optimization\.arXiv preprint arXiv:2512\.06337\.Cited by:[Gradient Conflict in LLM Post\-Training](https://arxiv.org/html/2608.17411#Sx2.SS0.SSSx1.p1.1)\.
- Zhanget al\.\(2024\)D\. Zhang, S\. Zhoubian, Z\. Hu, Y\. Yue, Y\. Dong, and J\. TangRest\-mcts\*: llm self\-training via process reward guided tree search\.Advances in Neural Information Processing Systems37,pp\. 64735–64772\.Cited by:[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.14.1),[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.23.1),[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.5.1),[Benchmarks and Baselines](https://arxiv.org/html/2608.17411#Sx5.SSx1.SSSx1.p1.1)\.
- Zhanget al\.\(2026\)K\. Zhang, Y\. Hong, J\. Bao, H\. Jiang, Y\. Song, H\. Dingqian, and H\. XiongGvpo: group variance policy optimization for large language model post\-training\.Advances in Neural Information Processing Systems38,pp\. 165798–165820\.Cited by:[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.15.1),[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.24.1),[Table 1](https://arxiv.org/html/2608.17411#Sx4.T1.1.1.6.1),[Benchmarks and Baselines](https://arxiv.org/html/2608.17411#Sx5.SSx1.SSSx1.p1.1)\.