Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPO
Summary
This paper identifies and addresses aggregation bias in GRPO-style reinforcement learning for LLMs, proposing Balanced Aggregation (BA) which improves training stability and final performance by computing token-level means separately for positive and negative subsets.
View Cached Full Text
Cached at: 05/08/26, 06:53 AM
Paper page - Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPO
Source: https://huggingface.co/papers/2605.04077 Authors:
,
,
,
,
,
,
,
,
,
Abstract
Balanced Aggregation improves reinforcement learning with verifiable rewards by addressing optimization biases in token-level policy gradient aggregation, leading to better training stability and performance.
Reinforcement learning with verifiable rewards(RLVR) has become a central paradigm for improving reasoning and code generation in large language models, andGRPO-style trainingis widely adopted for its simplicity and effectiveness. However, an important design choice remains underexplored: howtoken-level policy gradientterms are aggregated within each sampled group. Standard GRPO usessequence aggregation, while recent work has advocatedtoken aggregationas a better alternative. We show that these two rules induce differentoptimization biases:token aggregationintroduces sign-length coupling, whilesequence aggregationimplicitly downweights longer responses through sequence-level equal weighting. To address this tension, we proposeBalanced Aggregation(BA), a simple drop-in replacement that computes token-level means separately within the positive and negative subsets and then combines them with sequence-count-based weights. Experiments with Qwen2.5-Math-7B and Qwen3-1.7B on DAPO-17k and Polaris, evaluated on six reasoning and coding benchmarks, show that BA consistently improvestraining stabilityandfinal performanceover standard token andsequence aggregation. Our analysis further shows that the relative effectiveness of token andsequence aggregationis largely governed by response-length variation and the positive-negative length gap, highlighting aggregation as a critical design dimension in GRPO-style RLVR.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.04077
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.04077 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.04077 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.04077 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization
BiasGRPO proposes a framework using Group Relative Policy Optimization (GRPO) to stabilize social bias mitigation in LLMs by normalizing rewards across sampled completions, outperforming DPO and PPO on multiple benchmarks. The authors also release a compute-efficient bias reward model designed for integration into multi-objective RLHF pipelines.
Gradient-Based LoRA Rank Allocation Under GRPO: An Empirical Study
This study empirically demonstrates that gradient-based LoRA rank allocation, effective in supervised fine-tuning, degrades performance in GRPO-based reinforcement learning due to flatter gradient landscapes and a gradient amplification effect.
Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning
ProGPO is a learned-critic-free method for step-level advantage estimation in group-based RL for LLM agents, using exact-prefix action comparisons and rollout-based state potentials to improve credit assignment on long-horizon tasks. Experiments on ALFWorld and WebShop with Qwen2.5 models show it outperforms existing agentic RL baselines.
GAGPO: Generalized Advantage Grouped Policy Optimization
GAGPO proposes a critic-free RL method that uses a non-parametric grouped value proxy for step-level credit assignment in multi-turn agentic tasks, outperforming strong baselines on ALFWorld and WebShop.
Taming Extreme Tokens: Covariance-Aware GRPO with Gaussian-Kernel Advantage Reweighting
This paper proposes a covariance-aware variant of Group Relative Policy Optimization (GRPO) that uses Gaussian-kernel advantage reweighting to stabilize training entropy and improve reasoning performance in large language models.