CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learning
Summary
CorrGRPO proposes correlation-normalized GRPO that converts pairwise reward covariances into Pearson correlation coefficients, preventing large-scale rewards from dominating multi-reward normalization and improving performance on code generation, tool calling, and agent security across 0.5B-8B parameter models.
View Cached Full Text
Cached at: 10/02/26, 12:28 PM
Paper page - CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learning
Source: https://huggingface.co/papers/2609.36820
Abstract
GroupRelativePolicyOptimization(GRPO)iswidelyusedtotrainreasoninglanguagemodels,whereitcomputesadvantagesbycenteringandnormalizingrewardsacrossrolloutsofthesameprompt.Formultiplerewards,GRPOsumstherewardcomponentsandnormalizesthetotalrewardbyitswithin-groupstandarddeviation.Thecorrespondingvarianceequalsthesumofallpairwiserewardcovariances.Forafixedcenteredreward,largeraggregatecovarianceproducessmalleradvantages,andviceversa,allowingupdatemagnitudestoadapttorewarddependence.However,correlatedrewardswithlargescalescandominatethisnormalizationandsuppresssignalsfromsmaller-scalerewards.WeproposeCorrelation-NormalizedGRPO(CorrGRPO),whichnormalizespairwisecovariancesintoPearsoncorrelationcoefficients.CorrGRPOkeepsthecenteredtotalrewardunchangedwhilebalancingtheinfluenceofdifferentlyscaledrewardsonthecorrelation-basednormalization.Thisallowsadvantagemagnitudestoadapttorewardcorrelationswithoutthenormalizationbeingdominatedbylarge-scalerewardcomponents.WecompareCorrGRPOwithGRPOandothervariantsoncodegeneration,toolcalling,andagentsecurity,usingmodelsrangingfrom0.5Bto8Bparameters.Thesetasksallinvolvemultiplerewardsthatcanimprovetogetherorpresenttradeoffs.Resultsshowimprovementsacrossthreedomains,includingcodegeneration,toolcalling,andagentsecurity.Ourcodeisavailableathttps://github.com/HKUST-KnowComp/CorrGRPO.
View arXiv pageView PDFProject pageGitHub0Add to collection
Get this paper in your agent:
hf papers read 2609\.36820
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.36820 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.36820 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.36820 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Taming Extreme Tokens: Covariance-Aware GRPO with Gaussian-Kernel Advantage Reweighting
This paper proposes a covariance-aware variant of Group Relative Policy Optimization (GRPO) that uses Gaussian-kernel advantage reweighting to stabilize training entropy and improve reasoning performance in large language models.
BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization
BiasGRPO proposes a framework using Group Relative Policy Optimization (GRPO) to stabilize social bias mitigation in LLMs by normalizing rewards across sampled completions, outperforming DPO and PPO on multiple benchmarks. The authors also release a compute-efficient bias reward model designed for integration into multi-objective RLHF pipelines.
CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization
CoRT proposes a token-level credit weighting method for GRPO that uses counterfactual replay to compute token-wise log-likelihood contrasts, redistributing the signed advantage across tokens without an auxiliary scorer, achieving average gains of 4.4 percentage points over response-level GRPO.
Multi-module GRPO: Composing Policy Gradients and Prompt Optimization for Language Model Programs
The paper introduces mmGRPO, a multi-module extension of Group Relative Policy Optimization (GRPO) that improves accuracy in modular AI systems by optimizing language model calls and prompts. It reports an average 11% accuracy improvement across various tasks and provides an open-source implementation in DSPy.
LC-GRPO: Bridging Train-Inference Gap for Flow-Based GRPO with Langevin Correction
This paper introduces LC-GRPO, a flow-based GRPO framework with Langevin correction that bridges the train-inference gap by aligning stochastic training rollouts with deterministic ODE sampling, improving reward optimization on models like SD3.5, FLUX.1-Dev, and HunyuanVideo.