Group Entropy-Controlled Policy Optimization

Hugging Face Daily Papers Papers
reinforcement-learning llm entropy-control policy-optimization grpo alignment exploration-exploitation

Summary

This paper proposes Group Entropy-Controlled Policy Optimization (GEPO), a lightweight extension to GRPO that uses group entropy to perform entropy-conditioned asymmetric advantage shaping, addressing heterogeneous entropy regimes across tasks during RL-based alignment of LLMs. Experiments show consistent improvements over GRPO and recent entropy-controlled methods across multiple benchmarks.

Entropy control has become an effective tool in reinforcement learning (RL) of large language models (LLMs), helping balance exploration-exploitation trade-off during alignment process. Such RL paradigm is often conducted on mixtures of heterogeneous tasks, which induce distinct entropy regimes under the same policy, making global or token-level entropy regulation insufficient to corresponding heterogeneous needs of exploration. This heterogeneity further makes GRPO-style normalized advantages induce an entropy-dependent bias, making advantage signals across prompt groups statistically non-comparable. To address this issue, we propose Group Entropy-Controlled Policy Optimization (GEPO), a lightweight extension to GRPO that uses group entropy, estimated from existing grouped samples to perform entropy-conditioned asymmetric advantage shaping. GEPO attenuates positive advantages in low-entropy groups to reduce over-exploitation, and negative advantages in high-entropy groups to preserve exploration, with adaptive thresholds derived from historical entropy statistics. Extensive experiments on two base models across thirteen benchmarks spanning mathematics, physics, science, code generation, and instruction following show that GEPO consistently outperforms GRPO and recent entropy-controlled methods, delivering balanced cross-task improvements while preserving task-specific exploration levels throughout training.
Original Article
View Cached Full Text

Cached at: 07/21/26, 06:35 AM

Paper page - Group Entropy-Controlled Policy Optimization

Source: https://huggingface.co/papers/2607.16850

Abstract

Entropycontrolhasbecomeaneffectivetoolinreinforcementlearning(RL)oflargelanguagemodels(LLMs),helpingbalanceexploration-exploitationtrade-offduringalignmentprocess.SuchRLparadigmisoftenconductedonmixturesofheterogeneoustasks,whichinducedistinctentropyregimesunderthesamepolicy,makingglobalortoken-levelentropyregulationinsufficienttocorrespondingheterogeneousneedsofexploration.ThisheterogeneityfurthermakesGRPO-stylenormalizedadvantagesinduceanentropy-dependentbias,makingadvantagesignalsacrosspromptgroupsstatisticallynon-comparable.Toaddressthisissue,weproposeGroupEntropy-ControlledPolicyOptimization(GEPO),alightweightextensiontoGRPOthatusesgroupentropy,estimatedfromexistinggroupedsamplestoperformentropy-conditionedasymmetricadvantageshaping.GEPOattenuatespositiveadvantagesinlow-entropygroupstoreduceover-exploitation,andnegativeadvantagesinhigh-entropygroupstopreserveexploration,withadaptivethresholdsderivedfromhistoricalentropystatistics.Extensiveexperimentsontwobasemodelsacrossthirteenbenchmarksspanningmathematics,physics,science,codegeneration,andinstructionfollowingshowthatGEPOconsistentlyoutperformsGRPOandrecententropy-controlledmethods,deliveringbalancedcross-taskimprovementswhilepreservingtask-specificexplorationlevelsthroughouttraining.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.16850

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.16850 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.16850 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.16850 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Gradient Extrapolation-Based Policy Optimization

arXiv cs.LG

The article introduces Gradient Extrapolation-Based Policy Optimization (GXPO), a method that approximates multi-step lookahead in RL training for LLMs using only three backward passes. It demonstrates improved reasoning performance on math benchmarks over standard GRPO while maintaining fixed active-phase costs.

Hölder Policy Optimisation

Hugging Face Daily Papers

HölderPO introduces a generalized policy optimization framework that uses the Hölder mean for token-level probability aggregation in GRPO, with a dynamic annealing schedule to balance gradient concentration and variance. The method achieves state-of-the-art results on mathematical benchmarks (54.9% average, 7.2% relative gain over GRPO) and a 93.8% success rate on ALFWorld.