Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization
Summary
This paper introduces Contrastive Policy Optimization (CPO), which uses token-level contrastive disagreement between reference-guided and vanilla generation distributions for correctness-aware advantage shaping in reinforcement learning with verifiable rewards. CPO outperforms entropy-based RLVR methods on both in-domain and out-of-domain benchmarks.
View Cached Full Text
Cached at: 07/20/26, 09:39 AM
Paper page - Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization
Source: https://huggingface.co/papers/2607.14614
Abstract
Reinforcementlearningwithverifiablerewards(RLVR)commonlyusesentropyforadvantageshaping.However,entropycannotdistinguishusefuluncertaintyfromdetrimentalconfusion,limitingitseffectivenessasacorrectnesssignal.WeproposeContrastivePolicyOptimization(CPO),whichusestoken-levelcontrastivedisagreementbetweenreference-guidedandvanillagenerationdistributionsforcorrectness-awareadvantageshaping.Boththeoreticalandempiricalresultsshowthatthisdisagreementreliablyindicatestoken-levelcorrectness.WefurthershowthatOn-policyDistillationisaspecialcaseofCPO,wheretheposteriordistributionisinstantiatedbyanexternalteachermodel.CPOalsoresolvesthezero-advantageproblem.Experimentsonin-domainandout-of-domainbenchmarksdemonstratethatCPOsubstantiallyoutperformsentropy-basedRLVRmethodswhilemaintainingstronggeneralization.Furtheranalysisshowsthatcorrectandincorrectresponsesnaturallysupportexplorationandexploitationrespectively,andbalancingbothleadstothebestperformance.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2607\.14614
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.14614 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.14614 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.14614 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards
Proposes Correction-Oriented Policy Optimization (CIPO), an extension to RLVR that converts failed trajectories into correction-oriented supervision, improving reasoning and correction performance in LLMs across math and code benchmarks.
Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization
This paper introduces Guidance Contrastive Policy Optimization (GCPO), a novel algorithm that enables per-token credit assignment in reinforcement learning by contrasting model predictions under positive and negative prompts, consistently outperforming GRPO and DAPO baselines on text-to-image generation and chain-of-thought reasoning benchmarks.
CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization
CEPO improves reinforcement learning with verifiable rewards by using contrastive signals from rejected rollouts to distinguish decisive reasoning steps from filler tokens, achieving higher accuracy on multimodal math reasoning benchmarks compared to GRPO.
Group Entropy-Controlled Policy Optimization
This paper proposes Group Entropy-Controlled Policy Optimization (GEPO), a lightweight extension to GRPO that uses group entropy to perform entropy-conditioned asymmetric advantage shaping, addressing heterogeneous entropy regimes across tasks during RL-based alignment of LLMs. Experiments show consistent improvements over GRPO and recent entropy-controlled methods across multiple benchmarks.
ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy
Introduces ACPO, a token-level credit assignment framework for reinforcement learning in LLMs that uses fine-grained surrogate entropy to improve reasoning performance on math and coding benchmarks, outperforming strong baselines like DAPO, GTPO, and SAPO.