ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy

# ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy
Source: [https://arxiv.org/html/2607.03126](https://arxiv.org/html/2607.03126)
Zijun Xie1,2Yuyang You1,311footnotemark:1Yongzhi Li3Enlei Gong2 Zeyu Chen2Quan Chen3Yanhua Cheng3Peng Jiang3Yadong Mu1 1Peking University2Baidu Inc\.3Kuaishou Inc

###### Abstract

Reinforcement Learning \(RL\) has substantially improved the reasoning ability of large language models \(LLMs\), but sparse outcome rewards still make token\-level credit assignment difficult\. Existing scalable RL methods typically assign trajectory\-level rewards uniformly across tokens, while recent entropy\-aware approaches either rely on coarse detached heuristics or directly optimize true entropy, which can introduce non\-local gradient components misaligned with sampled\-token policy updates\. We propose Adaptive Credit Policy Optimization \(ACPO\), a token\-level credit assignment framework based on a mode\-local surrogate entropy\. ACPO asymmetrically modulates policy updates by emphasizing uncertain decisions in successful rollouts and overconfident tokens in failed rollouts\. We show that the surrogate admits deterministic entropy bounds and, under modal alignment and proximal updates, preserves the policy\-gradient direction to leading order\. Experiments on mathematical reasoning and coding benchmarks, including AIME 2025 and HumanEvalPro, show that ACPO consistently improves over strong RL baselines such as DAPO, GTPO, and SAPO\.

## 1Introduction

Recent advances in large language model \(LLM\) reasoning increasingly rely on reinforcement learning \(RL\) applied to supervised fine\-tuned \(SFT\) models, particularly in domains with verifiable objectives such as mathematics and coding\(DeepSeek\-AIet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib21); Yanget al\.,[2024b](https://arxiv.org/html/2607.03126#bib.bib36),[2025](https://arxiv.org/html/2607.03126#bib.bib37),[a](https://arxiv.org/html/2607.03126#bib.bib75); Dubeyet al\.,[2024](https://arxiv.org/html/2607.03126#bib.bib65)\)\. While SFT primarily imitates demonstrations\(Zhanget al\.,[2023](https://arxiv.org/html/2607.03126#bib.bib23); Ouyanget al\.,[2022](https://arxiv.org/html/2607.03126#bib.bib64); Dubeyet al\.,[2024](https://arxiv.org/html/2607.03126#bib.bib65)\), RL enables models to improve through exploration\(Zelikmanet al\.,[2022](https://arxiv.org/html/2607.03126#bib.bib70); Shaoet al\.,[2024](https://arxiv.org/html/2607.03126#bib.bib62); Zhanget al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib38)\)\. A pivotal advancement in this direction is Reinforcement Learning with Verifiable Rewards \(RLVR\)\(Shenget al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib43); Yueet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib39); Chenet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib26)\)\. Unlike traditional RLHF\(Schulmanet al\.,[2017](https://arxiv.org/html/2607.03126#bib.bib61); Ouyanget al\.,[2022](https://arxiv.org/html/2607.03126#bib.bib64)\), which depends on subjective and costly human preferences, RLVR leverages objective, deterministic signals, such as compiler outcomes or mathematical verifiers, to enable large\-scale training without human\-in\-the\-loop bottlenecks\(Shenget al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib43); Zhenget al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib29); Cobbeet al\.,[2021](https://arxiv.org/html/2607.03126#bib.bib66)\)\.

Central to this shift is Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.,[2024](https://arxiv.org/html/2607.03126#bib.bib62)\)\. By utilizing group\-wise relative rewards without a separate value network, GRPO significantly lowers the computational barrier for RL training\(DeepSeek\-AIet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib21); Baiet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib22)\)\. However, despite its efficiency, GRPO and similar outcome\-based methods\(Zhaoet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib27); Liet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib31)\)face a critical limitation: the sparsity of reward signals\. In long Chain\-of\-Thought \(CoT\) trajectories\(Weiet al\.,[2022](https://arxiv.org/html/2607.03126#bib.bib63)\), a scalar outcome is typically available only at the end\. Assigning this coarse signal indiscriminately to all tokens—ignoring their individual contributions to the correct or incorrect reasoning path—can introduce substantial variance and instability during training\(Guoet al\.,[2025b](https://arxiv.org/html/2607.03126#bib.bib30); Zhuet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib32)\)\.

To mitigate this credit assignment problem, two main paths have emerged\. The first relies on Process Reward Models \(PRMs\)\(Lightmanet al\.,[2023](https://arxiv.org/html/2607.03126#bib.bib81); Cuiet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib25)\)for dense, step\-by\-step feedback\. While effective, PRMs are hindered by prohibitive annotation costs and scalability issues\(Lightmanet al\.,[2023](https://arxiv.org/html/2607.03126#bib.bib81)\)\.

The second path, which we follow, pursues implicit algorithmic credit assignment\. Earlier methods such as DAPO\(Yuet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib28)\)and SAPO\(Gaoet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib3)\)modulate policy updates using asymmetric clipping or soft gating based on outcome advantages\(Wanget al\.,[2025a](https://arxiv.org/html/2607.03126#bib.bib33); Chenet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib26); Xiet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib35)\)\. Recognizing that pure outcome metrics ignore internal model states, recent works explore token entropy, or predictive uncertainty, as an importance proxy, hypothesizing that high\-uncertainty tokens correspond to critical reasoning “forking points”\(Wanget al\.,[2025b](https://arxiv.org/html/2607.03126#bib.bib41); Huanget al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib42); Tanget al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib82)\)\.

However, existing entropy\-based approaches still face two key limitations\. First, some methods rely on heuristic “hard buckets”, such as uniformly treating the top 20% highest\-entropy tokens, which may lack the granularity needed for complex reasoning\(Haoet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib34); Wanget al\.,[2025b](https://arxiv.org/html/2607.03126#bib.bib41)\)\. Second, directly integrating true entropy into the optimization objective can introduce an optimization mismatch\. As we analyze in Section[4\.3](https://arxiv.org/html/2607.03126#S4.SS3), true entropy contributes a full\-distribution gradient component that is not necessarily aligned with sampled\-token policy updates, and may introduce noisy or competing update signals\(Liuet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib40)\)\. Moreover, even when entropy is detached and used only as a scalar weight, full\-vocabulary entropy remains sensitive to tail\-distribution noise, motivating a more mode\-local uncertainty proxy\.

To address these challenges, we proposeAdaptive Credit Policy Optimization \(ACPO\)\. Our contributions are as follows:

- •Fine\-grained adaptive credit assignment\.We introduce a token\-level weighting scheme that dynamically modulates advantage estimates\. We treat positive\- and non\-positive\-advantage trajectories asymmetrically: emphasizing uncertain decisions in successful trajectories while penalizing overconfident tokens in failed ones\.
- •Surrogate entropy formulation\.We identify the optimization mismatch that arises when directly using true entropy for credit assignment\. To address this issue, we propose a mode\-local surrogate entropy formulation\. We show that this proxy admits deterministic entropy bounds and, under the empirically observed modal\-alignment and proximal\-update regime, yields a leading\-order modulation term that preserves the intended policy\-gradient direction\.
- •Empirical effectiveness\.We validate ACPO across three base models, including Qwen2\.5\-Math\-7B, DeepSeek\-R1\-Distill, and Qwen3\-8B\-Base, on competitive reasoning benchmarks\. ACPO consistently outperforms strong RL baselines, including GRPO, DAPO, and SAPO, demonstrating the effectiveness of surrogate\-entropy\-based credit assignment under sparse outcome supervision\.

## 2Related Work

### 2\.1From RLHF to RLVR

Traditional RLHF typically relies on PPO\(Ouyanget al\.,[2022](https://arxiv.org/html/2607.03126#bib.bib64); Schulmanet al\.,[2017](https://arxiv.org/html/2607.03126#bib.bib61)\)or critic\-free preference learning methods such as DPO\(Rafailovet al\.,[2023](https://arxiv.org/html/2607.03126#bib.bib50)\)\. However, aligning reasoning models increasingly favors objective, verifiable outcome supervision\. Central to this shift is Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.,[2024](https://arxiv.org/html/2607.03126#bib.bib62); Guoet al\.,[2025a](https://arxiv.org/html/2607.03126#bib.bib2)\), which replaces the critic network with the mean reward of sampled outputs and enables efficient iterative training\. Recent variants further stabilize this paradigm: DAPO\(Yuet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib28)\)employs decoupled clipping for large\-scale updates, while SAPO\(Gaoet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib3)\)introduces a soft, temperature\-controlled gating mechanism to handle off\-policy data\. Our work builds directly upon the SAPO formulation, extending its sequence\-level gating with intrinsic uncertainty signals for fine\-grained, token\-level credit assignment\.

### 2\.2The Credit Assignment Bottleneck

A persistent challenge in reasoning tasks is the “credit assignment” problem—determining which specific steps in a reasoning chain led to the final outcome\(Guoet al\.,[2025b](https://arxiv.org/html/2607.03126#bib.bib30); Zhanget al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib38)\)\.

Explicit supervision with PRMs\.Process Reward Models \(PRMs\) provide dense feedback by scoring individual reasoning steps and can outperform Outcome Reward Models \(ORMs\) in reasoning tasks\(Lightmanet al\.,[2023](https://arxiv.org/html/2607.03126#bib.bib81); Cuiet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib25)\)\. However, training PRMs often requires expensive dense human annotation or complex automated verifiers\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.03126#bib.bib47); Cobbeet al\.,[2021](https://arxiv.org/html/2607.03126#bib.bib66)\)\.

Implicit algorithmic supervision\.To provide finer\-grained supervision without prohibitive annotation costs, researchers have turned to implicit signals\. Methods such as CISPO\(Chenet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib26)\)and BAPO\(Xiet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib35)\)introduce granular constraints to regulate generation at the token level\. Other approaches leverage gradient norms or likelihood ratios to estimate token importance dynamically\(Liet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib31); Zhaoet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib27)\)\.

### 2\.3Entropy\-Aware Reinforcement Learning

Entropy has been widely used in RL\-based language model alignment\. One line of work uses entropy as an observational signal\. Methods such as the “80/20 rule”\(Wanget al\.,[2025b](https://arxiv.org/html/2607.03126#bib.bib41)\), Archer\(Huanget al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib42)\), A3PO\(Tanget al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib82)\), and GTPO\(Simoniet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib19)\)leverage policy entropy to identify uncertain tokens, reweight advantages, or filter completions\. While effective, these methods typically treat entropy as a detached heuristic rather than a differentiable optimization objective\.

Another line of work directly adds entropy regularization to the policy objective, as in PPO\-style entropy bonuses\(Schulmanet al\.,[2017](https://arxiv.org/html/2607.03126#bib.bib61)\)\. In autoregressive LLMs, this entropy is defined over the next\-token distribution at each decoding step\. However, it is typically aggregated via a global coefficient, serving as a generic exploration regularizer rather than a reward\-aware, token\-level credit signal\. Moreover, directly optimizing true entropy can introduce non\-local gradient components that are not necessarily aligned with sampled\-token policy updates, as we detail in Section[4\.3](https://arxiv.org/html/2607.03126#S4.SS3)\.

ACPO combines aspects of these two directions by using token\-level uncertainty for fine\-grained credit assignment while replacing full\-vocabulary entropy with a mode\-local surrogate\. This design enables asymmetric updates that emphasize uncertain tokens in positive\-advantage trajectories and penalize overconfident tokens in non\-positive\-advantage trajectories\.

## 3Preliminaries

Notation\.Letπθ\\pi\_\{\\theta\}be a stochastic policy that maps a query \(prompt\)𝐪\\mathbf\{q\}to a response𝐲\\mathbf\{y\}\. The likelihood of generating a response𝐲\\mathbf\{y\}of length\|𝐲\|\|\\mathbf\{y\}\|is given by:

πθ​\(𝐲\|𝐪\)=∏t=1\|𝐲\|πθ​\(yt\|𝐪,𝐲<t\)\.\\pi\_\{\\theta\}\(\\mathbf\{y\}\|\\mathbf\{q\}\)=\\prod\_\{t=1\}^\{\|\\mathbf\{y\}\|\}\\pi\_\{\\theta\}\(y\_\{t\}\|\\mathbf\{q\},\\mathbf\{y\}\_\{<t\}\)\.\(1\)Each pair\(𝐪,𝐲\)\(\\mathbf\{q\},\\mathbf\{y\}\)receives a scalar rewardr​\(𝐪,𝐲\)r\(\\mathbf\{q\},\\mathbf\{y\}\)\. For each query, we generateGGresponse samples and letℛ𝐪=\{r​\(𝐪,𝐲1\),…,r​\(𝐪,𝐲G\)\}\\mathcal\{R\}\_\{\\mathbf\{q\}\}=\\\{r\(\\mathbf\{q\},\\mathbf\{y\}\_\{1\}\),\\dots,r\(\\mathbf\{q\},\\mathbf\{y\}\_\{G\}\)\\\}denote the set of their corresponding rewards\.

Entropy\.For theii\-th response𝐲i\\mathbf\{y\}\_\{i\}, the uncertainty of the policy at each decoding stepttis quantified by the Shannon entropy of the conditional distribution over the vocabulary𝒱\\mathcal\{V\}\. Formally, the step\-wise entropyHi,t​\(θ\)H\_\{i,t\}\(\\theta\)is defined as:

Hi,t​\(θ\)=−∑v∈𝒱πθ​\(v\|𝐪,𝐲i,<t\)​log⁡πθ​\(v\|𝐪,𝐲i,<t\)\.H\_\{i,t\}\(\\theta\)=\-\\sum\_\{v\\in\\mathcal\{V\}\}\\pi\_\{\\theta\}\(v\|\\mathbf\{q\},\\mathbf\{y\}\_\{i,<t\}\)\\log\\pi\_\{\\theta\}\(v\|\\mathbf\{q\},\\mathbf\{y\}\_\{i,<t\}\)\.\(2\)The gradient of the entropy is:

∇θHi,t​\(θ\)=−∑v∈𝒱∇θπθ​\(v\|𝐪,𝐲i,<t\)​log⁡πθ​\(v\|𝐪,𝐲i,<t\),\\nabla\_\{\\theta\}H\_\{i,t\}\(\\theta\)=\-\\sum\_\{v\\in\\mathcal\{V\}\}\\nabla\_\{\\theta\}\\pi\_\{\\theta\}\(v\|\\mathbf\{q\},\\mathbf\{y\}\_\{i,<t\}\)\\log\\pi\_\{\\theta\}\(v\|\\mathbf\{q\},\\mathbf\{y\}\_\{i,<t\}\),\(3\)where the intermediate\+1\+1term from the derivative ofx​log⁡xx\\log xvanishes because probabilities sum to one, i\.e\.,∑v∈𝒱∇θπθ​\(v\|𝐪,𝐲i,<t\)=∇θ\(1\)=0\\sum\_\{v\\in\\mathcal\{V\}\}\\nabla\_\{\\theta\}\\pi\_\{\\theta\}\(v\|\\mathbf\{q\},\\mathbf\{y\}\_\{i,<t\}\)=\\nabla\_\{\\theta\}\(1\)=0\.

Soft Adaptive Policy Optimization \(SAPO\)\.SAPO refines GSPO’s\(Zhenget al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib29)\)sequence\-level importance weights by replacing hard clipping with a smooth gating mechanism, making the objective sequence\-coherent while retaining token\-level adaptivity:

𝒥SAPO​\(θ\)=𝔼𝐪∼𝒟,\{𝐲i\}i=1G∼πθold\(⋅\|𝐪\)​\[1G​∑i=1G1\|𝐲i\|​∑t=1\|𝐲i\|fi,t​\(wi,t​\(θ\)\)​A^i,t\],\\mathcal\{J\}\_\{\\text\{SAPO\}\}\(\\theta\)=\\mathbb\{E\}\_\{\\mathbf\{q\}\\sim\\mathcal\{D\},\\,\\\{\\mathbf\{y\}\_\{i\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\cdot\|\\mathbf\{q\}\)\}\\left\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{\|\\mathbf\{y\}\_\{i\}\|\}\\sum\_\{t=1\}^\{\|\\mathbf\{y\}\_\{i\}\|\}f\_\{i,t\}\\big\(w\_\{i,t\}\(\\theta\)\\big\)\\,\\widehat\{A\}\_\{i,t\}\\right\],\(4\)where

wi,t​\(θ\)=πθ​\(yi,t\|𝐪,𝐲i,<t\)πθold​\(yi,t\|𝐪,𝐲i,<t\)w\_\{i,t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(y\_\{i,t\}\|\\mathbf\{q\},\\mathbf\{y\}\_\{i,<t\}\)\}\{\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(y\_\{i,t\}\|\\mathbf\{q\},\\mathbf\{y\}\_\{i,<t\}\)\}\(5\)denotes the step\-level importance sampling weight\. Without a separate value function, the advantage estimateA^i,t\\widehat\{A\}\_\{i,t\}is typically sequence\-level, i\.e\.,A^i,t=A^i\\widehat\{A\}\_\{i,t\}=\\widehat\{A\}\_\{i\}, and standardized within the group:

A^i,t=A^i=r​\(𝐪,𝐲i\)−mean⁡\(ℛ𝐪\)std⁡\(ℛ𝐪\)\+ϵ,\\widehat\{A\}\_\{i,t\}=\\widehat\{A\}\_\{i\}=\\frac\{r\(\\mathbf\{q\},\\mathbf\{y\}\_\{i\}\)\-\\operatorname\{mean\}\(\\mathcal\{R\}\_\{\\mathbf\{q\}\}\)\}\{\\operatorname\{std\}\(\\mathcal\{R\}\_\{\\mathbf\{q\}\}\)\+\\epsilon\},\(6\)whereϵ\\epsilonis a small constant for numerical stability\. Using the standard sigmoid functionσ​\(⋅\)\\sigma\(\\cdot\), the soft gating functionfi,t​\(⋅\)f\_\{i,t\}\(\\cdot\)relies on an asymmetric temperatureτi,t\\tau\_\{i,t\}determined by the sign of the advantage:

fi,t​\(x\)=σ​\(τi,t​\(x−1\)\)⋅4τi,t,whereτi,t=\{τpos,A^i,t\>0,τneg,otherwise\.f\_\{i,t\}\(x\)=\\sigma\\left\(\\tau\_\{i,t\}\(x\-1\)\\right\)\\cdot\\frac\{4\}\{\\tau\_\{i,t\}\},\\quad\\text\{where\}\\quad\\tau\_\{i,t\}=\\begin\{cases\}\\tau\_\{\\text\{pos\}\},&\\widehat\{A\}\_\{i,t\}\>0,\\\\ \\tau\_\{\\text\{neg\}\},&\\text\{otherwise\.\}\\end\{cases\}\(7\)

## 4Method

### 4\.1Problem Formulation: Granularity Mismatch

Standard RLVR methods \(e\.g\., GRPO\) apply the trajectory\-level rewardR​\(𝐲\)R\(\\mathbf\{y\}\)uniformly to all tokens\. However, this ignores the non\-uniform contributions of individual reasoning steps\. We posit alatent reward decomposition,R​\(𝐲\)=∑t=1Trt∗R\(\\mathbf\{y\}\)=\\sum\_\{t=1\}^\{T\}r\_\{t\}^\{\*\}, wherert∗r\_\{t\}^\{\*\}denotes the unobservable contribution of tokenyty\_\{t\}\. To address this granularity mismatch, we seek a step\-wise weightαt\\alpha\_\{t\}such that the weighted sparse signal approximates the latent dense signal by minimizing the mean squared error \(MSE\):

minαt⁡𝒥​\(αt\)=𝔼𝐲​\[‖αt​R​\(𝐲\)−rt∗‖2\]\.\\min\_\{\\alpha\_\{t\}\}\\mathcal\{J\}\(\\alpha\_\{t\}\)=\\mathbb\{E\}\_\{\\mathbf\{y\}\}\\left\[\\left\\\|\\alpha\_\{t\}R\(\\mathbf\{y\}\)\-r\_\{t\}^\{\*\}\\right\\\|^\{2\}\\right\]\.\(8\)The optimal solutionαt∗=𝔼​\[rt∗​R​\(𝐲\)\]/𝔼​\[R​\(𝐲\)2\]\\alpha\_\{t\}^\{\*\}=\\mathbb\{E\}\[r\_\{t\}^\{\*\}R\(\\mathbf\{y\}\)\]/\\mathbb\{E\}\[R\(\\mathbf\{y\}\)^\{2\}\]indicates that credit should be proportional to the token’s latent importance\. Sincert∗r\_\{t\}^\{\*\}is unobservable, we consider predictive uncertainty a natural proxy\. In reasoning tasks, while “filler” tokens \(e\.g\., “Therefore,”\) tend to exhibit high confidence \(low entropy\), critical logical pivots and post\-error states often coincide with spikes in the step\-wise conditional entropyH\(πθ\(⋅∣𝐪,𝐲i,<t\)\)H\(\\pi\_\{\\theta\}\(\\cdot\\mid\\mathbf\{q\},\\mathbf\{y\}\_\{i,<t\}\)\)\. As shown in Section 5\.1, under the mass\-concentration property of LLMs, a mode\-based uncertainty proxy provides a bounded and empirically correlated approximation to the step\-wise entropy signal\.

### 4\.2Empirical Observation: Entropy Spikes at Reasoning Errors

To examine whether step\-wise entropy provides an informative proxy for token\-level contribution \(αt\\alpha\_\{t\}\), we conduct a fine\-grained rollout analysis on AIME24 and AIME25\. We align paired correct and incorrect trajectories by the manually identified first reasoning error in the incorrect trajectory, set this divergence point as the origin \(t=0t=0\), and compare their local entropy dynamics around the aligned region\. The full annotation protocol, including rollout filtering, trajectory pairing, cross\-model verification, and manual review, is provided in Appendix[A\.1](https://arxiv.org/html/2607.03126#A1.SS1)\.

This pattern is illustrated by the qualitative case study in Figure[1](https://arxiv.org/html/2607.03126#S4.F1)a\. At the divergence point, the incorrect trajectory introduces an ungrounded variable \(aa\), after which the entropy rises sharply and remains elevated \(visualized in red/orange\)\. In contrast, the correct trajectory—which factors the cubic polynomial to solve foraa—maintains lower entropy and a more confident profile \(visualized in blue\)\.

A broader statistical pattern corroborates this observation, as shown in Figure[1](https://arxiv.org/html/2607.03126#S4.F1)b\. Before the reasoning error \(t<0t<0\), both correct and incorrect trajectories exhibit relatively low and stable entropy\. Immediately after the error \(t≥0t\\geq 0\), however, the entropy of the incorrect trajectory increases sharply and remains elevated, whereas the correct trajectory stays comparatively stable\. To confirm that this phenomenon is not merely a local artifact of alignment, we provide extended visualizations of unaligned, full\-trajectory entropy profiles across multiple AIME questions in Appendix[A\.2](https://arxiv.org/html/2607.03126#A1.SS2), which show the broader trajectory\-level volatility associated with a single reasoning error\.

![Refer to caption](https://arxiv.org/html/2607.03126v1/x1.png)Figure 1:Empirical analysis of token entropy around the first reasoning error\.\(a\) Token\-level entropy visualization\. At the divergence point highlighted by the boxes, the incorrect trajectory introduces an ungrounded variable \(aa\), after which subsequent tokens exhibit substantially higher entropy, shown in red/orange\. In contrast, the correct trajectory derives the solution by factoring a cubic polynomial and maintains lower entropy, shown in blue\. \(b\) Aggregated entropy profile\. After aligning trajectories at the manually annotated first reasoning error \(t=0t=0\), incorrect trajectories exhibit a sharp and sustained entropy increase, whereas correct trajectories remain comparatively stable and low\-entropy\.
### 4\.3Optimization Mismatch of True Entropy

Given its empirical usefulness as an uncertainty signal, the true entropyHi,t​\(θ\)H\_\{i,t\}\(\\theta\)in Eq\.[2](https://arxiv.org/html/2607.03126#S3.E2)appears to be a natural candidate for the credit weightαt\\alpha\_\{t\}\. However, directly incorporating true entropy into the optimization objective introduces a non\-local gradient component that is not necessarily aligned with the sampled\-token policy update\. To see this, consider the gradient of a single\-step term modulated by true entropy:

∇θ\(fi,t​\(wi,t​\(θ\)\)⋅Hi,t​\(θ\)\)=Hi,t​\(θ\)⋅∇θfi,t​\(wi,t​\(θ\)\)⏟Entropy\-scaled policy update\+fi,t​\(wi,t​\(θ\)\)⋅∇θHi,t​\(θ\)⏟Full\-distribution entropy update\.\\nabla\_\{\\theta\}\\Big\(f\_\{i,t\}\(w\_\{i,t\}\(\\theta\)\)\\cdot H\_\{i,t\}\(\\theta\)\\Big\)=\\underbrace\{H\_\{i,t\}\(\\theta\)\\cdot\\nabla\_\{\\theta\}f\_\{i,t\}\(w\_\{i,t\}\(\\theta\)\)\}\_\{\\text\{Entropy\-scaled policy update\}\}\+\\underbrace\{f\_\{i,t\}\(w\_\{i,t\}\(\\theta\)\)\\cdot\\nabla\_\{\\theta\}H\_\{i,t\}\(\\theta\)\}\_\{\\text\{Full\-distribution entropy update\}\}\.\(9\)
The first term preserves the sampled\-token structure of the policy update: for positive\-advantage samples, it increases the probability of the sampled token through∇θfi,t\\nabla\_\{\\theta\}f\_\{i,t\}\. In contrast, the second term comes from the entropy gradient in Eq\.[3](https://arxiv.org/html/2607.03126#S3.E3), which acts on the entire next\-token distribution\. For positive samples, this entropy\-gradient term encourages distributional flattening, whereas the policy\-gradient term tends to concentrate probability mass around the sampled reasoning step\. Thus, directly optimizing true entropy does not merely rescale the policy update; it also injects an additional full\-vocabulary update component\.

This mismatch is undesirable for token\-level credit assignment, whose goal is to use uncertainty to modulate the sampled\-token update rather than to optimize a global exploration regularizer at every reasoning step\. A natural alternative is to detach entropy and use it only as a scalar weight\. While this removes the entropy\-gradient term in Eq\.[9](https://arxiv.org/html/2607.03126#S4.E9), true entropy remains sensitive to how residual probability mass is distributed over the long vocabulary tail\. Truncated or top\-kkdetached variants can reduce this tail effect, but they introduce an additional hyperparameter and still aggregate uncertainty over secondary candidates\. These considerations motivate a simpler mode\-local proxy that focuses on the model’s confidence in its dominant prediction\.

### 4\.4Surrogate Entropy: A Mode\-Local Uncertainty Proxy

To obtain a more local uncertainty signal, we introducesurrogate entropy, denoted byδi,t​\(θ\)\\delta\_\{i,t\}\(\\theta\)\.

#### Definition\.

Letvi,t∗=arg⁡maxv∈𝒱⁡πθ​\(v∣𝐪,𝐲i,<t\)v^\{\*\}\_\{i,t\}=\\arg\\max\_\{v\\in\\mathcal\{V\}\}\\pi\_\{\\theta\}\(v\\mid\\mathbf\{q\},\\mathbf\{y\}\_\{i,<t\}\)be the dominant mode\. We define the surrogate entropy as the complementary probability of this mode:

δi,t​\(θ\)=1−πθ​\(vi,t∗∣𝐪,𝐲i,<t\)\.\\delta\_\{i,t\}\(\\theta\)=1\-\\pi\_\{\\theta\}\\\!\\left\(v^\{\*\}\_\{i,t\}\\mid\\mathbf\{q\},\\mathbf\{y\}\_\{i,<t\}\\right\)\.\(10\)By focusing on the dominant mode,δi,t​\(θ\)\\delta\_\{i,t\}\(\\theta\)provides a mode\-local uncertainty signal\. It directly measures whether the model is confident in its most likely next\-token choice, which is especially relevant under the modal\-alignment phenomenon analyzed in Section[5\.2](https://arxiv.org/html/2607.03126#S5.SS2)\. Compared with full or top\-kkentropy variants, this proxy avoids aggregating uncertainty over secondary candidates and remains invariant to how the residual probability mass is dispersed\. As analyzed in Section 5\.1,δi,t​\(θ\)\\delta\_\{i,t\}\(\\theta\)admits deterministic entropy bounds and tracks the macroscopic uncertainty trend of Shannon entropy under mass\-concentrated LLM distributions\.

#### Gradient behavior\.

During backpropagation, we treat the selected modevi,t∗v^\{\*\}\_\{i,t\}as locally fixed, yielding∇θδi,t=−∇θπθ​\(vi,t∗\)\\nabla\_\{\\theta\}\\delta\_\{i,t\}=\-\\nabla\_\{\\theta\}\\pi\_\{\\theta\}\(v^\{\*\}\_\{i,t\}\)almost everywhere\. This gradient acts only on the dominant mode rather than the entire vocabulary distribution\. As shown in Section[5\.2](https://arxiv.org/html/2607.03126#S5.SS2), sampled tokens frequently coincide with the dominant mode, a phenomenon we refer to asmodal alignment\. Under this modal\-aligned and proximal\-update regime, the surrogate mainly acts as a local modulation factor\. In the leading\-order approximation, it changes the magnitude of the reinforcement or penalization signal without reversing its intended policy\-gradient direction\.

### 4\.5Adaptive Credit Policy Optimization \(ACPO\)

The final objective ofAdaptive Credit Policy Optimization \(ACPO\)integrates surrogate entropy with an asymmetric credit assignment strategy:

𝒥ACPO​\(θ\)=𝔼𝐪∼𝒟,\{𝐲i\}i=1G∼πθold\(⋅\|𝐪\)​\[1G​∑i=1G1\|𝐲i\|​∑t=1\|𝐲i\|fi,t​\(wi,t​\(θ\)\)⋅A^i,tACPO\]\.\\mathcal\{J\}\_\{\\text\{ACPO\}\}\(\\theta\)=\\mathbb\{E\}\_\{\\mathbf\{q\}\\sim\\mathcal\{D\},\\,\\\{\\mathbf\{y\}\_\{i\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\cdot\|\\mathbf\{q\}\)\}\\left\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{\|\\mathbf\{y\}\_\{i\}\|\}\\sum\_\{t=1\}^\{\|\\mathbf\{y\}\_\{i\}\|\}f\_\{i,t\}\(w\_\{i,t\}\(\\theta\)\)\\cdot\\widehat\{A\}\_\{i,t\}^\{\\text\{ACPO\}\}\\right\]\.\(11\)The core component is the entropy\-modulated advantageA^i,tACPO\\widehat\{A\}\_\{i,t\}^\{\\text\{ACPO\}\}, defined as:

A^i,tACPO=\{\(2\+δi,t​\(θ\)\)⋅A^i,if​A^i\>0,\(1−δi,t​\(θ\)\)⋅A^i,otherwise\.\\widehat\{A\}\_\{i,t\}^\{\\text\{ACPO\}\}=\\begin\{cases\}\(2\+\\delta\_\{i,t\}\(\\theta\)\)\\cdot\\widehat\{A\}\_\{i\},&\\text\{if \}\\widehat\{A\}\_\{i\}\>0,\\\\ \(1\-\\delta\_\{i,t\}\(\\theta\)\)\\cdot\\widehat\{A\}\_\{i\},&\\text\{otherwise\.\}\\end\{cases\}\(12\)
#### Asymmetric modulation\.

The functional form ofA^i,tACPO\\widehat\{A\}\_\{i,t\}^\{\\text\{ACPO\}\}is motivated by two complementary principles:

1. 1\.Positive\-advantage trajectories\.For trajectories with positive advantage, we assign larger weights to high\-uncertainty tokens, encouraging the model to reinforce uncertain but successful reasoning decisions\. The constant offset 2 acts as a positivity\-preserving margin; under the modal\-alignment and proximal\-update approximation analyzed in Section[5\.2](https://arxiv.org/html/2607.03126#S5.SS2), it keeps the leading\-order modulation factor positive, reducing the risk of gradient\-direction reversal\.
2. 2\.Non\-positive\-advantage trajectories\.For trajectories with non\-positive advantage, we aim to penalize overconfident mistakes while reducing the influence of uncertain post\-error tokens\. As illustrated in Figure[1](https://arxiv.org/html/2607.03126#S4.F1), incorrect trajectories often contain low\-uncertainty erroneous decisions followed by high\-entropy continuation noise\. ACPO uses1−δi,t​\(θ\)1\-\\delta\_\{i,t\}\(\\theta\)to assign larger penalties to confident tokens and smaller penalties to highly uncertain tokens\.

The effectiveness of this asymmetric design is examined in the ablation study in Figure[3](https://arxiv.org/html/2607.03126#S6.F3)\. The default ACPO configuration,\(2\+δ,1−δ\)\(2\+\\delta,1\-\\delta\), achieves stronger accuracy and stability than its variants, suggesting that successful uncertain steps and overconfident erroneous steps should be treated differently in token\-level credit assignment\.

## 5Analysis of the Surrogate Entropy

### 5\.1Bounding the Approximation Error

For brevity, letπθ​\(v\):=πθ​\(v\|𝐪,𝐲i,<t\)\\pi\_\{\\theta\}\(v\):=\\pi\_\{\\theta\}\(v\|\\mathbf\{q\},\\mathbf\{y\}\_\{i,<t\}\)\. We define the surrogate entropy asδi,t​\(θ\)=1−πθ​\(v∗\)∈\[0,\|𝒱\|−1\|𝒱\|\]\\delta\_\{i,t\}\(\\theta\)=1\-\\pi\_\{\\theta\}\(v^\{\*\}\)\\in\[0,\\frac\{\|\\mathcal\{V\}\|\-1\}\{\|\\mathcal\{V\}\|\}\], wherev∗v^\{\*\}is the distribution mode\. We first establish that the true entropyHi,t​\(θ\)H\_\{i,t\}\(\\theta\)is bounded above and below by functions ofδi,t​\(θ\)\\delta\_\{i,t\}\(\\theta\)\.

#### Entropy bounds\.

The true entropyHi,t​\(θ\)H\_\{i,t\}\(\\theta\)satisfiesL​\(δi,t​\(θ\)\)≤Hi,t​\(θ\)≤U​\(δi,t​\(θ\)\)L\(\\delta\_\{i,t\}\(\\theta\)\)\\leq H\_\{i,t\}\(\\theta\)\\leq U\(\\delta\_\{i,t\}\(\\theta\)\), where:

L​\(δ\)\\displaystyle L\(\\delta\)≜−log⁡\(1−δ\),\\displaystyle\\triangleq\-\\log\(1\-\\delta\),\(13\)U​\(δ\)\\displaystyle U\(\\delta\)≜−\(1−δ\)​log⁡\(1−δ\)−δ​log⁡\(δ\|𝒱\|−1\)\.\\displaystyle\\triangleq\-\(1\-\\delta\)\\log\(1\-\\delta\)\-\\delta\\log\\left\(\\frac\{\\delta\}\{\|\\mathcal\{V\}\|\-1\}\\right\)\.\(14\)
Detailed derivations are provided in Appendix[B](https://arxiv.org/html/2607.03126#A2)\. To assess the tightness of these bounds, we analyze the envelope widthg​\(δ\)≜U​\(δ\)−L​\(δ\)g\(\\delta\)\\triangleq U\(\\delta\)\-L\(\\delta\)\. While the worst\-case bound scales with\|𝒱\|\|\\mathcal\{V\}\|, practical LLM distributions often exhibit strong mass concentration\(Holtzmanet al\.,[2019](https://arxiv.org/html/2607.03126#bib.bib16)\)\. As detailed in Appendix[B\.4](https://arxiv.org/html/2607.03126#A2.SS4), when the effective support is restricted to typical top\-kkregions, this gap remains controlled, e\.g\., below10%10\\%relative error fork=10k=10\.

#### Remark on monotonicity\.

These bounds do not imply strict monotonicity under arbitrary tail perturbations\. This limitation is consistent with the design goal ofδi,t​\(θ\)\\delta\_\{i,t\}\(\\theta\): rather than tracking all tail\-distribution variations, the surrogate focuses on the model’s confidence in its dominant choice\. True entropy can be sensitive to residual probability mass dispersed over the vocabulary tail, whereasδi,t​\(θ\)\\delta\_\{i,t\}\(\\theta\)is invariant to the tail shape and maintains strong macroscopic correlation with entropy in mass\-concentrated distributions, as shown in Figure[5](https://arxiv.org/html/2607.03126#A2.F5)\.

### 5\.2Gradient Analysis: The Hybrid Modulation Mechanism

We analyze the ACPO gradient to clarify howδi,t​\(θ\)\\delta\_\{i,t\}\(\\theta\)modulates the policy update\. For positive\-advantage samples \(A^i\>0\\widehat\{A\}\_\{i\}\>0\), omitting the constant advantage factor, the gradient of the token\-level term can be decomposed by the product rule:

∇θ\[fi,t​\(wi,t​\(θ\)\)⋅\(2\+δi,t​\(θ\)\)\]=\(2\+δi,t​\(θ\)\)​∇θfi,t​\(wi,t​\(θ\)\)⏟policy update\+fi,t​\(wi,t​\(θ\)\)​∇θδi,t​\(θ\)⏟surrogate\-entropy modulation\.\\displaystyle\\nabla\_\{\\theta\}\\Big\[f\_\{i,t\}\(w\_\{i,t\}\(\\theta\)\)\\cdot\\big\(2\+\\delta\_\{i,t\}\(\\theta\)\\big\)\\Big\]=\\underbrace\{\\big\(2\+\\delta\_\{i,t\}\(\\theta\)\\big\)\\nabla\_\{\\theta\}f\_\{i,t\}\(w\_\{i,t\}\(\\theta\)\)\}\_\{\\text\{policy update\}\}\+\\underbrace\{f\_\{i,t\}\(w\_\{i,t\}\(\\theta\)\)\\nabla\_\{\\theta\}\\delta\_\{i,t\}\(\\theta\)\}\_\{\\text\{surrogate\-entropy modulation\}\}\.\(15\)
#### Justification via modal alignment\.

Expanding Eq\.[15](https://arxiv.org/html/2607.03126#S5.E15)requires considering the relationship between the sampled tokenyi,ty\_\{i,t\}and the dominant modev∗v^\{\*\}\. Our empirical analysis in Table[4](https://arxiv.org/html/2607.03126#A3.T4)shows a modal\-alignment phenomenon: sampled tokens coincide with the greedy mode in83\.90%83\.90\\%of cases\.

Leveraging this observation, we adopt the approximationπθ​\(yi,t\)≈πθ​\(v∗\)\\pi\_\{\\theta\}\(y\_\{i,t\}\)\\approx\\pi\_\{\\theta\}\(v^\{\*\}\)\. Under the proximal\-update regime \(wi,t≈1w\_\{i,t\}\\approx 1\), the ACPO gradient admits the following leading\-order form, with the detailed derivation provided in Appendix[C](https://arxiv.org/html/2607.03126#A3):

∇θ𝒥ACPO∝𝔼​\[∇θlog⁡πθ​\(yi,t\)⋅A^i⋅ℳi,t\],\\displaystyle\\nabla\_\{\\theta\}\\mathcal\{J\}\_\{\\text\{ACPO\}\}\\propto\\mathbb\{E\}\\left\[\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{i,t\}\)\\cdot\\widehat\{A\}\_\{i\}\\cdot\\mathcal\{M\}\_\{i,t\}\\right\],\(16\)where the hybrid modulation factorℳi,t\\mathcal\{M\}\_\{i,t\}is:

ℳi,t≈\{δi,t​\(θ\)\+2​δi,t​\(θold\),if​A^i\>0,3−δi,t​\(θ\)−2​δi,t​\(θold\),if​A^i≤0\.\\mathcal\{M\}\_\{i,t\}\\approx\\begin\{cases\}\\delta\_\{i,t\}\(\\theta\)\+2\\delta\_\{i,t\}\(\\theta\_\{\\text\{old\}\}\),&\\text\{if \}\\widehat\{A\}\_\{i\}\>0,\\\\ 3\-\\delta\_\{i,t\}\(\\theta\)\-2\\delta\_\{i,t\}\(\\theta\_\{\\text\{old\}\}\),&\\text\{if \}\\widehat\{A\}\_\{i\}\\leq 0\.\\end\{cases\}\(17\)Forδi,t​\(θ\),δi,t​\(θold\)∈\[0,1\]\\delta\_\{i,t\}\(\\theta\),\\delta\_\{i,t\}\(\\theta\_\{\\text\{old\}\}\)\\in\[0,1\],ℳi,t\\mathcal\{M\}\_\{i,t\}is non\-negative, and it is strictly positive under the finite\-vocabulary rangeδ∈\[0,\(\|𝒱\|−1\)/\|𝒱\|\]\\delta\\in\[0,\(\|\\mathcal\{V\}\|\-1\)/\|\\mathcal\{V\}\|\]\. This indicates that, under the modal\-alignment and proximal\-update approximation, ACPO preserves the intended reinforcement or penalization direction in the leading\-order gradient\.

#### Benefits of the hybrid modulation factor\.

The hybrid structure in Eq\.[17](https://arxiv.org/html/2607.03126#S5.E17)combines the current policy’s proxyδi,t​\(θ\)\\delta\_\{i,t\}\(\\theta\)with a historical anchorδi,t​\(θold\)\\delta\_\{i,t\}\(\\theta\_\{\\text\{old\}\}\)\. This provides two useful effects\. First, it enables adaptive recalibration: as the model becomes more confident, the current uncertainty term decreases, reducing unnecessary learning pressure on already confident tokens\. Second, it introduces a damping effect in off\-policy regions: when the policy deviates substantially from the old policy, the historical term helps moderate the update, as detailed in Appendix[D](https://arxiv.org/html/2607.03126#A4)\. Together, these terms encourage credit redistribution that adapts to the current policy while empirically improving training stability\.

![Refer to caption](https://arxiv.org/html/2607.03126v1/x2.png)Figure 2:Training dynamics of SAPO and ACPO on Qwen2\.5\-Math\-7B\. ACPO shows faster early\-stage improvement and maintains higher validation accuracy in later training\.Table 1:Main Results\.Comparison of different reinforcement learning methods across three base models on five mathematical benchmarks and one code benchmark\. We evaluate performance using the mean@8 metric for mathematical benchmarks and pass@1 for code benchmark\. Our method,ACPO, consistently outperforms baselines across all tasks\.

## 6Experiments

### 6\.1Experimental Setup

We evaluate ACPO on three base models: Qwen2\.5\-Math\-7B\(Yanget al\.,[2024c](https://arxiv.org/html/2607.03126#bib.bib11)\), Qwen3\-8B\-Base\(Team,[2025](https://arxiv.org/html/2607.03126#bib.bib10)\), and DeepSeek\-R1\-Distill\-Qwen\-7B\(Guoet al\.,[2025a](https://arxiv.org/html/2607.03126#bib.bib2)\)\. All models are trained on the DAPO\-Math\-17k\(Yuet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib28)\)dataset\. For ACPO, we useτpos=1\.0\\tau\_\{\\text\{pos\}\}=1\.0,τneg=1\.05\\tau\_\{\\text\{neg\}\}=1\.05, and an actor learning rate of1×10−61\\times 10^\{\-6\}\. We generateG=8G=8samples per prompt, with a global batch size of 256 and a mini\-batch size of 16\. The maximum response length is set to 2048 for Qwen2\.5\-Math\-7B and DeepSeek\-R1\-Distill\-Qwen\-7B, and 8192 for Qwen3\-8B\-Base\.

We compare ACPO with strong outcome\-based RLVR and entropy\-aware baselines, including GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2607.03126#bib.bib62)\), DAPO\(Yuet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib28)\), SAPO\(Gaoet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib3)\), 80/20\(Wanget al\.,[2025b](https://arxiv.org/html/2607.03126#bib.bib41)\), and GTPO\(Simoniet al\.,[2025](https://arxiv.org/html/2607.03126#bib.bib19)\)\. To ensure a fair comparison, all methods are trained with the same training data, sampling configuration, and compute budget on a cluster of 8 NVIDIA H200 GPUs\. Model performance is evaluated on five mathematical reasoning benchmarks: AIME24\(Zhang and Math\-AI,[2024](https://arxiv.org/html/2607.03126#bib.bib6)\), AIME25\(Zhang and Math\-AI,[2025](https://arxiv.org/html/2607.03126#bib.bib7)\), MATH500\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.03126#bib.bib47)\), HMMT25\(Harvard\-MIT Mathematics Tournament,[2024](https://arxiv.org/html/2607.03126#bib.bib9)\), and BeyondAIME\(ByteDance\-Seed,[2025](https://arxiv.org/html/2607.03126#bib.bib8)\)\. We further evaluate code reasoning performance on HumanEval Pro\(Yuet al\.,[2024](https://arxiv.org/html/2607.03126#bib.bib83)\)\.

### 6\.2Main Results and Training Dynamics

Table[1](https://arxiv.org/html/2607.03126#S5.T1)presents the comparative accuracy across three base models on six benchmarks\. We report the results of each method at the same number of training steps\. ACPO consistently outperforms strong RL baselines across most evaluated settings, demonstrating the effectiveness of surrogate\-entropy\-based credit assignment under sparse outcome supervision\.

Figure[2](https://arxiv.org/html/2607.03126#S5.F2)illustrates the training dynamics of ACPO compared with the strong SAPO baseline\. ACPO achieves faster early\-stage improvements in both training reward and validation accuracy\. This suggests that emphasizing uncertain tokens in positive\-advantage trajectories and overconfident tokens in non\-positive\-advantage trajectories provides a more informative credit signal\. In the mid\-to\-late training stage, ACPO maintains stronger validation performance than SAPO, indicating improved training stability and a higher empirical performance ceiling\.

Table 2:Ablation study of ACPO on AIME24\. Let\(WA^i\>0,WA^i≤0\)\(W\_\{\\widehat\{A\}\_\{i\}\>0\},W\_\{\\widehat\{A\}\_\{i\}\\leq 0\}\)denote the adjustment terms in Eq\.[12](https://arxiv.org/html/2607.03126#S4.E12), whereδ\\deltais the mode\-local surrogate entropy andeeis the detached true entropy\.![Refer to caption](https://arxiv.org/html/2607.03126v1/x3.png)Figure 3:Ablation study\. This figure shows the evolution of validation accuracy and entropy on Deepseek\-R1\-Distill\-Qwen\-7B\. Let\(WA^i\>0,WA^i≤0\)\(W\_\{\\widehat\{A\}\_\{i\}\>0\},W\_\{\\widehat\{A\}\_\{i\}\\leq 0\}\)denote the adjustment terms in Eq\.[12](https://arxiv.org/html/2607.03126#S4.E12), whereδ\\deltais the surrogate entropy andeeis the detached true entropy\.
### 6\.3Ablation Study

To empirically validate the effectiveness of our weighting design, we conduct a series of ablation studies evaluating the individual contributions of the positive\- and non\-positive\-advantage channels, different weighting configurations, and the use of detached true entropy\. Figure[3](https://arxiv.org/html/2607.03126#S6.F3)illustrates the training trajectories of mean@8 accuracy on the AIME24 benchmark across these settings\.

Applying either channel alone improves over some variants but does not match the full ACPO objective, suggesting that the two asymmetric channels play complementary roles\. Among the weighting configurations, the default ACPO design,\(2\+δ,1−δ\)\(2\{\+\}\\delta,1\{\-\}\\delta\), achieves the strongest late\-stage performance and more stable training dynamics\.

We further compare surrogate entropy with detached true entropy\. Although detaching true entropy removes the direct entropy\-gradient term discussed in Section[4\.3](https://arxiv.org/html/2607.03126#S4.SS3), it remains sensitive to tail\-distribution noise in large vocabularies\. This can inflate uncertainty estimates and misallocate credit to non\-critical tokens\. In contrast, the proposed surrogate focuses only on the model’s confidence in the dominant choice, making it more robust to tail dispersion\. This trend is further supported by our top\-kkablation in Figure[6](https://arxiv.org/html/2607.03126#A5.F6): increasingk∈\{5,20,100\}k\\in\\\{5,20,100\\\}does not consistently improve performance over the top\-1 signal, suggesting that dominant\-mode uncertainty provides a concise and effective proxy for identifying reasoning forks in our setting\. A more detailed analysis is provided in Appendix[6](https://arxiv.org/html/2607.03126#A5.F6)\.

## 7Limitations

ACPO still has several limitations\. Its empirical validation is currently limited to 7B\-scale dense models and single\-turn verifiable reasoning tasks, leaving larger models, MoE architectures, and multi\-turn agentic settings unexplored\. In addition, the theoretical analysis relies on modal alignment, proximal updates, and a leading\-order approximation, so it should be interpreted as a local characterization rather than a global guarantee\.

## 8Conclusion

In this work, we introducedAdaptive Credit Policy Optimization \(ACPO\), an RLVR framework that addresses token\-level credit assignment through a mode\-local surrogate entropy and an asymmetric weighting strategy\. ACPO modulates policy updates by emphasizing uncertain decisions in positive\-advantage trajectories and overconfident tokens in non\-positive\-advantage trajectories\. Our evaluations on competitive reasoning benchmarks show that ACPO improves training dynamics and final performance over strong RL baselines\. While the proposed surrogate is theoretically analyzed and empirically effective, it represents only one design choice in the broader space of uncertainty\-aware credit assignment\. Future work will explore its scalability to larger model architectures and its applicability to multi\-turn agentic reinforcement learning scenarios, where precise credit assignment remains a central challenge\.

## References

- Y\. Bai, Y\. Bao, G\. Chen, J\. Chen, N\. Chen, R\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Z\. Chen, J\. Cui, H\. Ding, M\. Dong, A\. Du, C\. Du, D\. Du, Y\. Du, Y\. Fan, Y\. Feng, K\. Fu, B\. Gao, H\. Gao, P\. Gao, T\. Gao, X\. Gu, L\. Guan, H\. Guo, J\. Guo, H\. Hu, X\. Hao, T\. He, W\. He, W\. He, C\. Hong, Y\. Hu, Z\. Hu, W\. Huang, Z\. Huang, Z\. Huang, T\. Jiang, Z\. Jiang, X\. Jin, Y\. Kang, G\. Lai, C\. Li, F\. Li, H\. Li, M\. Li, W\. Li, Y\. Li, Y\. Li, Z\. Li, Z\. Li, H\. Lin, X\. Lin, Z\. Lin, C\. Liu, C\. Liu, H\. Liu, J\. Liu, J\. Liu, L\. Liu, S\. Liu, T\. Y\. Liu, T\. Liu, W\. Liu, Y\. Liu, Y\. Liu, Y\. Liu, Y\. Liu, Z\. Liu, E\. Lu, L\. Lu, S\. Ma, X\. Ma, Y\. Ma, S\. Mao, J\. Mei, X\. Men, Y\. Miao, S\. Pan, Y\. Peng, R\. Qin, B\. Qu, Z\. Shang, L\. Shi, S\. Shi, F\. Song, J\. Su, Z\. Su, X\. Sun, F\. Sung, H\. Tang, J\. Tao, Q\. Teng, C\. Wang, D\. Wang, F\. Wang, and H\. Wang \(2025\)Kimi K2: open agentic intelligence\.CoRRabs/2507\.20534\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p2.1)\.
- ByteDance\-Seed \(2025\)BeyondAIME: advancing math reasoning evaluation beyond high school olympiads\.Cited by:[§6\.1](https://arxiv.org/html/2607.03126#S6.SS1.p2.1)\.
- A\. Chen, A\. Li, B\. Gong, B\. Jiang, B\. Fei, B\. Yang, B\. Shan, C\. Yu, C\. Wang, C\. Zhu, C\. Xiao, C\. Du, C\. Zhang, C\. Qiao, C\. Zhang, C\. Du, C\. Guo, D\. Chen, D\. Ding, D\. Sun, D\. Li, E\. Jiao, H\. Zhou, H\. Zhang, H\. Ding, H\. Sun, H\. Feng, H\. Cai, H\. Zhu, J\. Sun, J\. Zhuang, J\. Cai, J\. Song, J\. Zhu, J\. Li, J\. Tian, J\. Liu, J\. Xu, J\. Yan, J\. Liu, J\. He, K\. Feng, K\. Yang, K\. Xiao, L\. Han, L\. Wang, L\. Yu, L\. Feng, L\. Li, L\. Zheng, L\. Du, L\. Yang, L\. Zeng, M\. Yu, M\. Tao, M\. Chi, M\. Zhang, M\. Lin, N\. Hu, N\. Di, P\. Gao, P\. Li, P\. Zhao, Q\. Ren, Q\. Xu, Q\. Li, Q\. Wang, R\. Tian, R\. Leng, S\. Chen, S\. Chen, S\. Shi, S\. Weng, S\. Guan, S\. Yu, S\. Li, S\. Zhu, T\. Li, T\. Cai, T\. Liang, W\. Cheng, W\. Kong, W\. Li, X\. Chen, X\. Song, X\. Luo, X\. Su, X\. Li, X\. Han, X\. Hou, X\. Lu, X\. Zou, X\. Shen, Y\. Gong, Y\. Ma, Y\. Wang, Y\. Shi, Y\. Zhong, and Y\. Duan \(2025\)MiniMax\-m1: scaling test\-time compute efficiently with lightning attention\.CoRRabs/2506\.13585\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p1.1),[§1](https://arxiv.org/html/2607.03126#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.03126#S2.SS2.p3.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.03126#S2.SS2.p2.1)\.
- G\. Cui, L\. Yuan, Z\. Wang, H\. Wang, W\. Li, B\. He, Y\. Fan, T\. Yu, Q\. Xu, W\. Chen, J\. Yuan, H\. Chen, K\. Zhang, X\. Lv, S\. Wang, Y\. Yao, X\. Han, H\. Peng, Y\. Cheng, Z\. Liu, M\. Sun, B\. Zhou, and N\. Ding \(2025\)Process reinforcement through implicit rewards\.CoRRabs/2502\.01456\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.03126#S2.SS2.p2.1)\.
- DeepSeek\-AI, D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi, X\. Zhang, X\. Yu, Y\. Wu, Z\. F\. Wu, Z\. Gou, Z\. Shao, Z\. Li, Z\. Gao, A\. Liu, B\. Xue, B\. Wang, B\. Wu, B\. Feng, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan, D\. Dai, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Dai, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Bao, H\. Xu, H\. Wang, H\. Ding, H\. Xin, H\. Gao, H\. Qu, H\. Li, J\. Guo, J\. Li, J\. Wang, J\. Chen, J\. Yuan, J\. Qiu, J\. Li, J\. L\. Cai, J\. Ni, J\. Liang, J\. Chen, K\. Dong, K\. Hu, K\. Gao, K\. Guan, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Zhao, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, M\. Zhang, M\. Zhang, M\. Tang, M\. Li, M\. Wang, M\. Li, N\. Tian, P\. Huang, P\. Zhang, Q\. Wang, Q\. Chen, Q\. Du, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. J\. Chen, R\. L\. Jin, R\. Chen, S\. Lu, S\. Zhou, S\. Chen, S\. Ye, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, and S\. S\. Li \(2025\)DeepSeek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.CoRRabs/2501\.12948\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p1.1),[§1](https://arxiv.org/html/2607.03126#S1.p2.1)\.
- A\. Dubey, A\. Jauhri, A\. Pandey, A\. Keshwam, A\. Al\-dahle, A\. Raman,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p1.1)\.
- K\. Ethayarajh \(2019\)How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt\-2 embeddings\.arXiv preprint arXiv:1909\.00512\.Cited by:[§B\.4](https://arxiv.org/html/2607.03126#A2.SS4.p1.2)\.
- C\. Gao, C\. Zheng, X\. Chen, K\. Dang, S\. Liu, B\. Yu, A\. Yang, S\. Bai, J\. Zhou, and J\. Lin \(2025\)Soft adaptive policy optimization\.arXiv preprint arXiv:2511\.20347\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.03126#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2607.03126#S5.T1.35.35.35.8),[§6\.1](https://arxiv.org/html/2607.03126#S6.SS1.p2.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025a\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§2\.1](https://arxiv.org/html/2607.03126#S2.SS1.p1.1),[§6\.1](https://arxiv.org/html/2607.03126#S6.SS1.p1.4)\.
- Y\. Guo, L\. Xu, J\. Liu, D\. Ye, and S\. Qiu \(2025b\)Segment policy optimization: effective segment\-level credit assignment in RL for large language models\.CoRRabs/2505\.23564\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.03126#S2.SS2.p1.1)\.
- Z\. Hao, H\. Wang, H\. Liu, J\. Luo, J\. Yu, H\. Dong, Q\. Lin, C\. Wang, and J\. Chen \(2025\)Rethinking entropy interventions in rlvr: an entropy change perspective\.External Links:2510\.10150,[Link](https://arxiv.org/abs/2510.10150)Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p5.1)\.
- Harvard\-MIT Mathematics Tournament \(2024\)Harvard\-MIT Mathematics Tournament \(HMMT\)\.Note:[https://www\.hmmt\.org](https://www.hmmt.org/)Accessed: 2024\-05\-20Cited by:[§6\.1](https://arxiv.org/html/2607.03126#S6.SS1.p2.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the MATH dataset\.InNeurIPS Datasets and Benchmarks,Cited by:[§2\.2](https://arxiv.org/html/2607.03126#S2.SS2.p2.1),[§6\.1](https://arxiv.org/html/2607.03126#S6.SS1.p2.1)\.
- A\. Holtzman, J\. Buys, and L\. Du \(2019\)THE curious case of neural text degeneration\.arXiv preprint arXiv:1904\.09751\.Cited by:[§B\.4](https://arxiv.org/html/2607.03126#A2.SS4.p1.2),[§5\.1](https://arxiv.org/html/2607.03126#S5.SS1.SSS0.Px1.p2.5)\.
- G\. Huang, T\. Xu, M\. Wang, Q\. Yi, X\. Gong, S\. Li, R\. Xiong, K\. Li, Y\. Jiang, and B\. Zhou \(2025\)Low\-probability tokens sustain exploration in reinforcement learning with verifiable reward\.CoRRabs/2510\.03222\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p4.1),[§2\.3](https://arxiv.org/html/2607.03126#S2.SS3.p1.1)\.
- Y\. Li, Q\. Gu, Z\. Wen, Z\. Li, T\. Xing, S\. Guo, T\. Zheng, X\. Zhou, X\. Qu, W\. Zhou, Z\. Zhang, W\. Shen, Q\. Liu, C\. Lin, J\. Yang, G\. Zhang, and W\. Huang \(2025\)TreePO: bridging the gap of policy optimization and efficacy and inference efficiency with heuristic tree\-based modeling\.CoRRabs/2508\.17445\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.03126#S2.SS2.p3.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2023\)Let’s verify step by step\.arXiv preprint arXiv:2305\.20050\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.03126#S2.SS2.p2.1)\.
- M\. Liu, S\. Diao, X\. Lu, J\. Hu, X\. Dong, Y\. Choi, J\. Kautz, and Y\. Dong \(2025\)ProRL: prolonged reinforcement learning expands reasoning boundaries in large language models\.CoRRabs/2505\.24864\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p5.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in Neural Information Processing Systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.03126#S2.SS1.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.arXiv preprint arXiv:2305\.18290\.Cited by:[§2\.1](https://arxiv.org/html/2607.03126#S2.SS1.p1.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.03126#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2607.03126#S2.SS3.p2.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p1.1),[§1](https://arxiv.org/html/2607.03126#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.03126#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2607.03126#S5.T1.7.7.7.9),[§6\.1](https://arxiv.org/html/2607.03126#S6.SS1.p2.1)\.
- G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu \(2025\)HybridFlow: A flexible and efficient RLHF framework\.InEuroSys,pp\. 1279–1297\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p1.1)\.
- M\. Simoni, A\. Fontana, and G\. Rossolini \(2025\)GTPO: stabilizing group relative policy optimization via gradient and entropy control\.arXiv preprint arXiv:2508\.03772\.Cited by:[§2\.3](https://arxiv.org/html/2607.03126#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2607.03126#S5.T1.28.28.28.8),[§6\.1](https://arxiv.org/html/2607.03126#S6.SS1.p2.1)\.
- X\. Tang, Y\. Zhan, Z\. Li, W\. X\. Zhao, Z\. Zhang, Z\. Wen, Z\. Zhang, and J\. Zhou \(2025\)Rethinking sample polarity in reinforcement learning with verifiable rewards\.arXiv preprint arXiv:2512\.21625\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p4.1),[§2\.3](https://arxiv.org/html/2607.03126#S2.SS3.p1.1)\.
- Q\. Team \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§6\.1](https://arxiv.org/html/2607.03126#S6.SS1.p1.4)\.
- J\. Wang, R\. Liu, L\. Lin, W\. Hu, X\. Li, F\. Zhang, G\. Zhou, and K\. Gai \(2025a\)ASPO: asymmetric importance sampling policy optimization\.External Links:2510\.06062,[Link](https://arxiv.org/abs/2510.06062)Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p4.1)\.
- S\. Wang, L\. Yu, C\. Gao, C\. Zheng, S\. Liu, R\. Lu, K\. Dang, X\. Chen, J\. Yang, Z\. Zhang, Y\. Liu, A\. Yang, A\. Zhao, Y\. Yue, S\. Song, B\. Yu, G\. Huang, and J\. Lin \(2025b\)Beyond the 80/20 rule: high\-entropy minority tokens drive effective reinforcement learning for LLM reasoning\.CoRRabs/2506\.01939\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p4.1),[§1](https://arxiv.org/html/2607.03126#S1.p5.1),[§2\.3](https://arxiv.org/html/2607.03126#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2607.03126#S5.T1.21.21.21.8),[§6\.1](https://arxiv.org/html/2607.03126#S6.SS1.p2.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in Neural Information Processing Systems35,pp\. 24824–24837\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p2.1)\.
- Z\. Xi, X\. Guo, Y\. Nan, E\. Zhou, J\. Shen, W\. Chen, J\. Liu, J\. Huang, Z\. Zhang, H\. Guo, X\. Deng, Z\. Lei, M\. Zheng, G\. Wang, S\. Zhang, P\. Sun, R\. Zheng, H\. Yan, T\. Gui, Q\. Zhang, and X\. Huang \(2025\)BAPO: stabilizing off\-policy reinforcement learning for llms via balanced policy optimization with adaptive clipping\.External Links:2510\.18927,[Link](https://arxiv.org/abs/2510.18927)Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.03126#S2.SS2.p3.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.CoRRabs/2505\.09388\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p1.1)\.
- A\. Yang, B\. Yang, B\. Hui, B\. Zheng, B\. Yu, C\. Zhou, C\. Li, C\. Li, D\. Liu, F\. Huang,et al\.\(2024a\)Qwen2 technical report\.arXiv preprint arXiv:2407\.10670\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p1.1)\.
- A\. Yang, B\. Zhang, B\. Hui, B\. Gao, B\. Yu, C\. Li, D\. Liu, J\. Tu, J\. Zhou, J\. Lin, K\. Lu, M\. Xue, R\. Lin, T\. Liu, X\. Ren, and Z\. Zhang \(2024b\)Qwen2\.5\-math technical report: toward mathematical expert model via self\-improvement\.CoRRabs/2409\.12122\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p1.1)\.
- A\. Yang, B\. Zhang, B\. Hui, B\. Gao, B\. Yu, C\. Li, D\. Liu, J\. Tu, J\. Zhou, J\. Lin, K\. Lu, M\. Xue, R\. Lin, T\. Liu, X\. Ren, and Z\. Zhang \(2024c\)Qwen2\.5\-math technical report: toward mathematical expert model via self\-improvement\.arXiv preprint arXiv:2409\.12122\.Cited by:[§6\.1](https://arxiv.org/html/2607.03126#S6.SS1.p1.4)\.
- Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, T\. Fan, G\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, W\. Dai, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, M\. Qiao, Y\. Wu, and M\. Wang \(2025\)DAPO: an open\-source LLM reinforcement learning system at scale\.CoRRabs/2503\.14476\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.03126#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2607.03126#S5.T1.14.14.14.8),[§6\.1](https://arxiv.org/html/2607.03126#S6.SS1.p1.4),[§6\.1](https://arxiv.org/html/2607.03126#S6.SS1.p2.1)\.
- Z\. Yu, Y\. Zhao, A\. Cohan, and X\. Zhang \(2024\)Humaneval pro and mbpp pro: evaluating large language models on self\-invoking code generation\.arXiv preprint arXiv:2412\.21199\.Cited by:[§6\.1](https://arxiv.org/html/2607.03126#S6.SS1.p2.1)\.
- Y\. Yue, Z\. Chen, R\. Lu, A\. Zhao, Z\. Wang, Y\. Yue, S\. Song, and G\. Huang \(2025\)Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?\.CoRRabs/2504\.13837\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p1.1)\.
- E\. Zelikman, Y\. Wu, J\. Mu, and N\. Goodman \(2022\)STaR: bootstrapping reasoning with reasoning\.Advances in Neural Information Processing Systems35,pp\. 15476–15488\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p1.1)\.
- K\. Zhang, Y\. Zuo, B\. He, Y\. Sun, R\. Liu, C\. Jiang, Y\. Fan, K\. Tian, G\. Jia, P\. Li, Y\. Fu, X\. Lv, Y\. Zhang, S\. Zeng, S\. Qu, H\. Li, S\. Wang, Y\. Wang, X\. Long, F\. Liu, X\. Xu, J\. Ma, X\. Zhu, E\. Hua, Y\. Liu, Z\. Li, H\. Chen, X\. Qu, Y\. Li, W\. Chen, Z\. Yuan, J\. Gao, D\. Li, Z\. Ma, G\. Cui, Z\. Liu, B\. Qi, N\. Ding, and B\. Zhou \(2025\)A survey of reinforcement learning for large reasoning models\.CoRRabs/2509\.08827\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.03126#S2.SS2.p1.1)\.
- S\. Zhang, L\. Dong, X\. Li, S\. Zhang, X\. Sun, S\. Wang, J\. Li, R\. Hu, T\. Zhang, F\. Wu, and G\. Wang \(2023\)Instruction tuning for large language models: A survey\.CoRRabs/2308\.10792\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p1.1)\.
- Y\. Zhang and T\. Math\-AI \(2024\)American invitational mathematics examination \(aime\) 2024\.Cited by:[§6\.1](https://arxiv.org/html/2607.03126#S6.SS1.p2.1)\.
- Y\. Zhang and T\. Math\-AI \(2025\)American invitational mathematics examination \(aime\) 2025\.Cited by:[§6\.1](https://arxiv.org/html/2607.03126#S6.SS1.p2.1)\.
- Y\. Zhao, Y\. Liu, J\. Liu, J\. Chen, X\. Wu, Y\. Hao, T\. Lv, S\. Huang, L\. Cui, Q\. Ye, F\. Wan, and F\. Wei \(2025\)Geometric\-mean policy optimization\.CoRRabs/2507\.20673\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.03126#S2.SS2.p3.1)\.
- C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang, J\. Zhou, and J\. Lin \(2025\)Group sequence policy optimization\.CoRRabs/2507\.18071\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p1.1),[§3](https://arxiv.org/html/2607.03126#S3.p3.7)\.
- X\. Zhu, M\. Xia, Z\. Wei, W\. Chen, D\. Chen, and Y\. Meng \(2025\)The surprising effectiveness of negative reinforcement in LLM reasoning\.CoRRabs/2506\.01347\.Cited by:[§1](https://arxiv.org/html/2607.03126#S1.p2.1)\.

## Appendix AFirst\-Error Annotation and Full\-Trajectory Entropy Analysis

### A\.1Annotation Protocol

To support the local entropy analysis in Section[4\.2](https://arxiv.org/html/2607.03126#S4.SS2), we construct a manually verified first\-error annotation set from AIME24 and AIME25\. The two benchmarks contain 60 prompts in total\. For each prompt, we sampleN=8N=8trajectories using Qwen3\-8B\-Base, resulting in 480 trajectories\.

We first remove prompts whose sampled trajectories are either all correct or all incorrect, since such cases do not provide meaningful within\-prompt contrasts between successful and failed reasoning paths\. We also exclude erroneous trajectories whose failures are primarily caused by exceeding the maximum generation length, rather than by an identifiable mathematical or logical mistake\. After this filtering process, we retain 8 prompts and 64 trajectories for fine\-grained annotation\.

For each retained prompt, since multiple correct and incorrect trajectories may be available, we construct all valid correct–incorrect contrastive pairs within the same prompt, yielding 91 pairs in total\. For each pair, we identify the first logical error in the incorrect trajectory, defined as the earliest substantive mathematical or reasoning deviation that changes the subsequent solution path\. Minor wording differences, formatting variations, or alternative but valid derivations are not considered errors\.

To improve annotation reliability, we use an LLM\-assisted cross\-validation procedure with Claude Opus 4\.6 and GPT\-5\.5\. Each model independently proposes the first logical error location in the incorrect trajectory and explains the corresponding reasoning deviation\. We then manually review the proposed locations and retain only annotations judged to be correct after human verification\. Finally, for each verified first\-error location, we identify the semantically corresponding position in the paired correct trajectory and record the associated token interval\. These paired token intervals are used to align local entropy profiles around the first\-error region\.

### A\.2Extended Visualization of Full\-Trajectory Entropy

To complement the local alignment analysis in Section[4\.2](https://arxiv.org/html/2607.03126#S4.SS2), Figure[4](https://arxiv.org/html/2607.03126#A1.F4)provides unaligned, full\-trajectory entropy profiles for 8 representative AIME prompts sampled via Qwen3\-8B\-Base\.

These macro\-level visualizations clearly demonstrate the “cascade of confusion” phenomenon discussed in Section[4\.5](https://arxiv.org/html/2607.03126#S4.SS5)\. While correct trajectories \(green\) maintain a stable, low\-entropy profile throughout the generation, incorrect trajectories \(red\) frequently exhibit a pattern where an initial logical flaw \(often committed with relatively high confidence\) acts as a tipping point\. This initial error inevitably destabilizes the subsequent reasoning chain, leading to prolonged phases of severe entropy volatility\. This macroscopic divergence further justifies our asymmetric credit assignment strategy: penalizing the initial confident error rather than the subsequent chaotic noise\.

![Refer to caption](https://arxiv.org/html/2607.03126v1/x4.png)Figure 4:Per\-token entropy along full reasoning trajectories \(Correct vs\. Incorrect\)\.For each of the 8 AIME prompts, we plot the smoothed step\-wise entropy of a correct CoT \(green\) and an incorrect CoT \(red\)\. The vertical dotted lines annotate the exact position of the first fatal reasoning error and its logical counterpart\. The global profiles illustrate that a single reasoning flaw typically induces a persistent, trajectory\-wide elevation in model uncertainty\.

## Appendix BDetailed Derivations for Approximation Error

In this section, we provide the detailed derivation for the bounds of the true entropyℋi,t​\(θ\)\\mathcal\{H\}\_\{i,t\}\(\\theta\)using the surrogate entropyδi,t​\(θ\)\\delta\_\{i,t\}\(\\theta\), as discussed in Section[5\.1](https://arxiv.org/html/2607.03126#S5.SS1)\.

Recall the notation:πθ​\(v\):=πθ​\(v\|𝐪,𝐲i,<t\)\\pi\_\{\\theta\}\(v\):=\\pi\_\{\\theta\}\(v\|\\mathbf\{q\},\\mathbf\{y\}\_\{i,<t\}\),v∗=argmaxv∈𝒱⁡πθ​\(v\)v^\{\*\}=\\operatorname\{argmax\}\_\{v\\in\\mathcal\{V\}\}\\pi\_\{\\theta\}\(v\), andδi,t​\(θ\)=1−πθ​\(v∗\)\\delta\_\{i,t\}\(\\theta\)=1\-\\pi\_\{\\theta\}\(v^\{\*\}\)\.

### B\.1Derivation of Lower and Upper Bounds

Lower Bound\.Sincev∗v^\{\*\}is the token with the highest probability, replacing alllog⁡\(πθ​\(v\)\)\\log\(\\pi\_\{\\theta\}\(v\)\)terms with the larger valuelog⁡\(πθ​\(v∗\)\)\\log\(\\pi\_\{\\theta\}\(v^\{\*\}\)\)\(note the negative sign\) yields:

ℋi,t​\(θ\)\\displaystyle\\mathcal\{H\}\_\{i,t\}\(\\theta\)=−∑v∈𝒱πθ​\(v\)​log⁡\(πθ​\(v\)\)\\displaystyle=\-\\sum\_\{v\\in\\mathcal\{V\}\}\\pi\_\{\\theta\}\(v\)\\log\(\\pi\_\{\\theta\}\(v\)\)≥−∑v∈𝒱πθ​\(v\)​log⁡\(πθ​\(v∗\)\)\\displaystyle\\geq\-\\sum\_\{v\\in\\mathcal\{V\}\}\\pi\_\{\\theta\}\(v\)\\log\(\\pi\_\{\\theta\}\(v^\{\*\}\)\)=−log⁡\(πθ​\(v∗\)\)​∑v∈𝒱πθ​\(v\)\\displaystyle=\-\\log\(\\pi\_\{\\theta\}\(v^\{\*\}\)\)\\sum\_\{v\\in\\mathcal\{V\}\}\\pi\_\{\\theta\}\(v\)=−log⁡\(1−δi,t​\(θ\)\)≜L​\(δi,t​\(θ\)\)\.\\displaystyle=\-\\log\(1\-\\delta\_\{i,t\}\(\\theta\)\)\\triangleq L\(\\delta\_\{i,t\}\(\\theta\)\)\.\(18\)
Upper Bound\.We separate the max probability term and apply Jensen’s inequality to the remaining terms:

ℋi,t​\(θ\)\\displaystyle\\mathcal\{H\}\_\{i,t\}\(\\theta\)=−πθ​\(v∗\)​log⁡\(πθ​\(v∗\)\)−∑v≠v∗πθ​\(v\)​log⁡\(πθ​\(v\)\)\\displaystyle=\-\\pi\_\{\\theta\}\(v^\{\*\}\)\\log\(\\pi\_\{\\theta\}\(v^\{\*\}\)\)\-\\sum\_\{v\\neq v^\{\*\}\}\\pi\_\{\\theta\}\(v\)\\log\(\\pi\_\{\\theta\}\(v\)\)≤−πθ​\(v∗\)​log⁡\(πθ​\(v∗\)\)−\(1−πθ​\(v∗\)\)​log⁡\(1−πθ​\(v∗\)\|𝒱\|−1\)\\displaystyle\\leq\-\\pi\_\{\\theta\}\(v^\{\*\}\)\\log\(\\pi\_\{\\theta\}\(v^\{\*\}\)\)\-\(1\-\\pi\_\{\\theta\}\(v^\{\*\}\)\)\\log\\left\(\\frac\{1\-\\pi\_\{\\theta\}\(v^\{\*\}\)\}\{\|\\mathcal\{V\}\|\-1\}\\right\)=−\(1−δi,t​\(θ\)\)​log⁡\(1−δi,t​\(θ\)\)−δi,t​\(θ\)​log⁡\(δi,t​\(θ\)\|𝒱\|−1\)\\displaystyle=\-\(1\-\\delta\_\{i,t\}\(\\theta\)\)\\log\(1\-\\delta\_\{i,t\}\(\\theta\)\)\-\\delta\_\{i,t\}\(\\theta\)\\log\\left\(\\frac\{\\delta\_\{i,t\}\(\\theta\)\}\{\|\\mathcal\{V\}\|\-1\}\\right\)≜U​\(δi,t​\(θ\)\)\.\\displaystyle\\triangleq U\(\\delta\_\{i,t\}\(\\theta\)\)\.\(19\)

### B\.2Analysis of the Approximation Gap

The gap function is defined asg​\(δi,t​\(θ\)\)≜U​\(δ\)−L​\(δ\)g\(\\delta\_\{i,t\}\(\\theta\)\)\\triangleq U\(\\delta\)\-L\(\\delta\)\. Substituting the bounds:

g​\(δi,t​\(θ\)\)=δi,t​\(θ\)​log⁡\(\|𝒱\|−1\)\+δi,t​\(θ\)​log⁡\(1−δi,t​\(θ\)δi,t​\(θ\)\)\.g\(\\delta\_\{i,t\}\(\\theta\)\)=\\delta\_\{i,t\}\(\\theta\)\\log\(\|\\mathcal\{V\}\|\-1\)\+\\delta\_\{i,t\}\(\\theta\)\\log\\left\(\\frac\{1\-\\delta\_\{i,t\}\(\\theta\)\}\{\\delta\_\{i,t\}\(\\theta\)\}\\right\)\.\(20\)The first and second derivatives with respect toδi,t​\(θ\)\\delta\_\{i,t\}\(\\theta\)are:

g′​\(δi,t​\(θ\)\)\\displaystyle g^\{\\prime\}\(\\delta\_\{i,t\}\(\\theta\)\)=log⁡\(\|𝒱\|−1\)\+log⁡\(1−δi,t​\(θ\)δi,t​\(θ\)\)−11−δi,t​\(θ\),\\displaystyle=\\log\(\|\\mathcal\{V\}\|\-1\)\+\\log\\left\(\\frac\{1\-\\delta\_\{i,t\}\(\\theta\)\}\{\\delta\_\{i,t\}\(\\theta\)\}\\right\)\-\\frac\{1\}\{1\-\\delta\_\{i,t\}\(\\theta\)\},\(21\)g′′​\(δi,t​\(θ\)\)\\displaystyle g^\{\\prime\\prime\}\(\\delta\_\{i,t\}\(\\theta\)\)=−1δi,t​\(θ\)​\(1−δi,t​\(θ\)\)2<0\.\\displaystyle=\-\\frac\{1\}\{\\delta\_\{i,t\}\(\\theta\)\(1\-\\delta\_\{i,t\}\(\\theta\)\)^\{2\}\}<0\.\(22\)Sinceg′′<0g^\{\\prime\\prime\}<0, the function is concave\. Boundary analysis shows:

- •Whenδ=0\\delta=0,ℋi,t=0\\mathcal\{H\}\_\{i,t\}=0andg​\(0\)=0g\(0\)=0\(vialimδ→0δ​log⁡δ=0\\lim\_\{\\delta\\to 0\}\\delta\\log\\delta=0\)\.
- •Whenδ=\|𝒱\|−1\|𝒱\|\\delta=\\frac\{\|\\mathcal\{V\}\|\-1\}\{\|\\mathcal\{V\}\|\},g​\(\|𝒱\|−1\|𝒱\|\)=\|𝒱\|−1\|𝒱\|​\[log⁡\(\|𝒱\|−1\)\+log⁡\(1\|𝒱\|−1\)\]=0g\(\\frac\{\|\\mathcal\{V\}\|\-1\}\{\|\\mathcal\{V\}\|\}\)=\\frac\{\|\\mathcal\{V\}\|\-1\}\{\|\\mathcal\{V\}\|\}\[\\log\(\|\\mathcal\{V\}\|\-1\)\+\\log\(\\frac\{1\}\{\|\\mathcal\{V\}\|\-1\}\)\]=0\.

### B\.3Maximum Gap Derivation

Letg′​\(δi,t​\(θ\)\)=0g^\{\\prime\}\(\\delta\_\{i,t\}\(\\theta\)\)=0\. From Eq\.[21](https://arxiv.org/html/2607.03126#A2.E21)and lettingt=δi,t​\(θ\)1−δi,t​\(θ\)t=\\frac\{\\delta\_\{i,t\}\(\\theta\)\}\{1\-\\delta\_\{i,t\}\(\\theta\)\}, we have:

log⁡\(\|𝒱\|−1\)−log⁡t=t\+1\.\\log\(\|\\mathcal\{V\}\|\-1\)\-\\log t=t\+1\.\(23\)Substituting Eq\.[23](https://arxiv.org/html/2607.03126#A2.E23)back into the expression forg​\(δi,t​\(θ\)\)g\(\\delta\_\{i,t\}\(\\theta\)\):

maxδ⁡g​\(δ\)\\displaystyle\\max\_\{\\delta\}g\(\\delta\)=δi,t​\(θ\)​\(log⁡\(\|𝒱\|−1\)−log⁡t\)\\displaystyle=\\delta\_\{i,t\}\(\\theta\)\\left\(\\log\(\|\\mathcal\{V\}\|\-1\)\-\\log t\\right\)=δi,t​\(θ\)​\(t\+1\)=δi,t​\(θ\)1−δi,t​\(θ\)=t\.\\displaystyle=\\delta\_\{i,t\}\(\\theta\)\(t\+1\)=\\frac\{\\delta\_\{i,t\}\(\\theta\)\}\{1\-\\delta\_\{i,t\}\(\\theta\)\}=t\.\(24\)Thus, the maximum absolute error is exactly the solutionttto the characteristic equationt\+log⁡t=log⁡\(\|𝒱\|−1\)−1t\+\\log t=\\log\(\|\\mathcal\{V\}\|\-1\)\-1\.

### B\.4Tightness in Practical LLM Scenarios

As derived in Appendix[B\.3](https://arxiv.org/html/2607.03126#A2.SS3), the maximum approximation gap scales with the vocabulary size\|𝒱\|\|\\mathcal\{V\}\|\. However, practical LLM distributions are typically concentrated in the top\-kktokens due to softmax anisotropy\[Ethayarajh,[2019](https://arxiv.org/html/2607.03126#bib.bib18)\]and mass concentration\[Holtzmanet al\.,[2019](https://arxiv.org/html/2607.03126#bib.bib16)\]\.

By considering this truncation, which effectively replaces\|𝒱\|\|\\mathcal\{V\}\|withkk, we define theAbsolute Error Boundtabst\_\{\\text\{abs\}\}as the solution to the characteristic equation derived in Eq\.[23](https://arxiv.org/html/2607.03126#A2.E23):t\+log⁡t=log⁡\(k−1\)−1t\+\\log t=\\log\(k\-1\)\-1, wheret=δ/\(1−δ\)t=\\delta/\(1\-\\delta\)\.

As shown in Table[3](https://arxiv.org/html/2607.03126#A2.T3), the approximation error shrinks substantially under top\-kkregimes\. For typical reasoning steps where probability mass is concentrated in a few tokens, the relative error remains controlled, e\.g\., below10%10\\%fork=10k=10\. This quantitative analysis suggests thatδi,t​\(θ\)\\delta\_\{i,t\}\(\\theta\)can serve as a practically useful and mathematically bounded proxy for true entropy when the next\-token distribution is sufficiently mass\-concentrated\. In addition to this bound\-based analysis, Figure[5](https://arxiv.org/html/2607.03126#A2.F5)provides an empirical trajectory\-level example showing that the surrogate tracks the macroscopic trend of normalized true entropy while remaining mode\-local and invariant to tail dispersion\.

Table 3:Quantitative analysis of approximation gaps across different Top\-kkregimes\. All values are derived from the characteristic equationt\+log⁡t=log⁡\(k−1\)−1t\+\\log t=\\log\(k\-1\)\-1\.![Refer to caption](https://arxiv.org/html/2607.03126v1/x5.png)Figure 5:Surrogate entropy tracks normalized true entropy\.Along a real rollout trajectory from Qwen2\.5\-Math\-7B on the AIME25 benchmark, the proposed surrogate entropy closely follows the macroscopic trend of normalized true entropy while remaining mode\-local and invariant to tail dispersion\.

## Appendix CDetailed Derivation of the Gradient Modulation Factor

In this section, we provide the step\-by\-step derivation for the modulation factorsℳi,t\+\\mathcal\{M\}\_\{i,t\}^\{\+\}andℳi,t−\\mathcal\{M\}\_\{i,t\}^\{\-\}introduced in Section[5\.2](https://arxiv.org/html/2607.03126#S5.SS2)\. This derivation characterizes the leading\-order behavior of ACPO under the empirically observed modal\-alignment regime and the proximal\-update condition\. It is intended to explain the dominant gradient modulation effect rather than establish a global convergence guarantee\.

### C\.1Expansion for Positive Samples

ForA^i\>0\\widehat\{A\}\_\{i\}\>0, the objective component for a single token is𝒥i,t\+=fi,t​\(wi,t​\(θ\)\)⋅\(2\+δi,t​\(θ\)\)\\mathcal\{J\}\_\{i,t\}^\{\+\}=f\_\{i,t\}\(w\_\{i,t\}\(\\theta\)\)\\cdot\(2\+\\delta\_\{i,t\}\(\\theta\)\), whereδi,t​\(θ\)=1−πθ​\(v∗\)\\delta\_\{i,t\}\(\\theta\)=1\-\\pi\_\{\\theta\}\(v^\{\*\}\)\. Expanding the gradient:

∇θ𝒥i,t\+\\displaystyle\\nabla\_\{\\theta\}\\mathcal\{J\}\_\{i,t\}^\{\+\}=∇θ\[fi,t​\(wi,t​\(θ\)\)⋅\(3−πθ​\(v∗\)\)\]\\displaystyle=\\nabla\_\{\\theta\}\\Big\[f\_\{i,t\}\(w\_\{i,t\}\(\\theta\)\)\\cdot\(3\-\\pi\_\{\\theta\}\(v^\{\*\}\)\)\\Big\]=\(3−πθ​\(v∗\)\)​∇wfi,t​\(w\)​wi,t​∇θlog⁡πθ​\(yi,t\)−fi,t​\(w\)​πθ​\(v∗\)​∇θlog⁡πθ​\(v∗\)\\displaystyle=\(3\-\\pi\_\{\\theta\}\(v^\{\*\}\)\)\\nabla\_\{w\}f\_\{i,t\}\(w\)w\_\{i,t\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{i,t\}\)\-f\_\{i,t\}\(w\)\\pi\_\{\\theta\}\(v^\{\*\}\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(v^\{\*\}\)=∇wfi,t​\(w\)​\[\(3−πθ​\(v∗\)\)​wi,t​∇θlog⁡πθ​\(yi,t\)−fi,t​\(w\)∇wfi,t​\(w\)​wi,t∗​πθold​\(v∗\)​∇θlog⁡πθ​\(v∗\)\]\\displaystyle=\\nabla\_\{w\}f\_\{i,t\}\(w\)\\bigg\[\(3\-\\pi\_\{\\theta\}\(v^\{\*\}\)\)w\_\{i,t\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{i,t\}\)\-\\frac\{f\_\{i,t\}\(w\)\}\{\\nabla\_\{w\}f\_\{i,t\}\(w\)\}w\_\{i,t\}^\{\*\}\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(v^\{\*\}\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(v^\{\*\}\)\\bigg\]\(25\)wherewi,t∗=πθ​\(v∗\)/πθold​\(v∗\)w\_\{i,t\}^\{\*\}=\\pi\_\{\\theta\}\(v^\{\*\}\)/\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(v^\{\*\}\)\.

Applying the Modal Alignment approximation \(πθ​\(yi,t\)≈πθ​\(v∗\)\\pi\_\{\\theta\}\(y\_\{i,t\}\)\\approx\\pi\_\{\\theta\}\(v^\{\*\}\)and∇θlog⁡πθ​\(yi,t\)≈∇θlog⁡πθ​\(v∗\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{i,t\}\)\\approx\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(v^\{\*\}\)\), and assuming the proximal update regimewi,t≈1,wi,t∗≈1w\_\{i,t\}\\approx 1,w\_\{i,t\}^\{\*\}\\approx 1, Eq\.[25](https://arxiv.org/html/2607.03126#A3.E25)simplifies to:

∇θ𝒥i,t\+\\displaystyle\\nabla\_\{\\theta\}\\mathcal\{J\}\_\{i,t\}^\{\+\}≈wi,t​∇wfi,t​\(w\)​∇θlog⁡πθ​\(yi,t\)​\[3−πθ​\(yi,t\)−2​πθold​\(yi,t\)\]\\displaystyle\\approx w\_\{i,t\}\\nabla\_\{w\}f\_\{i,t\}\(w\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{i,t\}\)\\left\[3\-\\pi\_\{\\theta\}\(y\_\{i,t\}\)\-2\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(y\_\{i,t\}\)\\right\]\(26\)Substitutingπ​\(y\)=1−δ\\pi\(y\)=1\-\\delta, we obtain:

ℳi,t\+=3−\(1−δi,t​\(θ\)\)−2​\(1−δi,t​\(θold\)\)=δi,t​\(θ\)\+2​δi,t​\(θold\)\.\\displaystyle\\mathcal\{M\}\_\{i,t\}^\{\+\}=3\-\(1\-\\delta\_\{i,t\}\(\\theta\)\)\-2\(1\-\\delta\_\{i,t\}\(\\theta\_\{\\text\{old\}\}\)\)=\\delta\_\{i,t\}\(\\theta\)\+2\\delta\_\{i,t\}\(\\theta\_\{\\text\{old\}\}\)\.\(27\)

### C\.2Expansion for Negative Samples

ForA^i≤0\\widehat\{A\}\_\{i\}\\leq 0, the objective is𝒥i,t−=fi,t​\(wi,t​\(θ\)\)⋅\(1−δi,t​\(θ\)\)=fi,t​\(wi,t​\(θ\)\)⋅πθ​\(v∗\)\\mathcal\{J\}\_\{i,t\}^\{\-\}=f\_\{i,t\}\(w\_\{i,t\}\(\\theta\)\)\\cdot\(1\-\\delta\_\{i,t\}\(\\theta\)\)=f\_\{i,t\}\(w\_\{i,t\}\(\\theta\)\)\\cdot\\pi\_\{\\theta\}\(v^\{\*\}\)\.

∇θ𝒥i,t−\\displaystyle\\nabla\_\{\\theta\}\\mathcal\{J\}\_\{i,t\}^\{\-\}=πθ​\(v∗\)​∇wfi,t​\(w\)​wi,t​∇θlog⁡πθ​\(yi,t\)\+fi,t​\(w\)​πθ​\(v∗\)​∇θlog⁡πθ​\(v∗\)\\displaystyle=\\pi\_\{\\theta\}\(v^\{\*\}\)\\nabla\_\{w\}f\_\{i,t\}\(w\)w\_\{i,t\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{i,t\}\)\+f\_\{i,t\}\(w\)\\pi\_\{\\theta\}\(v^\{\*\}\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(v^\{\*\}\)≈wi,t​∇wfi,t​\(w\)​∇θlog⁡πθ​\(yi,t\)​\[πθ​\(yi,t\)\+2​πθold​\(yi,t\)\]\\displaystyle\\approx w\_\{i,t\}\\nabla\_\{w\}f\_\{i,t\}\(w\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{i,t\}\)\\left\[\\pi\_\{\\theta\}\(y\_\{i,t\}\)\+2\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(y\_\{i,t\}\)\\right\]\(28\)Substituting the entropy proxies:

ℳi,t−=\(1−δi,t​\(θ\)\)\+2​\(1−δi,t​\(θold\)\)=3−δi,t​\(θ\)−2​δi,t​\(θold\)\.\\displaystyle\\mathcal\{M\}\_\{i,t\}^\{\-\}=\(1\-\\delta\_\{i,t\}\(\\theta\)\)\+2\(1\-\\delta\_\{i,t\}\(\\theta\_\{\\text\{old\}\}\)\)=3\-\\delta\_\{i,t\}\(\\theta\)\-2\\delta\_\{i,t\}\(\\theta\_\{\\text\{old\}\}\)\.\(29\)

### C\.3Error Analysis Remark

As indicated by the empirical data in Table[4](https://arxiv.org/html/2607.03126#A3.T4), the approximation errorϵ=\|π​\(yi,t\)−π​\(v∗\)\|\\epsilon=\|\\pi\(y\_\{i,t\}\)\-\\pi\(v^\{\*\}\)\|is bounded by0\.10\.1for over86%86\\%of samples\. In the context of stochastic gradient descent, this high\-probability alignment empirically supports the use of the derived modulation factors as a leading\-order approximation to the dominant gradient direction\.

Table 4:Analysis of Error Cumulative Coverage and Modal Alignment\.

## Appendix DTheoretical Analysis of Dynamic Damping in Off\-policy Scenarios

In Section[5\.2](https://arxiv.org/html/2607.03126#S5.SS2), we approximated the hybrid modulation factorℳi,t\+\\mathcal\{M\}\_\{i,t\}^\{\+\}by assuming the proximal update regime \(wi,t≈1w\_\{i,t\}\\approx 1\)\. This analysis illustrates a damping tendency in off\-policy regions, rather than a formal guarantee against all unstable updates\.

### D\.1Deriving the Coupling Coefficient

Let us evaluate the exact gradient expansion for a positive sample \(A^i\>0\\widehat\{A\}\_\{i\}\>0\)\. Using the product rule on𝒥i,t\+=fi,t​\(wi,t\)⋅\(2\+δi,t​\(θ\)\)\\mathcal\{J\}\_\{i,t\}^\{\+\}=f\_\{i,t\}\(w\_\{i,t\}\)\\cdot\(2\+\\delta\_\{i,t\}\(\\theta\)\)and factoring out the scaled policy gradientwi,t​∇wfi,t​\(wi,t\)​∇θlog⁡πθ​\(yi,t\)w\_\{i,t\}\\nabla\_\{w\}f\_\{i,t\}\(w\_\{i,t\}\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{i,t\}\), we obtain:

∇θ𝒥i,t\+∝\[\(2\+δi,t​\(θ\)\)−fi,t​\(wi,t\)∇wfi,t​\(wi,t\)⏟C​\(wi,t\)​\(1−δi,t​\(θold\)\)\]\.\\displaystyle\\nabla\_\{\\theta\}\\mathcal\{J\}\_\{i,t\}^\{\+\}\\propto\\left\[\(2\+\\delta\_\{i,t\}\(\\theta\)\)\-\\underbrace\{\\frac\{f\_\{i,t\}\(w\_\{i,t\}\)\}\{\\nabla\_\{w\}f\_\{i,t\}\(w\_\{i,t\}\)\}\}\_\{C\(w\_\{i,t\}\)\}\\big\(1\-\\delta\_\{i,t\}\(\\theta\_\{\\text\{old\}\}\)\\big\)\\right\]\.\(30\)Here,C​\(wi,t\)C\(w\_\{i,t\}\)is the off\-policy coupling coefficient, dictating the relative strength of the historical damping term\(1−δi,t​\(θold\)\)\(1\-\\delta\_\{i,t\}\(\\theta\_\{\\text\{old\}\}\)\)against the driving term\(2\+δi,t​\(θ\)\)\(2\+\\delta\_\{i,t\}\(\\theta\)\)\.

Recall the SAPO soft\-gating function defined in Eq\.[7](https://arxiv.org/html/2607.03126#S3.E7):f​\(w\)=4τ​σ​\(τ​\(w−1\)\)f\(w\)=\\frac\{4\}\{\\tau\}\\sigma\(\\tau\(w\-1\)\)\. Its derivative is∇wf​\(w\)=4​σ​\(τ​\(w−1\)\)​\[1−σ​\(τ​\(w−1\)\)\]\\nabla\_\{w\}f\(w\)=4\\sigma\(\\tau\(w\-1\)\)\\big\[1\-\\sigma\(\\tau\(w\-1\)\)\\big\]\. Substituting these intoC​\(wi,t\)C\(w\_\{i,t\}\), the terms cleanly cancel out, yielding the precise analytical form:

C​\(wi,t\)=1τpos​\(1−σ​\(τpos​\(wi,t−1\)\)\)\.\\displaystyle C\(w\_\{i,t\}\)=\\frac\{1\}\{\\tau\_\{\\text\{pos\}\}\\Big\(1\-\\sigma\\big\(\\tau\_\{\\text\{pos\}\}\(w\_\{i,t\}\-1\)\\big\)\\Big\)\}\.\(31\)

### D\.2Quantitative Analysis of the Mathematical Brake

Eq\.[31](https://arxiv.org/html/2607.03126#A4.E31)exposes an intrinsic mathematical brake mechanism\. As the active policy becomes aggressively overconfident compared to the historical policy \(wi,t→∞w\_\{i,t\}\\to\\infty\), the sigmoid outputσ​\(⋅\)\\sigma\(\\cdot\)approaches11\. Consequently, the denominator\(1−σ\)\(1\-\\sigma\)rapidly approaches0, causingC​\(wi,t\)C\(w\_\{i,t\}\)to grow exponentially\.

To visualize this dynamic stabilization, Table[5](https://arxiv.org/html/2607.03126#A4.T5)calculates the damping coefficientC​\(w\)C\(w\)across various off\-policy divergence levels \(assumingτpos=1\\tau\_\{\\text\{pos\}\}=1\)\.

Table 5:Quantitative evaluation of the Dynamic Damping effect under different off\-policy deviations \(Positive Samples,A^i\>0\\widehat\{A\}\_\{i\}\>0\)\. The Relative Damping Multiplier indicates the strength of the historical pull\-back force compared to the stable proximal baseline \(wi,t=1\.0w\_\{i,t\}=1\.0\)\.#### Asymmetric Stabilizing Behavior\.

As illustrated in Table[5](https://arxiv.org/html/2607.03126#A4.T5), the coupling coefficientC​\(wi,t\)C\(w\_\{i,t\}\)orchestrates a remarkably intelligent, asymmetric stabilization mechanism for positive reasoning steps:

1. 1\.Aggressive Regimes \(wi,t≫1w\_\{i,t\}\\gg 1\):When the policy becomes excessively overconfident compared to the historical baseline,C​\(w\)C\(w\)grows exponentially \(e\.g\., jumping to8\.408\.40atw=3\.0w=3\.0\)\. The historical anchor’s penalty escalates drastically, acting as a mathematical brake to prevent the policy from collapsing into overconfident local minima\.
2. 2\.Under\-confident Regimes \(wi,t<1w\_\{i,t\}<1\):Conversely, if the current policy underestimates a correct step relative to the historical policy,C​\(w\)C\(w\)naturally decays below the baseline2\.02\.0\. By reducing the damping force \(0\.72×0\.72\\timesatw=0\.2w=0\.2\), the mechanism inherently accelerates learning, encouraging the policy to quickly recover probability mass on verified correct trajectories\.

Rather than relying solely on the static global clipping off​\(w\)f\(w\), ACPO utilizes this non\-linear property to actively recalibrate the update magnitude\. It penalizes reckless gradient steps while facilitating the recovery of lost reasoning paths, theoretically validating ACPO’s robust convergence across long, complex reasoning CoTs\.

## Appendix EAblation study of Top\-k approximation

![Refer to caption](https://arxiv.org/html/2607.03126v1/x6.png)Figure 6:Impact of differentkkvalues on training dynamics\.The figure illustrates the learning curves of mean@8 accuracy on the AIME24 dataset during training, comparing the effects of settingk∈\{1,5,20,100\}k\\in\\\{1,5,20,100\\\}\.As theoretically derived in Appendix B\.4, the approximation error of our surrogate entropyδi,t​\(θ\)\\delta\_\{i,t\}\(\\theta\)is tightly bounded when the next\-token probability mass is highly concentrated\. To empirically investigate how the truncation windowkkinfluences the actual policy optimization dynamics, we conduct an ablation study settingk∈\{1,5,20,100\}k\\in\\\{1,5,20,100\\\}\.

Figure[6](https://arxiv.org/html/2607.03126#A5.F6)illustrates the learning curves of mean@8 accuracy on the AIME24 dataset during training\. Overall, the training trajectories are similar across differentkkvalues, suggesting that the dominant\-mode signal captures much of the useful uncertainty information in this setting\. This observation is consistent with the mass\-concentration behavior discussed in Appendix[B\.4](https://arxiv.org/html/2607.03126#A2.SS4)\.

Upon closer inspection,k=1k=1, the default ACPO configuration, achieves a slight but consistent late\-stage advantage\. Larger values ofkkdo not provide clear additional gains and may introduce noise from secondary low\-probability candidates\. These results suggest that the top\-1 surrogate offers a concise and effective choice for scaling the advantage signals in our experiments\.

Upon closer inspection,k=1k=1\(the default configuration of our proposed ACPO\) demonstrates a slight but consistent performance lead in the late stages of training\. The configurationsk=5k=5andk=100k=100yield almost identical performance, whereask=20k=20exhibits a marginal degradation\. This phenomenon can be attributed to the signal\-to\-noise ratio in the probability distribution\. Settingk=1k=1directly utilizes the maximum probability mass, providing the sharpest and most immediate signal of the model’s localized confidence\. Expandingkkadds minimal useful cognitive information due to the heavy\-tailed nature of the distribution, and may occasionally introduce minor gradient noise from low\-probability tokens\. Consequently, selectingk=1k=1is not only computationally the most efficient, but also empirically optimal for scaling the advantage signals in ACPO\.

## Appendix FBroader Impacts

The proposed ACPO framework may improve mathematical and logical reasoning performance by providing a lightweight mechanism for token\-level credit assignment without relying on human\-annotated process\-level feedback\. This could support beneficial applications such as AI\-assisted education, automated theorem proving, and scientific discovery\. By assigning larger penalties to overconfident erroneous tokens, ACPO may also contribute to more reliable reasoning behavior, although further evaluation is needed to assess calibration and safety properties directly\.

However, advancing the reasoning and planning abilities of LLMs also introduces potential societal risks\. The dual\-use nature of foundation models means that enhanced logical capabilities could be exploited by malicious actors to automate sophisticated cyberattacks, generate highly persuasive disinformation, or conduct complex social engineering campaigns\. Additionally, as models become more capable in solving complex problems, there is an increased risk of “automation bias,” where users might overly trust the model’s outputs in high\-stakes domains \(e\.g\., legal or medical reasoning\) without sufficient human oversight\.

To mitigate these risks, it is imperative that the deployment of advanced reasoning models is paired with rigorous safety alignment and red\-teaming\. While our current reward formulation targets mathematical correctness, future work could adapt the ACPO framework to explicitly penalize unsafe, biased, or malicious reasoning steps, thereby leveraging our dense credit assignment mechanism to improve both model capability and AI safety\.