When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learning

# Tail-Aware Credit Calibration for LLM Reinforcement Learning
Source: [https://arxiv.org/html/2607.07976](https://arxiv.org/html/2607.07976)
## When Implausible Tokens Get Reinforced: Tail\-Aware Credit Calibration for LLM Reinforcement Learning

Xiuyi Lou1,Zicheng Xu111footnotemark:1,Yu\-Neng Chuang2,Hoang Anh Duy Le2, Zhaozhuo Xu3,Guanchu Wang4,Vladimir Braverman1 1Johns Hopkins University,2Rice University, 3Workato,4University of North Carolina at Charlotte

###### Abstract

Reinforcement learning \(RL\) has achieved remarkable success in enhancing the reasoning capabilities of large language models \(LLMs\)\. However, widely used critic\-free RL methods rely on uniform credit assignment, broadcasting the same advantage to all tokens regardless of their differences\. We identify a critical failure mode of this design, which we refer to as*Positive\-Credit Contamination*: low\-probability tail tokens that are contextually erroneous receive identical positive credit to plausible ones within the same trajectory, resulting in the indiscriminate reinforcement of flawed reasoning behavior\. To mitigate this issue, we proposeTail\-AwareCredit calibratiOn \(TACO\), a method that calibrates uniform credit assignment to suppress undesirable positive updates\.TACOfirst computes a tail\-risk score that incorporates the local generation context to assess each token’s risk of falling into the unreliable tail, distinguishing unexpected rarity from uncertainty\-driven exploration\.TACOthen uses this score to tune positive credit for risky tokens without removing their gradients entirely, so that recurring useful rare patterns can accumulate reinforcement while incidental noise is progressively dampened\. Experimental results across three LLMs and eight benchmarks show thatTACOconsistently outperforms GRPO\-style baselines\. Notably,TACOimproves training stability, supporting sustained performance gains in long\-horizon RL\. The source code is available at:[https://github\.com/xiuyilou/TACO](https://github.com/xiuyilou/TACO)\.

## 1Introduction

Reinforcement learning with verifiable rewards \(RLVR\) has become a common post\-training paradigm for reasoning\-oriented large language models \(LLMs\)\. Recent models such as OpenAI’s o\-series and DeepSeek\-R1 demonstrate that large\-scale RL post\-training can substantially improve performance on automatically verifiable tasks, including mathematics and programming\[[14](https://arxiv.org/html/2607.07976#bib.bib1),[2](https://arxiv.org/html/2607.07976#bib.bib2)\]\. Early RLVR methods often build on Proximal Policy Optimization \(PPO\), which typically relies on a learned critic for advantage estimation\[[18](https://arxiv.org/html/2607.07976#bib.bib3),[15](https://arxiv.org/html/2607.07976#bib.bib4)\]\. In contrast, Group Relative Policy Optimization \(GRPO\) replaces the learned critic with group\-relative advantages, reducing value\-modeling overhead while achieving strong empirical performance\[[19](https://arxiv.org/html/2607.07976#bib.bib5)\]\. The simple yet effective design has made GRPO a widely adopted backbone for reasoning RL, motivating variants such as Decoupled Clip and Dynamic Sampling Policy Optimization \(DAPO\) and Group Sequence Policy Optimization \(GSPO\)\[[34](https://arxiv.org/html/2607.07976#bib.bib6),[38](https://arxiv.org/html/2607.07976#bib.bib8)\]\.

However, this simplification computes a single completion\-level advantage and broadcasts it uniformly to every generated token, even though not every token in a rewarded completion is equally reliable\. Specifically, tokens from the low\-probability tail of the policy distribution, which are unlikely under the generation context, can be sampled in rewarded completions even when they are semantically irrelevant or erroneous; we refer to such locally unreliable tokens as implausible tail tokens\. Yet, these tokens’ local effects may be bypassed or overlooked by the overall correct reasoning process and final answer\[[22](https://arxiv.org/html/2607.07976#bib.bib11),[10](https://arxiv.org/html/2607.07976#bib.bib12),[6](https://arxiv.org/html/2607.07976#bib.bib15)\]\. These tokens then receive the same positive credit as reliable ones, resulting in indiscriminate reinforcement of both well\-formed reasoning behavior and locally flawed continuations\. Consequently, the accumulation of such updates throughout training progressively biases the policy away from well\-calibrated reasoning patterns\. We refer to this failure mode as*Positive\-Credit Contamination*\.

To improve upon the uniform credit assignment in GRPO\-style methods, existing work generally seeks to differentiate token\-level updates based on their estimated importance to the final outcome\. One line of work leverages external signals such as counterfactual analysis, temporal\-difference propagation, or execution feedback to evaluate each token’s contribution and scale its credit accordingly\[[9](https://arxiv.org/html/2607.07976#bib.bib20),[16](https://arxiv.org/html/2607.07976#bib.bib9),[21](https://arxiv.org/html/2607.07976#bib.bib10),[7](https://arxiv.org/html/2607.07976#bib.bib21)\]\. Another line incorporates intrinsic signals such as token entropy or distributional divergence as proxies for per\-token significance, concentrating updates on tokens deemed critical to the reasoning process and attenuating routine ones\[[20](https://arxiv.org/html/2607.07976#bib.bib22),[12](https://arxiv.org/html/2607.07976#bib.bib16),[35](https://arxiv.org/html/2607.07976#bib.bib44),[26](https://arxiv.org/html/2607.07976#bib.bib45)\]\. However, existing methods exhibit two fundamental limitations\. First, most of these methods depend on additional inference or auxiliary models, introducing non\-trivial resource overhead\. Second, current methods lack a local semantic perspective on token reliability\. Implausible tail tokens may appear in high\-contribution or high\-entropy positions that existing methods regard as informative, causing unreliable credit to be mistakenly amplified\. Together, these limitations restrict the effectiveness of existing credit\-assignment methods in reasoning\-oriented RL training\.

To overcome these limitations, we introduceTail\-AwareCredit calibratiOn \(TACO\)\. Unlike existing methods,TACOcalibrates uniform credit assignment from a local semantic lens to suppress undesirable positive updates, with only negligible additional computational cost\. Specifically,TACOestimates each token’s risk of being an implausible tail token under its generation context, softly suppressing positive credit for high\-risk tokens while preserving full reinforcement for low\-risk ones\. Therefore,TACOreduces harmful positive reinforcement of unreliable behaviors while preserving useful exploration\.

![Refer to caption](https://arxiv.org/html/2607.07976v1/x1.png)Figure 1:TACOconsistently improves over GRPO across representative benchmarks and models\.Empirically, we evaluateTACOacross three LLMs and six mathematical reasoning benchmarks, as well as two out\-of\-distribution \(OOD\) scientific reasoning benchmarks\.TACOconsistently improves over GRPO\-style baselines across all settings, with Figure[1](https://arxiv.org/html/2607.07976#S1.F1)highlighting the gains on five representative benchmarks\. Our contributions can be summarized as follows:

- •Positive\-Credit Contamination\.We identify a failure mode of GRPO\-style RLVR in which locally implausibletokens receive positive reinforcement\.
- •Tail\-Aware Credit Calibration\.We introduceTACO, a context\-aware method that calibrates positive token\-level credit with negligible computational overhead\.
- •Comprehensive Evaluation\.TACOconsistently improves over baselines across multiple benchmarks, maintaining stable performance under long\-horizon training\.

## 2Related Work

#### Low\-probability tokens in RLVR\.

Recent research suggests that tokens exhibit heterogeneity, making uniform update rules in RLVR suboptimal, particularly for low\-probability tokens\[[12](https://arxiv.org/html/2607.07976#bib.bib16)\]\. On one hand, rare tokens sustain exploration by preserving diverse reasoning continuations and preventing premature policy collapse\[[5](https://arxiv.org/html/2607.07976#bib.bib17)\], motivating protective mechanisms such as clip\-higher\[[34](https://arxiv.org/html/2607.07976#bib.bib6)\]and low\-probability regularization\[[5](https://arxiv.org/html/2607.07976#bib.bib17)\]\. On the other hand, rare tokens can destabilize optimization: their large gradient magnitudes can overshadow updates for high\-probability tokens, and tokens with both low probability and low local entropy have been identified as primary drivers of training instability\[[33](https://arxiv.org/html/2607.07976#bib.bib18),[11](https://arxiv.org/html/2607.07976#bib.bib19)\]\. These findings motivated gradient\-aware methods that dampen extreme token updates to enhance training robustness\. Together, both directions highlight the importance of low\-probability tokens in GRPO\-style optimization, motivating further examination of their properties\.

#### Token\-level credit allocation in RLVR\.

GRPO\-style training assigns the same trajectory\-level reward to every token in a completion, which can misallocate credit when tokens contribute unequally to the final outcome\. Two main directions address this limitation\. The first leverages external signals to redistribute credit: OAR estimates per\-token outcome influence\[[9](https://arxiv.org/html/2607.07976#bib.bib20)\]; GRPO\-λ\\lambdapropagates credit backward via temporal\-difference methods\[[16](https://arxiv.org/html/2607.07976#bib.bib9)\]; TEMPO and EGCA concentrate updates on pivotal decision points using response structure and intermediate execution feedback\[[21](https://arxiv.org/html/2607.07976#bib.bib10),[7](https://arxiv.org/html/2607.07976#bib.bib21)\]\. The second reshapes token\-level updates based on intrinsic signals: GTPO/GRPO\-S and HAPO use uncertainty measures such as token entropy as importance indicators, so that high\-uncertainty or exploratory tokens receive prioritized treatment over routine ones\[[20](https://arxiv.org/html/2607.07976#bib.bib22),[12](https://arxiv.org/html/2607.07976#bib.bib16)\]\. However, these methods primarily model contribution or importance, leaving the contextual validity of credited tokens underexplored\.

## 3Preliminary

### 3\.1Notations

Letπθ\\pi\_\{\\theta\}be an autoregressive policy over vocabulary𝒱\\mathcal\{V\}\. For a promptqqand completionoi=\(oi,1,…,oi,Ti\)o\_\{i\}=\(o\_\{i,1\},\\dots,o\_\{i,T\_\{i\}\}\), we denoteci,t=\(q,oi,<t\)c\_\{i,t\}=\(q,o\_\{i,<t\}\)as the context at steptt, andpi,t=πθ​\(oi,t∣ci,t\)p\_\{i,t\}=\\pi\_\{\\theta\}\(o\_\{i,t\}\\mid c\_\{i,t\}\)as the sampled\-token probability\. The local entropy isHi,t=−∑v∈𝒱πθ​\(v∣ci,t\)​log⁡πθ​\(v∣ci,t\)H\_\{i,t\}=\-\\sum\_\{v\\in\\mathcal\{V\}\}\\pi\_\{\\theta\}\(v\\mid c\_\{i,t\}\)\\log\\pi\_\{\\theta\}\(v\\mid c\_\{i,t\}\)\. In this work, we aim to identify unreliable tokens from generation\-time statistics underci,tc\_\{i,t\}to calibrate their credit, suppressing undesirable positive updates while preserving sound ones\.

### 3\.2Group Relative Policy Optimization

GRPO is a widely adopted RLVR framework for reasoning\-oriented LLM post\-training\. Given a promptqq, GRPO samples a group ofGGcompletions\{oi\}i=1G\\\{o\_\{i\}\\\}\_\{i=1\}^\{G\}using the policyπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\. Each completionoio\_\{i\}receives an outcome\-level verified rewardRiR\_\{i\}, and its sequence\-level advantage is computed asA^i=Ri−μσ\\hat\{A\}\_\{i\}=\\frac\{R\_\{i\}\-\\mu\}\{\\sigma\}, whereμ\\muandσ\\sigmaare the mean and standard deviation of rewards within the sampled group\. GRPO then updates the policy by optimizing the following objective, which aggregates over all generated tokens:

𝒥GRPO​\(θ\)=𝔼q,\{oi\}∼πθold​\[1∑iTi​∑i=1G∑t=1Tiℓi,t​\(θ;A^i,t\)\],\\mathcal\{J\}\_\{\\mathrm\{GRPO\}\}\(\\theta\)=\\mathbb\{E\}\_\{q,\\,\\\{o\_\{i\}\\\}\\sim\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\}\\left\[\\frac\{1\}\{\\sum\_\{i\}T\_\{i\}\}\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{T\_\{i\}\}\\ell\_\{i,t\}\(\\theta;\\hat\{A\}\_\{i,t\}\)\\right\],\(1\)whereℓi,t​\(θ;A^i,t\)\\ell\_\{i,t\}\(\\theta;\\hat\{A\}\_\{i,t\}\)is the PPO\-style clipped surrogate term111The full GRPO loss function can be referred to Equation \(1\) in the DeepSeek\-R1 technical report\[[2](https://arxiv.org/html/2607.07976#bib.bib2)\]\.\. Since updates are token\-level while advantages are completion\-level, GRPO broadcasts the same sequence\-level advantage to every token position:A^i,t=A^i\\hat\{A\}\_\{i,t\}=\\hat\{A\}\_\{i\}\. This broadcast rule assigns identical credit regardless of token\-level differences\. In reasoning traces, this can dilute useful learning signals and lead to suboptimal policy updates\[[32](https://arxiv.org/html/2607.07976#bib.bib43)\]\. We examine this issue in detail below\.

### 3\.3Positive\-Credit Contamination

Correct reasoning traces can contain locally unreliable tokens even when the final answer is correct\. This occurs because erroneous calculations, irrelevant detours, or malformed continuations may be corrected or bypassed before the final response\. Prior work on process supervision and step\-level verification shows that outcome\-based feedback can overlook flawed intermediate reasoning\[[22](https://arxiv.org/html/2607.07976#bib.bib11),[10](https://arxiv.org/html/2607.07976#bib.bib12)\]\. In addition, models may produce repeated or incoherent text inside generated traces without necessarily disrupting the overall logical flow\[[4](https://arxiv.org/html/2607.07976#bib.bib14),[25](https://arxiv.org/html/2607.07976#bib.bib46)\]\. We observe such locally unreliable tokens in real correct reasoning traces, as illustrated by the case studies in Figure[2](https://arxiv.org/html/2607.07976#S3.F2)\.

The phenomenon that LLMs can produce locally unreliable tokens has been widely studied in the decoding literature, typically through the low\-probability tail of auto\-regressive policies\. Prior work identifies an unreliable subset of this region, where sampled continuations can become incoherent or off\-topic; we refer to tokens from this locally implausible subset asimplausible tail tokens\[[4](https://arxiv.org/html/2607.07976#bib.bib14)\]\. Standard inference\-time strategies such as nucleus sampling suppress these tokens by truncating the tail, which has been shown to improve generation quality\[[4](https://arxiv.org/html/2607.07976#bib.bib14)\]\. In GRPO\-style training, however, full\-vocabulary rollouts allow implausible tail tokens to enter the trajectories used for policy updates\. While implausible tail tokens in failed completions can be penalized by negative advantages, they become problematic when they occur in completions with positive advantages, since the broadcast rule assigns them the same positive credit as contextually sound tokens\. We formalize this as a critical failure mode of the broadcast rule, which we refer to as*Positive\-Credit Contamination*\. The root cause is that the outcome reward verifies only the final answer, but does not establish the contextual validity of each individual token continuation\. As a result, implausible tail tokens can accumulate positive reinforcement across training, progressively biasing the policy toward bad continuations\.

![Refer to caption](https://arxiv.org/html/2607.07976v1/figures/12.png)Figure 2:Examples of unreliable tokens from various sources in real correct reasoning traces\.![Refer to caption](https://arxiv.org/html/2607.07976v1/x2.png)\(a\)Trace lengthHH
![Refer to caption](https://arxiv.org/html/2607.07976v1/x3.png)\(b\)Optimal\-action countnoptn\_\{\\mathrm\{opt\}\}
![Refer to caption](https://arxiv.org/html/2607.07976v1/x4.png)\(c\)Group sizeGG

Figure 3:Demonstration of the effect of*Positive\-Credit Contamination*on a synthetic sequential MDP\.Experiments are conducted to analyze how \(a\) trace lengthHH, \(b\) optimal\-action countnoptn\_\{\\mathrm\{opt\}\}, and \(c\) group sizeGGaffect the contamination behavior\.Δ\\Deltadenotes the final\-step reward gap between oracle and broadcast; the largest gap in each panel is bolded\.To demonstrate the effect of*Positive\-Credit Contamination*on training performance, we conduct a controlled experiment in a synthetic sequential Markov Decision Process \(MDP\) where optimal and tail actions are explicitly defined\. Each trajectory consists of multiple independent steps with local step rewards, while the trajectory\-level return is the sum of these rewards\. This design allows us to isolate credit contamination by comparing two credit assignment rules under the same rollout distribution and reward function: the broadcast rule, which assigns the group\-normalized trajectory advantage to every step, and oracle step\-level credit assignment, which assigns credit using each step’s own reward\. Therefore, any performance gap between the two rules reflects cross\-step credit contamination caused by trajectory\-level advantage broadcast\.

We study three axes that mirror LLM reasoning training: trace lengthHH, optimal\-action sparsity controlled bynoptn\_\{\\mathrm\{opt\}\}, and group sizeGG\. As shown in Figure[3](https://arxiv.org/html/2607.07976#S3.F3), oracle credit assignment consistently outperforms the broadcast rule across all settings, and the gap widens with longer traces, sparser correct actions, and smaller group sizes\. These worst\-case conditions directly correspond to core properties of LLM RLVR: long chain\-of\-thought generation, scarce optimal continuations, and limited rollout budgets in practice, suggesting that*Positive\-Credit Contamination*poses a significant challenge for reasoning\-oriented post\-training\. Details of the synthetic MDP are provided in Appendix[B](https://arxiv.org/html/2607.07976#A2)\.

## 4Methodology

In this section, we introduceTail\-AwareCredit calibratiOn \(TACO\) to calibrate token\-level credit assignments and mitigate*Positive\-Credit Contamination*\.TACOconsists of two modules: adaptive tail\-risk estimation, which scores each token’s likelihood of being a tail token; and tail\-aware credit calibration, which softly suppresses positive credit for high\-risk tokens while leaving low\-risk ones unchanged\. Together, these modules allowTACOto suppress undesirable positive updates while preserving useful exploration with negligible computational cost\.

### 4\.1Adaptive Tail\-Risk Estimation

Mitigating*Positive\-Credit Contamination*requires identifying tail tokens within rewarded completions, so that their positive credit can be selectively suppressed\. However, directly recognizing these tokens during training requires token\-level oracle labels, incurring significant annotation cost and is computationally infeasible\.TACOtherefore uses observable generation\-time statistics as an efficient proxy for estimating whether a sampled token falls into the implausible tail of the policy distribution\.

To estimate tail risk,TACOexamines the local next\-token distribution at each generation step\. A natural signal is the sampled\-token probability: a token assigned very low probability under the current context is more likely to fall outside the policy’s reliable generation region\. However, probability alone is insufficient since rare tokens are heterogeneous\. In high\-entropy contexts, a low probability token may still reflect useful exploration among many plausible alternatives\. In low\-entropy contexts, the same level of rarity is more likely to indicate a locally implausible continuation\. Therefore,TACOcombines token\-level rarity with the uncertainty of the local policy distribution\. Formally, given a tokenoi,to\_\{i,t\}sampled at positionttof completionii,TACOuses its sampled\-token probabilitypi,tp\_\{i,t\}together with the local entropyHi,tH\_\{i,t\}\.

Entropy serves as a context\-level reference for how surprising a token is expected to be\. The further a token’s surprisal exceeds the level expected by the local entropy, the more likely it is to be in the unreliable tail, as such deviations indicate generation beyond the policy’s well\-calibrated region\.TACOtherefore defines the tail\-risk scoreri,ttailr^\{\\mathrm\{tail\}\}\_\{i,t\}as how far the token’s surprisal exceeds that expected by the local entropy:

ri,ttail=−log⁡pi,t⏟token surprisal−Hi,t⏟expected surprisal\+log⁡α,r^\{\\mathrm\{tail\}\}\_\{i,t\}=\\underbrace\{\-\\log p\_\{i,t\}\}\_\{\\text\{token surprisal\}\}\-\\underbrace\{H\_\{i,t\}\}\_\{\\text\{expected surprisal\}\}\+\\log\\alpha,\(2\)A largerα\\alphamakes tail\-risk identification more aggressive, identifying more low\-probability tokens as risky, but risks suppressing useful rare tokens together with implausible ones\. Withα\\alphafixed, a positive score indicates that the token is considered risky and subject to credit suppression\.

### 4\.2Tail\-Aware Credit Calibration

The tail\-risk score provides a continuous estimate of how likely each sampled token is to be locally implausible\. Based on this estimate,TACOassigns each token a risk\-dependent weight that softly reduces its positive credit as the risk increases\. This smooth down\-weighting preserves partial gradients for genuinely useful low\-probability patterns, allowing them to accumulate reinforcement across rewarded trajectories, while progressively dampening accidental noise\. Formally,TACOdefines the token weightwi,tw\_\{i,t\}as:

wi,t=\{1−λ​\(1−exp⁡\(−ri,ttail\)\),ri,ttail\>0,1,ri,ttail≤0,\.w\_\{i,t\}=\\begin\{cases\}1\-\\lambda\\left\(1\-\\exp\\left\(\-r^\{\\mathrm\{tail\}\}\_\{i,t\}\\right\)\\right\),&r^\{\\mathrm\{tail\}\}\_\{i,t\}\>0,\\\\\[3\.0pt\] 1,&r^\{\\mathrm\{tail\}\}\_\{i,t\}\\leq 0,\\par\\end\{cases\}\.\(3\)Tokens with non\-positive risk scores retain full weight, while high\-risk tokens receive smoothly decaying weights, bounded below by1−λ1\-\\lambda, whereλ∈\(0,1\)\\lambda\\in\(0,1\)is a hyperparameter controlling the maximum suppression strength\.TACOthen applies this weight to the broadcast advantage to produce a calibrated token\-level advantage:

A^i,tTACO=wi,t𝕀​\[A^i\>0\]​A^i\.\\hat\{A\}^\{\\texttt\{TACO\}\{\}\}\_\{i,t\}=w\_\{i,t\}^\{\\mathbb\{I\}\[\\hat\{A\}\_\{i\}\>0\]\}\\hat\{A\}\_\{i\}\.\(4\)
Only positive advantages are modulated; negative advantages are left unchanged to preserve the suppression signal from failed trajectories\.

### 4\.3Algorithm ofTACO

TACOintegrates into standard GRPO training loop by replacing the broadcast advantageA^i\\hat\{A\}\_\{i\}with calibrated token\-level creditA^i,tTACO\\hat\{A\}^\{\\texttt\{TACO\}\{\}\}\_\{i,t\}from Eq\. \([4](https://arxiv.org/html/2607.07976#S4.E4)\)\. All other components of the GRPO surrogate remain unchanged\. SinceTACOrelies solely on the token probabilities and entropy computed during the forward pass, it introduces negligible overhead\. Algorithm[1](https://arxiv.org/html/2607.07976#alg1)summarizes the full procedure\.

Algorithm 1TACO: Tail\-Aware Credit Calibration1:Prompt dataset

𝒟\\mathcal\{D\}; policy

πθ\\pi\_\{\\theta\}; verifier

ℛ\\mathcal\{R\}; group size

GG; hyperparameters

α,λ\\alpha,\\lambda
2:foreach training iterationdo

3:\#\# Standard GRPO rollout

4:

πθold←πθ\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\\leftarrow\\pi\_\{\\theta\}
5:Sample prompt batch

ℬ∼𝒟\\mathcal\{B\}\\sim\\mathcal\{D\}
6:Generate

GGcompletions for each prompt using

πθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}
7:Compute rewards and group\-normalized advantages

A^i\\hat\{A\}\_\{i\}⊳\\trianglerightsequence\-level credit

8:

9:\#\# Tail\-aware credit calibration

10:Estimate token\-level tail risk

ri,ttailr^\{\\mathrm\{tail\}\}\_\{i,t\}using local surprisal and entropy by Eq\. \([2](https://arxiv.org/html/2607.07976#S4.E2)\)

11:Convert tail risk into credit\-suppression weights

wi,tw\_\{i,t\}by Eq\. \([3](https://arxiv.org/html/2607.07976#S4.E3)\)

12:Calibrate positive sequence\-level credit into token\-level advantages

A^i,tTACO\\hat\{A\}^\{\\texttt\{TACO\}\{\}\}\_\{i,t\}by Eq\. \([4](https://arxiv.org/html/2607.07976#S4.E4)\)

13:

14:\#\# Standard GRPO policy update

15:Update

πθ\\pi\_\{\\theta\}with the GRPO clipped surrogate using

A^i,tTACO\\hat\{A\}^\{\\texttt\{TACO\}\{\}\}\_\{i,t\}⊳\\trianglerighttoken\-level credit

16:endfor

## 5Experiments

In this section, we conduct experiments to verify the effectiveness of our method, aiming to answer the following research questions:RQ1:DoesTACOimprove performance across diverse reasoning benchmarks?RQ2:DoesTACOsustain performance gains under longer RL training while avoiding late\-stage collapse?RQ3:How doesTACOaffect token\-level training behavior?

### 5\.1Experimental Setup

#### Models\.

We evaluateTACOon three LLMs from different model families and scales: Qwen3\-1\.7B\-Base, Qwen3\-4B\-Base\[[30](https://arxiv.org/html/2607.07976#bib.bib25)\], and Qwen2\.5\-Math\-7B\[[31](https://arxiv.org/html/2607.07976#bib.bib34)\]\. For the main experiments, we use DAPO\-Math\-17K222[https://huggingface\.co/datasets/BytedTsinghua\-SIA/DAPO\-Math\-17k](https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k)as the training dataset\. Our training codebase is built onverl\[[23](https://arxiv.org/html/2607.07976#bib.bib24)\], and we follow its standard GRPO training recipe\. All methods share the same configuration, with method\-specific hyperparameters set to the values reported in their original papers\.

#### Datasets\.

We evaluate the trained models on six mathematical reasoning benchmarks: AIME 2024\[[36](https://arxiv.org/html/2607.07976#bib.bib27)\], AIME 2025\[[37](https://arxiv.org/html/2607.07976#bib.bib28)\], AMC 2023\[[13](https://arxiv.org/html/2607.07976#bib.bib29)\], MATH\-500\[[10](https://arxiv.org/html/2607.07976#bib.bib12)\], Minerva Math\[[8](https://arxiv.org/html/2607.07976#bib.bib30)\], and OlympiadBench\[[3](https://arxiv.org/html/2607.07976#bib.bib31)\]\. To assess out\-of\-distribution generalization beyond mathematics, we further evaluate on two scientific reasoning benchmarks: MMLU\-Pro\[[24](https://arxiv.org/html/2607.07976#bib.bib32)\]and GPQA\-Diamond\[[17](https://arxiv.org/html/2607.07976#bib.bib33)\]\. We report avg@32 for AIME and AMC, avg@4 for MATH\-500, Minerva Math, and OlympiadBench, and avg@16 for MMLU\-Pro and GPQA\-Diamond\.

#### Baselines\.

We compareTACOagainst three baselines\.GRPOis the standard critic\-free RLVR baseline, we implement it with clip\-higher asymmetric clipping to enable long\-term training\[[19](https://arxiv.org/html/2607.07976#bib.bib5),[34](https://arxiv.org/html/2607.07976#bib.bib6)\]\.GRPO w/ Adv\. Reweightingreweights low\-probability tokens to reduce their over\-dominance in policy\-gradient updates\[[33](https://arxiv.org/html/2607.07976#bib.bib18)\]\.STAPOdampens updates from tokens with disproportionately large gradients for more stable optimization\[[11](https://arxiv.org/html/2607.07976#bib.bib19)\]\. Additional details are provided in Appendix[A](https://arxiv.org/html/2607.07976#A1)\.

Table 1:Main results on in\-domain mathematical reasoning benchmarks and out\-of\-distribution scientific reasoning benchmarks\. Best results are in bold and second\-best results are underlined\. TheΔ\\Deltarow reports absolute improvement over GRPO\.MethodAIME24AIME25AMC23MATH\-500MinervaOlympiadMMLU\-ProGPQA\-DAvg\.Qwen3\-1\.7B\-BaseGRPO9\.487\.2946\.7266\.2025\.8329\.0124\.9420\.7128\.77GRPO w/ Adv\. Reweighting11\.258\.4447\.0364\.7524\.6329\.1129\.8423\.3629\.80STAPO12\.299\.3846\.4168\.3523\.0730\.3624\.3217\.8029\.00TACO\(Ours\)14\.389\.0649\.4568\.3525\.7431\.7130\.4524\.4331\.70Δ\\Deltavs\. GRPO\(\+4\.90\)\(\+1\.77\)\(\+2\.73\)\(\+2\.15\)\(\-0\.09\)\(\+2\.70\)\(\+5\.51\)\(\+3\.72\)\(\+2\.93\)Qwen3\-4B\-BaseGRPO25\.7320\.8368\.1376\.5034\.6536\.3536\.2626\.1440\.57GRPO w/ Adv\. Reweighting23\.5421\.5669\.3878\.8535\.9438\.2839\.2026\.6441\.67STAPO24\.6921\.9872\.8977\.0033\.6436\.8338\.8425\.8841\.47TACO\(Ours\)27\.0823\.8571\.8880\.0535\.6740\.0641\.9029\.1743\.71Δ\\Deltavs\. GRPO\(\+1\.35\)\(\+3\.02\)\(\+3\.75\)\(\+3\.55\)\(\+1\.02\)\(\+3\.71\)\(\+5\.64\)\(\+3\.03\)\(\+3\.14\)Qwen2\.5\-Math\-7BGRPO29\.9016\.7772\.6683\.3051\.0146\.7430\.7523\.6444\.35GRPO w/ Adv\. Reweighting28\.9617\.2974\.8482\.9550\.1846\.7729\.0524\.4344\.31STAPO28\.7517\.0873\.2884\.3053\.5847\.8127\.6424\.4944\.62TACO\(Ours\)32\.4019\.7978\.4484\.6555\.5149\.4130\.5324\.8446\.95Δ\\Deltavs\. GRPO\(\+2\.50\)\(\+3\.02\)\(\+5\.78\)\(\+1\.35\)\(\+4\.50\)\(\+2\.67\)\(\-0\.22\)\(\+1\.20\)\(\+2\.60\)

### 5\.2Performance on Reasoning Tasks \(RQ1\)

Table[1](https://arxiv.org/html/2607.07976#S5.T1)shows the accuracy \(%\) ofTACOand baseline methods across all benchmarks\.

#### Performance Gain\.

As shown in Table[1](https://arxiv.org/html/2607.07976#S5.T1),TACOsurpasses all baselines on average and consistently improves performance across different models and benchmarks\. These gains suggest thatTACOimproves performance by calibrating implausible positive credit among low\-probability tokens, going beyond merely stabilizing their updates as in STAPO and GRPO w/ Adv\. Reweighting\. By down\-weighting unreliable positive credit while preserving useful rare\-token exploration,TACOprovides a more robust optimization signal\.

#### OOD Generalization\.

TACOdemonstrates strong generalization beyond the mathematical training domain\.TACOmaintains stable gains on MMLU\-Pro and GPQA\-Diamond, which cover scientific fields including physics, chemistry, and biology \. By selectively suppressing tail tokens while preserving credit for useful exploratory ones,TACOavoids overly aggressive gradient suppression that could limit cross\-domain generalization\. This design helps the policy develop general reasoning capabilities rather than overfitting to domain\-specific patterns\.

#### Model Scale Consistency\.

TACOachieves consistent improvements across models of different parameter scales\. Because the tail\-risk score uses each policy’s token probability and local entropy, the calibration naturally adapts to each model’s uncertainty profile, allowingTACOto improve reasoning capability without requiring model\-specific tuning\.

### 5\.3Analysis of Training Dynamics

![Refer to caption](https://arxiv.org/html/2607.07976v1/x5.png)\(a\)Training reward
![Refer to caption](https://arxiv.org/html/2607.07976v1/x6.png)\(b\)Policy entropy
![Refer to caption](https://arxiv.org/html/2607.07976v1/x7.png)\(c\)Response length

Figure 4:Training dynamics ofTACOand baseline methods on Qwen3\-1\.7B\-Base\. Results on other models are provided in Appendix[C](https://arxiv.org/html/2607.07976#A3)\.![Refer to caption](https://arxiv.org/html/2607.07976v1/x8.png)\(a\)AIME24 test accuracyprogre
![Refer to caption](https://arxiv.org/html/2607.07976v1/x9.png)\(b\)Policy entropy

Figure 5:Comparison ofTACOand GRPO on Qwen3\-1\.7B\-Base with extended training\.To examine howTACOaffects training, we compare its reward, policy entropy, and response length dynamics with baseline methods in Figure[4](https://arxiv.org/html/2607.07976#S5.F4)\.

#### Learning Effectiveness\.

As shown in Figure[4\(a\)](https://arxiv.org/html/2607.07976#S5.F4.sf1),TACOachieves consistently higher training accuracy than baselines\. This improvement stems from the selective nature ofTACO’s credit calibration: by suppressing positive credit for locally implausible tokens, each policy update concentrates reinforcement on contextually more reliable reasoning steps, leading to more effective policy optimization\.

#### Stable Entropy and Sustained Exploration\.

As shown in Figure[4\(b\)](https://arxiv.org/html/2607.07976#S5.F4.sf2)and[4\(c\)](https://arxiv.org/html/2607.07976#S5.F4.sf3),TACOmaintains relatively lower and more stable token entropy compared to baselines while simultaneously producing longer responses throughout training\. The stable entropy indicates thatTACOsuppresses erratic tail\-token updates without collapsing the policy’s exploration capacity\. Meanwhile, the increased response length suggests that the policy develops more complex and complete reasoning ability under this stable optimization\. Together, these dynamics suggest thatTACO’s credit calibration enables the policy to maintain effective exploration and develop more robust reasoning capabilities\.

### 5\.4Long\-Horizon Training Stability \(RQ2\)

A common challenge in GRPO\-style training is that the policy tends to plateau or degrade as training progresses, limiting the benefit of extended optimization\. To evaluateTACO’s robustness in this regime, we extend training of both GRPO andTACOon Qwen3\-1\.7B\-Base from the default 300 steps to 600 steps\. As shown in Figure[5](https://arxiv.org/html/2607.07976#S5.F5), GRPO’s test accuracy on AIME24 plateaus around step 450 and begins to degrade thereafter, whileTACOcontinues to improve throughout the extended phase\. The entropy dynamics further show that GRPO exhibits large fluctuations as training proceeds, whereasTACOmaintains a smooth and stable entropy profile, avoiding both entropy collapse and explosion\. These results suggest thatTACOenables more stable long\-horizon optimization, allowing the policy to continue benefiting from extended training in regimes where standard GRPO stagnates\.

### 5\.5Behavioral Analysis \(RQ3\)

To further illustrate howTACOmitigates*Positive\-Credit Contamination*, we present a rewarded trace sampled during training in Figure[6](https://arxiv.org/html/2607.07976#S5.F6)\. Although the completion reaches the correct answer, its early generation contains unreliable tail tokens, including unnecessary table formatting, corrupted math terms \(i\), broken formula \(In\), mixed\-language noise \(Daisy\), and irrelevant text \(\-\-the,list\)\. Under the GRPO broadcast rule, these unreliable behaviors would be mis\-amplified by receiving the same positive advantage as useful reasoning tokens\. In contrast,TACOassigns suppressed credit to these tokens while preserving full credit for the later valid solution steps, showing thatTACOcan help suppress harmful local behaviors without weakening coherent reasoning\. Additional qualitative cases and training diagnostics are provided in Appendix[C](https://arxiv.org/html/2607.07976#A3)\.

![Refer to caption](https://arxiv.org/html/2607.07976v1/x10.png)Figure 6:A case fromQwen3\-4B\-Base\.TACOselectively downweights tail tokens \(shown in red; darker red indicates stronger suppression\), while preserving coherent reasoning steps \(shown in blue\)\.Table 2:Sensitivity Analysis ofα\\alphaandλ\\lambdaon Qwen3\-1\.7B\-Base model\. Best results are in bold and second\-best results are underlined\.α\\alphaλ\\lambdaAIME24AIME25AMC23MATH\-500MinervaOlympiadMMLU\-ProGPQA\-DAvg\.Qwen3\-1\.7B\-Base0\.010\.615\.109\.5847\.5068\.1524\.8228\.9730\.9523\.9931\.130\.010\.914\.389\.0649\.4568\.3525\.7431\.7130\.4524\.4331\.700\.0050\.612\.818\.6448\.1567\.2024\.7229\.3828\.8526\.2630\.750\.0050\.913\.239\.0646\.5767\.5525\.5530\.0829\.8525\.8130\.96

### 5\.6Hyperparameter Sensitivity Study

We evaluate the sensitivity ofTACOto its two key hyperparameters on Qwen3\-1\.7B\-Base model: the tail\-risk strictnessα\\alphaand the suppression strengthλ\\lambda\. We varyα\\alphaandλ\\lambdaaround the default configuration while keeping all other training settings fixed\. As shown in Table[2](https://arxiv.org/html/2607.07976#S5.T2), our default setting,\(α,λ\)=\(0\.01,0\.9\)\(\\alpha,\\lambda\)=\(0\.01,0\.9\), achieves the best average performance, whileTACOremains effective across a reasonable range of settings\. This suggests that the gains do not rely on a brittle hyperparameter choice\. We also observe that settingα\\alphatoo large, e\.g\.,α=0\.1\\alpha=0\.1, leads to clear early\-stage performance degradation\. This suggests that overly aggressive tail\-risk identification may suppress useful low\-probability behaviors together with unreliable ones\.

## 6Conclusion

In this work, we identify*Positive\-Credit Contamination*as a failure mode of GRPO\-style RLVR, where unreliable tail tokens can receive undesired positive updates\. To address this issue, we proposeTail\-AwareCredit calibratiOn \(TACO\), which calibrates token\-level credit to avoid reinforcing flawed local continuations\.TACOestimates each token’s tail risk from its sampled probability and local entropy, then uses this risk to softly reduce positive credit for high\-risk tokens while preserving sound reasoning patterns\. Experiments across three LLMs and eight reasoning benchmarks show thatTACOconsistently improves over GRPO\-style baselines and supports more stable long\-horizon training\. These results demonstrate the effectiveness ofTACOfor improving reasoning\-oriented RLVR\.

## References

- \[1\]Z\. Chen, X\. Ma, G\. Fang, R\. Yu, and X\. Wang\(2025\)VeriThinker: learning to verify makes reasoning model efficient\.CoRRabs/2505\.17941\.External Links:[Link](https://doi.org/10.48550/arXiv.2505.17941),[Document](https://dx.doi.org/10.48550/ARXIV.2505.17941),2505\.17941Cited by:[Appendix A](https://arxiv.org/html/2607.07976#A1.SS0.SSS0.Px2.p1.7)\.
- \[2\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.Cited by:[§1](https://arxiv.org/html/2607.07976#S1.p1.1),[footnote 1](https://arxiv.org/html/2607.07976#footnote1)\.
- \[3\]C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. L\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang, J\. Liu, L\. Qi, Z\. Liu, and M\. Sun\(2024\)OlympiadBench: A challenging benchmark for promoting AGI with olympiad\-level bilingual multimodal scientific problems\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2024, Bangkok, Thailand, August 11\-16, 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),pp\. 3828–3850\.External Links:[Link](https://doi.org/10.18653/v1/2024.acl-long.211),[Document](https://dx.doi.org/10.18653/V1/2024.ACL-LONG.211)Cited by:[§5\.1](https://arxiv.org/html/2607.07976#S5.SS1.SSS0.Px2.p1.1)\.
- \[4\]A\. Holtzman, J\. Buys, L\. Du, M\. Forbes, and Y\. Choi\(2020\)The curious case of neural text degeneration\.In8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26\-30, 2020,External Links:[Link](https://openreview.net/forum?id=rygGQyrFvH)Cited by:[§3\.3](https://arxiv.org/html/2607.07976#S3.SS3.p1.1),[§3\.3](https://arxiv.org/html/2607.07976#S3.SS3.p2.1)\.
- \[5\]G\. Huang, T\. Xu, M\. Wang, Q\. Yi, X\. Gong, S\. Li, R\. Xiong, K\. Li, Y\. Jiang, and B\. Zhou\(2025\)Low\-probability tokens sustain exploration in reinforcement learning with verifiable reward\.CoRRabs/2510\.03222\.External Links:[Link](https://doi.org/10.48550/arXiv.2510.03222),[Document](https://dx.doi.org/10.48550/ARXIV.2510.03222),2510\.03222Cited by:[§2](https://arxiv.org/html/2607.07976#S2.SS0.SSS0.Px1.p1.1)\.
- \[6\]J\. Huang, X\. Chen, S\. Mishra, H\. S\. Zheng, A\. W\. Yu, X\. Song, and D\. Zhou\(2024\)Large language models cannot self\-correct reasoning yet\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=IkmD3fKBPQ)Cited by:[§1](https://arxiv.org/html/2607.07976#S1.p2.1)\.
- \[7\]A\. Kumar, N\. Kumar, and S\. Gupta\(2026\)Execution\-grounded credit assignment for grpo in code generation\.arXiv preprint arXiv:2603\.16158\.Cited by:[§1](https://arxiv.org/html/2607.07976#S1.p3.1),[§2](https://arxiv.org/html/2607.07976#S2.SS0.SSS0.Px2.p1.1)\.
- \[8\]A\. Lewkowycz, A\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo, Y\. Wu, B\. Neyshabur, G\. Gur\-Ari, and V\. Misra\(2022\)Solving quantitative reasoning problems with language models\.InAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 \- December 9, 2022,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2022/hash/18abbeef8cfe9203fdf9053c9c4fe191-Abstract-Conference.html)Cited by:[§5\.1](https://arxiv.org/html/2607.07976#S5.SS1.SSS0.Px2.p1.1)\.
- \[9\]Z\. Li, L\. Kang, F\. Xiao, L\. Xing, Q\. Si, Z\. Li, W\. Gong, D\. Yang, Y\. Xiao, and H\. Guo\(2026\)Outcome\-grounded advantage reshaping for fine\-grained credit assignment in mathematical reasoning\.CoRRabs/2601\.07408\.External Links:[Link](https://doi.org/10.48550/arXiv.2601.07408),[Document](https://dx.doi.org/10.48550/ARXIV.2601.07408),2601\.07408Cited by:[§1](https://arxiv.org/html/2607.07976#S1.p3.1),[§2](https://arxiv.org/html/2607.07976#S2.SS0.SSS0.Px2.p1.1)\.
- \[10\]H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe\(2023\)Let’s verify step by step\.External Links:2305\.20050,[Link](https://arxiv.org/abs/2305.20050)Cited by:[§1](https://arxiv.org/html/2607.07976#S1.p2.1),[§3\.3](https://arxiv.org/html/2607.07976#S3.SS3.p1.1),[§5\.1](https://arxiv.org/html/2607.07976#S5.SS1.SSS0.Px2.p1.1)\.
- \[11\]S\. Liu, Z\. He, G\. Zhan, L\. Tao, Z\. Zheng, J\. Wu, Y\. Wang, Y\. Guan, K\. Sheng, B\. Zhang, K\. Li, J\. Duan, and S\. E\. Li\(2026\)STAPO: stabilizing reinforcement learning for llms by silencing rare spurious tokens\.CoRRabs/2602\.15620\.External Links:[Link](https://doi.org/10.48550/arXiv.2602.15620),[Document](https://dx.doi.org/10.48550/ARXIV.2602.15620),2602\.15620Cited by:[§2](https://arxiv.org/html/2607.07976#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2607.07976#S5.SS1.SSS0.Px3.p1.1)\.
- \[12\]Z\. Liu, M\. Liu, S\. Wen, M\. Cai, B\. Cui, C\. He, and W\. Zhang\(2026\)Heterogeneous adaptive policy optimization: tailoring optimization to every token’s nature\.External Links:2509\.16591,[Link](https://arxiv.org/abs/2509.16591)Cited by:[§1](https://arxiv.org/html/2607.07976#S1.p3.1),[§2](https://arxiv.org/html/2607.07976#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.07976#S2.SS0.SSS0.Px2.p1.1)\.
- \[13\]math\-ai Team\(2024\)AMC23: american mathematics competitions 2023 test set\.Note:[https://huggingface\.co/datasets/math\-ai/amc23](https://huggingface.co/datasets/math-ai/amc23)Hugging Face dataset\. Accessed: 2026\-05\-06Cited by:[§5\.1](https://arxiv.org/html/2607.07976#S5.SS1.SSS0.Px2.p1.1)\.
- \[14\]OpenAI\(2024\)OpenAI o1 system card\.CoRRabs/2412\.16720\.External Links:[Link](https://doi.org/10.48550/arXiv.2412.16720),[Document](https://dx.doi.org/10.48550/ARXIV.2412.16720),2412\.16720Cited by:[§1](https://arxiv.org/html/2607.07976#S1.p1.1)\.
- \[15\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. F\. Christiano, J\. Leike, and R\. Lowe\(2022\)Training language models to follow instructions with human feedback\.CoRRabs/2203\.02155\.External Links:[Link](https://doi.org/10.48550/arXiv.2203.02155),[Document](https://dx.doi.org/10.48550/ARXIV.2203.02155),2203\.02155Cited by:[§1](https://arxiv.org/html/2607.07976#S1.p1.1)\.
- \[16\]P\. Parthasarathi, M\. Reymond, B\. Chen, Y\. Cui, and S\. Chandar\(2025\)GRPO\-λ\\lambda: credit assignment improves llm reasoning\.External Links:2510\.00194,[Link](https://arxiv.org/abs/2510.00194)Cited by:[§1](https://arxiv.org/html/2607.07976#S1.p3.1),[§2](https://arxiv.org/html/2607.07976#S2.SS0.SSS0.Px2.p1.1)\.
- \[17\]D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman\(2023\)Gpqa: a graduate\-level google\-proof q&a benchmark\.arXiv preprint arXiv:2311\.12022\.Cited by:[§5\.1](https://arxiv.org/html/2607.07976#S5.SS1.SSS0.Px2.p1.1)\.
- \[18\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.CoRRabs/1707\.06347\.External Links:[Link](http://arxiv.org/abs/1707.06347),1707\.06347Cited by:[§1](https://arxiv.org/html/2607.07976#S1.p1.1)\.
- \[19\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo\(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.CoRRabs/2402\.03300\.External Links:[Link](https://doi.org/10.48550/arXiv.2402.03300),[Document](https://dx.doi.org/10.48550/ARXIV.2402.03300),2402\.03300Cited by:[§1](https://arxiv.org/html/2607.07976#S1.p1.1),[§5\.1](https://arxiv.org/html/2607.07976#S5.SS1.SSS0.Px3.p1.1)\.
- \[20\]H\. Tan, Z\. Wang, J\. Pan, J\. Lin, H\. Wang, Y\. Wu, T\. Chen, Z\. Zheng, Z\. Tang, and H\. Yang\(2025\)Gtpo and grpo\-s: token and sequence\-level reward shaping with policy entropy\.arXiv preprint arXiv:2508\.04349\.Cited by:[§1](https://arxiv.org/html/2607.07976#S1.p3.1),[§2](https://arxiv.org/html/2607.07976#S2.SS0.SSS0.Px2.p1.1)\.
- \[21\]H\. Tran, Z\. Yao, and H\. Yu\(2025\)Exploiting tree structure for credit assignment in RL training of llms\.CoRRabs/2509\.18314\.External Links:[Link](https://doi.org/10.48550/arXiv.2509.18314),[Document](https://dx.doi.org/10.48550/ARXIV.2509.18314),2509\.18314Cited by:[§1](https://arxiv.org/html/2607.07976#S1.p3.1),[§2](https://arxiv.org/html/2607.07976#S2.SS0.SSS0.Px2.p1.1)\.
- \[22\]J\. Uesato, N\. Kushman, R\. Kumar, H\. F\. Song, N\. Y\. Siegel, L\. Wang, A\. Creswell, G\. Irving, and I\. Higgins\(2022\)Solving math word problems with process\- and outcome\-based feedback\.CoRRabs/2211\.14275\.External Links:[Link](https://doi.org/10.48550/arXiv.2211.14275),[Document](https://dx.doi.org/10.48550/ARXIV.2211.14275),2211\.14275Cited by:[§1](https://arxiv.org/html/2607.07976#S1.p2.1),[§3\.3](https://arxiv.org/html/2607.07976#S3.SS3.p1.1)\.
- \[23\]verl\-project\(2024\)verl: volcano engine reinforcement learning for llms\.Note:[https://github\.com/verl\-project/verl](https://github.com/verl-project/verl)GitHub repository\. Accessed: 2026\-05\-06Cited by:[§5\.1](https://arxiv.org/html/2607.07976#S5.SS1.SSS0.Px1.p1.1)\.
- \[24\]Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. Chen\(2024\)MMLU\-pro: A more robust and challenging multi\-task language understanding benchmark\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2024/hash/ad236edc564f3e3156e1b2feafb99a24-Abstract-Datasets%5C_and%5C_Benchmarks%5C_Track.html)Cited by:[§5\.1](https://arxiv.org/html/2607.07976#S5.SS1.SSS0.Px2.p1.1)\.
- \[25\]S\. Welleck, I\. Kulikov, S\. Roller, E\. Dinan, K\. Cho, and J\. Weston\(2020\)Neural text generation with unlikelihood training\.In8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26\-30, 2020,External Links:[Link](https://openreview.net/forum?id=SJeYe0NtvH)Cited by:[§3\.3](https://arxiv.org/html/2607.07976#S3.SS3.p1.1)\.
- \[26\]C\. Xie, R\. Pan, X\. Wu, Y\. Zhang, J\. Fu, T\. Gao, and G\. Zhou\(2025\)Unlocking exploration in RLVR: uncertainty\-aware advantage shaping for deeper reasoning\.CoRRabs/2510\.10649\.External Links:[Link](https://doi.org/10.48550/arXiv.2510.10649),[Document](https://dx.doi.org/10.48550/ARXIV.2510.10649),2510\.10649Cited by:[§1](https://arxiv.org/html/2607.07976#S1.p3.1)\.
- \[27\]Z\. Xu, G\. Wang, Y\. Chuang, G\. Zheng, A\. S\. Szalay, Z\. Liu, and V\. Braverman\(2025\)DTS: enhancing large reasoning models via decoding tree sketching\.CoRRabs/2511\.00640\.External Links:[Link](https://doi.org/10.48550/arXiv.2511.00640),[Document](https://dx.doi.org/10.48550/ARXIV.2511.00640),2511\.00640Cited by:[Appendix A](https://arxiv.org/html/2607.07976#A1.SS0.SSS0.Px2.p1.7)\.
- \[28\]Z\. Xu, G\. Wang, G\. Zheng, Y\. Chuang, A\. Szalay, X\. Hu, and V\. Braverman\(2025\)Self\-ensemble: mitigating confidence distortion for large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, November 4\-9, 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),pp\. 16603–16615\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.902/)Cited by:[Appendix A](https://arxiv.org/html/2607.07976#A1.SS0.SSS0.Px2.p1.7)\.
- \[29\]Z\. Xu, R\. Zhang, Y\. Chuang, X\. Lou, H\. A\. D\. Le, O\. Gal, A\. S\. Szalay, Z\. Xu, G\. Wang, and V\. Braverman\(2026\)Learning at the right pace: adaptive data scheduling improves llm reinforcement learning\.arXiv preprint arXiv:2606\.22305\.Cited by:[Appendix A](https://arxiv.org/html/2607.07976#A1.SS0.SSS0.Px2.p1.7)\.
- \[30\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu\(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§5\.1](https://arxiv.org/html/2607.07976#S5.SS1.SSS0.Px1.p1.1)\.
- \[31\]A\. Yang, B\. Zhang, B\. Hui, B\. Gao, B\. Yu, C\. Li, D\. Liu, J\. Tu, J\. Zhou, J\. Lin, K\. Lu, M\. Xue, R\. Lin, T\. Liu, X\. Ren, and Z\. Zhang\(2024\)Qwen2\.5\-math technical report: toward mathematical expert model via self\-improvement\.CoRRabs/2409\.12122\.External Links:[Link](https://doi.org/10.48550/arXiv.2409.12122),[Document](https://dx.doi.org/10.48550/ARXIV.2409.12122),2409\.12122Cited by:[§5\.1](https://arxiv.org/html/2607.07976#S5.SS1.SSS0.Px1.p1.1)\.
- \[32\]M\. Y\. R\. Yang, H\. Bai, I\. Wu, G\. Yang, A\. Setlur, and A\. Kumar\(2026\)InT: self\-proposed interventions enable credit assignment in LLM reasoning\.CoRRabs/2601\.14209\.External Links:[Link](https://doi.org/10.48550/arXiv.2601.14209),[Document](https://dx.doi.org/10.48550/ARXIV.2601.14209),2601\.14209Cited by:[§3\.2](https://arxiv.org/html/2607.07976#S3.SS2.p1.11)\.
- \[33\]Z\. Yang, X\. Luo, Z\. Wang, D\. Han, Z\. He, D\. Li, and Y\. Xu\(2025\)Do not let low\-probability tokens over\-dominate in rl for llms\.arXiv preprint arXiv:2505\.12929\.Cited by:[§2](https://arxiv.org/html/2607.07976#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2607.07976#S5.SS1.SSS0.Px3.p1.1)\.
- \[34\]Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, T\. Fan, G\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, W\. Dai, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, M\. Qiao, Y\. Wu, and M\. Wang\(2025\)DAPO: an open\-source LLM reinforcement learning system at scale\.CoRRabs/2503\.14476\.External Links:[Link](https://doi.org/10.48550/arXiv.2503.14476),[Document](https://dx.doi.org/10.48550/ARXIV.2503.14476),2503\.14476Cited by:[§1](https://arxiv.org/html/2607.07976#S1.p1.1),[§2](https://arxiv.org/html/2607.07976#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2607.07976#S5.SS1.SSS0.Px3.p1.1)\.
- \[35\]S\. Yu and L\. Li\(2026\)ERPO: token\-level entropy\-regulated policy optimization for large reasoning models\.CoRRabs/2603\.28204\.External Links:[Link](https://doi.org/10.48550/arXiv.2603.28204),[Document](https://dx.doi.org/10.48550/ARXIV.2603.28204),2603\.28204Cited by:[§1](https://arxiv.org/html/2607.07976#S1.p3.1)\.
- \[36\]Y\. Zhang and Math\-AI Team\(2024\)American invitational mathematics examination \(AIME\) 2024\.Note:[https://huggingface\.co/datasets/math\-ai/aime24](https://huggingface.co/datasets/math-ai/aime24)Hugging Face dataset\. Accessed: 2026\-05\-06Cited by:[§5\.1](https://arxiv.org/html/2607.07976#S5.SS1.SSS0.Px2.p1.1)\.
- \[37\]Y\. Zhang and Math\-AI Team\(2025\)American invitational mathematics examination \(AIME\) 2025\.Note:[https://huggingface\.co/datasets/math\-ai/aime25](https://huggingface.co/datasets/math-ai/aime25)Hugging Face dataset\. Accessed: 2026\-05\-06Cited by:[§5\.1](https://arxiv.org/html/2607.07976#S5.SS1.SSS0.Px2.p1.1)\.
- \[38\]C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang, J\. Zhou, and J\. Lin\(2025\)Group sequence policy optimization\.CoRRabs/2507\.18071\.External Links:[Link](https://doi.org/10.48550/arXiv.2507.18071),[Document](https://dx.doi.org/10.48550/ARXIV.2507.18071),2507\.18071Cited by:[§1](https://arxiv.org/html/2607.07976#S1.p1.1)\.

## Appendix AImplementation Details

#### Training Setup\.

We train all methods with vLLM\-based rollout generation and no KL penalty\. Unless otherwise specified, we use group size88, learning rate1×10−61\\times 10^\{\-6\}with 10 warmup steps, maximum prompt length 1024, maximum response length 4096, rollout temperature 1\.0, top\-p=1\.0p=1\.0, and asymmetric clipping with lower and upper clip ranges\(0\.2,0\.28\)\(0\.2,0\.28\)\. For the Qwen3 series, the rollout batch size is 256 and the PPO mini\-batch size is 64; Qwen3\-1\.7B\-Base and Qwen3\-4B\-Base are trained for 300 and 250 steps, respectively\. For Qwen2\.5\-Math\-7B, the rollout batch size is 512, the PPO mini\-batch size is 32, and the model is trained for 200 steps\.

#### Baselines and Evaluation\.

For method\-specific hyperparameters, TACO uses coefficients0\.010\.01and0\.90\.9\. For GRPO with Advantage Reweighting, we set the reweighting coefficient to0\.10\.1\. For STAPO, we set the selected token ratio to20%20\\%and the threshold to0\.0020\.002\. During evaluation, we use top\-p=0\.7p=0\.7and temperature0\.90\.9for all models and benchmarks\. We follow existing work\[[27](https://arxiv.org/html/2607.07976#bib.bib49),[1](https://arxiv.org/html/2607.07976#bib.bib48),[28](https://arxiv.org/html/2607.07976#bib.bib47),[29](https://arxiv.org/html/2607.07976#bib.bib50)\]to construct evaluation prompts, extract final answers, and build validation sets\. We select the checkpoint with the highest validation accuracy for each method and model for final reporting\.

## Appendix BControlled Experiment

#### Task Setting\.

We instantiate the synthetic sequential MDP described in Section[3\.3](https://arxiv.org/html/2607.07976#S3.SS3)as follows\. Each trajectory has lengthHH\. At each step, the agent samples from a large action space\|𝒜\|=105\|\\mathcal\{A\}\|=10^\{5\}\. Among them,noptn\_\{\\mathrm\{opt\}\}actions are optimal with reward11and all remaining actions receive reward0\. The trajectory return is the sum of step rewards, and we report the normalized return by dividing it byHH\. The policy is a per\-step tabular softmaxθ∈ℝH×\|𝒜\|\\theta\\in\\mathbb\{R\}^\{H\\times\|\\mathcal\{A\}\|\}\. To mimic a pretrained non\-uniform prior, logits of optimal actions are initialized from𝒩​\(1,1\)\\mathcal\{N\}\(1,1\), while all other logits are initialized from𝒩​\(0,1\)\\mathcal\{N\}\(0,1\)\.

#### Hyper\-parameters\.

The policy is trained with SGD using learning rate0\.20\.2for80008000steps, with evaluation every100100steps\. We varyHH,noptn\_\{\\mathrm\{opt\}\}, and group sizeGGto study the effects of trace length, optimal\-action sparsity, and group\-level credit noise\. All curves are averaged over ten random seeds\.

## Appendix CAdditional Training Dynamics and Analysis

As illustrated in Figure[7](https://arxiv.org/html/2607.07976#A3.F7)and Figure[8](https://arxiv.org/html/2607.07976#A3.F8), the training dynamics on both Qwen3\-4B\-Base and Qwen2\.5\-Math\-7B exhibit patterns consistent with those observed on Qwen3\-1\.7B\-Base\. TACO achieves higher training accuracy while maintaining stable token entropy and producing longer responses during training\. These dynamics provide empirical evidence that TACO improves optimization efficiency and helps develop more robust reasoning capabilities\.

![Refer to caption](https://arxiv.org/html/2607.07976v1/x11.png)\(a\)Training reward
![Refer to caption](https://arxiv.org/html/2607.07976v1/x12.png)\(b\)Policy entropy
![Refer to caption](https://arxiv.org/html/2607.07976v1/x13.png)\(c\)Response length

Figure 7:Training dynamics ofTACOand baseline methods on Qwen3\-4B\-Base\.![Refer to caption](https://arxiv.org/html/2607.07976v1/x14.png)\(a\)Training reward
![Refer to caption](https://arxiv.org/html/2607.07976v1/x15.png)\(b\)Policy entropy
![Refer to caption](https://arxiv.org/html/2607.07976v1/x16.png)\(c\)Response length

Figure 8:Training dynamics ofTACOand baseline methods on Qwen2\.5\-Math\-7B\.![Refer to caption](https://arxiv.org/html/2607.07976v1/x17.png)\(a\)Reliable\-token ratio
![Refer to caption](https://arxiv.org/html/2607.07976v1/x18.png)\(b\)Thresholdτ\\tau\.
![Refer to caption](https://arxiv.org/html/2607.07976v1/x19.png)\(c\)Average weightww\.

Figure 9:Training diagnostics over positive\-advantage response tokens\.Figure[9](https://arxiv.org/html/2607.07976#A3.F9)shows diagnostics over positive\-advantage response tokens\. The thresholdτ\\taurepresents the mean probability boundary below which tokens are identified as risky\. The reliable\-token ratio averages0\.9810\.981with a median of0\.9980\.998, indicating that only about1\.9%1\.9\\%of positive\-advantage tokens are downweighted on average\. The average credit weightwwremains close to one, increasing from0\.9960\.996to0\.99980\.9998, whileτ\\tauincreases mildly from0\.00710\.0071to0\.00950\.0095\. These results show that the calibration modifies only a small subset of low\-confidence tokens, yet has a substantial effect on training dynamics and final performance\. This suggests that sparse tail tokens being mis\-amplified during training can exert a disproportionate influence on policy optimization, while our calibration effectively suppresses these harmful local updates\.

![Refer to caption](https://arxiv.org/html/2607.07976v1/x20.png)Figure 10:A case fromQwen3\-1\.7B\-Base\.![Refer to caption](https://arxiv.org/html/2607.07976v1/x21.png)Figure 11:A case fromQwen2\.5\-Math\-7B\.Figures[10](https://arxiv.org/html/2607.07976#A3.F10)and[11](https://arxiv.org/html/2607.07976#A3.F11)show two additional rewarded traces from different models\. The Qwen3\-1\.7B\-Base case contains tail tokens related to instruction or search leakage, such as off\-context assistant/search text and unrelated help messages\. The Qwen2\.5\-Math\-7B case contains malformed template fragments and serialized control artifacts, such asist,\_that,Pro, andabb\. In both cases,TACOassigns lower credit to these unreliable tokens while preserving credit for the later coherent solution steps\.

## Appendix DLimitations and Future Works

In this work, we proposeTACOto mitigate*Positive\-Credit Contamination*in GRPO\-style RLVR by calibrating token\-level credit for unreliable tail tokens\. While our experiments focus mainly on mathematical reasoning, a natural future direction is to extendTACOto other verifiable domains, such as code generation and tool use, as well as to open\-ended tasks such as creative writing, which may require richer reward signals and more careful calibration designs\. In these broader settings, the main idea ofTACOremains applicable: credit assignment should distinguish unreliable local continuations from useful behaviors\. Future work can also combineTACOwith methods that use model\-generated reasoning traces for self\-improvement, further enhancing the stability and effectiveness of post\-training\.

## Appendix EComputational Infrastructure

The computational infrastructure information is given in Table[3](https://arxiv.org/html/2607.07976#A5.T3)\.

Table 3:Experiment configuration and computing infrastructure\.NameValueData typetorch\.bfloat16Flash\-AttentionTrueComputing InfrastructureGPUGPU ModelNVIDIA\-H200GPU Memory141 GBGPU Number4CUDA Version12\.9CPU Memory512GB