DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards
Summary
Introduces DelTA, a discriminative token credit assignment method for reinforcement learning from verifiable rewards (RLVR) that amplifies distinctive token-gradient directions and reduces noise from shared patterns, achieving significant improvements on mathematical and code generation benchmarks.
View Cached Full Text
Cached at: 05/22/26, 06:30 AM
Paper page - DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards
Source: https://huggingface.co/papers/2605.21467
Abstract
Reinforcement learning from verifiable rewards is enhanced through a discriminative token credit assignment method that improves reward-based training by amplifying distinctive token-gradient directions and reducing noise from shared patterns.
Reinforcement learning from verifiable rewards(RLVR) has emerged as a central technique for improving the reasoning capabilities of large language models. Despite its effectiveness, how response-level rewards translate into token-level probability changes remains poorly understood. We introduce a discriminator view of RLVR updates, showing that thepolicy-gradient updatedirection implicitly acts as a linear discriminator overtoken-gradient vectorsand thereby determines which token probabilities are increased or decreased during learning. Under standard sequence-level RLVR, this discriminator is constructed from positive- and negative-side centroids formed byadvantage-weighted averagingoftoken-gradient vectors. However, such centroid construction can be dominated by shared high-frequency patterns, such as formatting tokens, diluting sparse yet discriminative directions that better distinguish high-reward responses from low-reward ones. To address this limitation, we propose DelTA, adiscriminative token credit assignmentmethod that estimates token coefficients to amplify side-specific token-gradient directions and downweight shared or weakly discriminative ones. These coefficients reweight aself-normalized RLVR surrogate, making the effectiveside-wise centroidsmore contrastive and thereby reshaping the RLVR update direction. On seven mathematical benchmarks, DelTA outperforms the strongest same-scale baselines by 3.26 and 2.62 average points on Qwen3-8B-Base and Qwen3-14B-Base, respectively. Additional results on code generation, a different backbone, and out-of-domain evaluations further demonstrate the generalization ability of DelTA.
View arXiv pageView PDFGitHub2Add to collection
Get this paper in your agent:
hf papers read 2605\.21467
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.21467 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.21467 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.21467 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Luth-2: New State-of-the-Art French Small Language Models
Luth-2 releases two French small language models (0.8B and 2B) that achieve state-of-the-art results on French benchmarks for their size, with open-weights and data on Hugging Face.
Learning from Environmental Feedback: Credit Assignment across Multiple Timescales for Agentic Reinforcement Learning
This paper introduces EFCA, a multi-timescale credit assignment method for agentic reinforcement learning that uses short-term feedback and medium-term state-history signals from environment interaction to improve task success and quality on ALFWorld and WebShop.
Control-Diverse Reinforcement Fine-Tuning: Decoupling the Shared Control Bottleneck of RL Post-Training
The paper introduces CD-RFT, a method to decouple the shared control bottleneck in RL post-training by regularizing a novel control coefficient, improving multi-task capability on models like Qwen2.5-7B and Llama-3.2-3B.
Ground-Truth Neighborhood Regularization for Reinforcement Learning Post-Training of Time Series Foundation Models
This paper identifies 'suboptimal collapse' in RL post-training of time series foundation models and proposes Ground-Truth Neighborhood Regularization (GTN-R) to keep output distributions near the ground truth, improving forecasting performance.
V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control
This paper introduces V-Simba, a visual RL architecture that adds normalization layers and pointwise convolutions to improve sample efficiency and stability. It matches or outperforms state-of-the-art methods across DMC, Adroit, and Meta-World benchmarks while being more computationally efficient than DrQ-v2.