Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning

# Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning

Source: https://arxiv.org/html/2506.08125

Hanbing Liu¹, Lang Cao²¹, Yuanyi Ren³¹, Mengyu Zhou⁴, Haoyu Dong⁵, Xiaojun Ma⁵, Shi Han⁵, Dongmei Zhang⁵

¹Tsinghua University
²University of Illinois Urbana-Champaign
³Peking University
⁴Qwen Large Model Application Team, Alibaba
⁵Microsoft

[email protected], [email protected], [email protected]

## Abstract

Large language models (LLMs) show strong reasoning abilities but often produce unnecessarily long explanations that reduce efficiency. Although reinforcement learning (RL) has been used to improve reasoning, most methods focus on accuracy and rely on uniform length-based rewards that overlook the differing contributions of individual tokens, often harming correctness. We revisit length optimization in RL through the perspective of token significance. Observing that many chain-of-thought (CoT) tokens contribute little to the final answer, we introduce a significance-aware length reward that selectively penalizes insignificant tokens, reducing redundancy while preserving essential reasoning. We also propose a dynamic length reward that encourages more detailed reasoning early in training and gradually shifts toward conciseness as learning progresses. Integrating these components into standard policy optimization yields a framework that improves both reasoning efficiency and accuracy. Experiments across multiple benchmarks demonstrate substantial reductions in response length while preserving or improving correctness, highlighting the importance of modeling token significance for efficient LLM reasoning. Code is available at https://github.com/microsoft/Bingo.

## 1 Introduction

Large language models (LLMs) have demonstrated impressive reasoning capabilities across a variety of tasks, from arithmetic problem solving to commonsense reasoning. A key observation from recent work is that sufficiently large models can exhibit emergent reasoning abilities, such as chain-of-thought (CoT) reasoning, without explicit supervision. Despite these successes, a major challenge persists: LLMs often generate unnecessarily verbose or redundant reasoning traces, leading to inefficiencies in computational cost, redundancy, and latency. Improving reasoning efficiency of LLMs has thus emerged as an important research direction.

Prior work in this area can be broadly categorized into supervised fine-tuning (SFT) approaches and reinforcement learning (RL) approaches. SFT-based methods focus on constructing compressed reasoning traces and training models to imitate them. While these approaches can be effective, they rely on high-quality compressed supervision, which is costly to obtain and often lacks generalizability across diverse tasks. RL-based methods typically introduce length-based rewards that penalize overly long responses to encourage brevity. However, the design of such rewards or penalties in RL-based methods remains underexplored and is often overly simplistic. For example, O1-Pruner applies a uniform penalty to all samples, assuming that every response should be shortened. This assumption often leads to performance degradation, as not all reasoning traces are equally verbose—some require more detailed steps to arrive at the correct answer. To address this, other works have proposed more selective penalty strategies, conditioning penalties on sample correctness or estimated difficulty. These approaches typically assign stronger penalties to simpler questions and weaker ones to more challenging cases. However, accurately estimating question difficulty remains a fundamental challenge, and unresolved hard questions often lead to unnecessarily long responses, further undermining reasoning efficiency.

Despite growing interest, current designs of length-based rewards remain limited, as they often fail to adequately promote concise reasoning while preserving answer accuracy. For example, prior work has largely overlooked the impact of token-level contributions on the overall efficiency of reasoning as illustrated in Figure 1(A).

In this work, we approach the problem from a novel perspective grounded in the concept of token significance. Our motivation arises from observed token redundancy in LLMs, where many tokens in chain-of-thought (CoT) reasoning contribute little to the final answer. We posit that not all tokens are equally important for efficient reasoning—many are insignificant, such as redundant phrases or unnecessary intermediate steps, and can be removed without degrading performance. Existing reward designs often overlook this distinction. In contrast, we introduce a significance-aware length reward that selectively penalizes only those insignificant tokens which do not meaningfully contribute to the final answer, while preserving essential reasoning steps. We also observe that effectively handling hard questions is essential for efficient reasoning. Prior work has shown that encouraging extended CoT reasoning can improve performance by enabling deeper exploration, which may help solve more difficult questions. Therefore, it is intuitive to use length as an incentive for hard questions. However, LLMs should solve difficult questions not only accurately but also concisely. Applying a static length incentive can lead to unnecessarily long responses, which may still fail to produce correct answers. To address this, we incorporate a dynamic length reward that adapts over the course of training. This reward is applied to significant tokens in incorrect samples to balance exploration and efficiency. Specifically, it encourages longer reasoning in the early training phase to promote exploration, and gradually shifts toward penalizing excessive length in later stages to promote conciseness.

Building on these insights, we introduce Bingo (Boosting Efficient ReasonING in Policy Optimization), an RL framework that incorporates our two proposed reward mechanisms into standard RL algorithms such as Proximal Policy Optimization (PPO) as illustrated in Figure 1(B). This enables joint optimization of both reasoning accuracy and efficiency. Extensive experiments across diverse reasoning benchmarks show that Bingo consistently outperforms strong baselines by reducing redundant computation while maintaining or improving accuracy.

In summary, this paper makes the following key contributions:

- **Token Significance Insight.** We introduce the concept of token significance in policy optimization, distinguishing between significant and insignificant tokens in reasoning traces. This insight motivates our significance-aware length reward, which explicitly penalizes uninformative tokens while preserving critical reasoning content, enabling more targeted and effective length control.

- **Dynamic Length Control.** We propose a dynamic length reward strategy that adjusts the reward signal over the course of training—encouraging longer reasoning in the early stages to foster exploration, and gradually promoting conciseness as the model converges.

- **Efficiency-Oriented RL Framework.** We develop Bingo, a new reinforcement learning framework that integrates both reward strategies. Extensive experiments across multiple reasoning benchmarks, along with comprehensive analyses, demonstrate its effectiveness.

## 2 Related Work

**Reinforcement Learning for Large Language Models.** Reinforcement Learning (RL) has emerged as a powerful paradigm for aligning large language models (LLMs) with human preferences. In Reinforcement Learning from Human Feedback (RLHF), the Proximal Policy Optimization (PPO) algorithm is employed alongside human preference data to train a reward model that steers the fine-tuning of LLMs. Building on PPO, subsequent works like GRPO and REINFORCE++ have proposed improved variants to address its limitations. Beyond alignment, RL has also shown promise in improving the reasoning capabilities of LLMs. Early studies demonstrated that reward-guided training can enhance multi-step reasoning performance. More recently, DeepSeek-R1 demonstrated that large-scale RL can substantially boost reasoning ability across a wide range of tasks, pointing to a promising direction for future work.

**Efficient Reasoning with Large Language Models.** Recent advances have empowered language models to perform strong reasoning via inference-time techniques such as chain-of-thought prompting and post-training. More recent work has shifted to optimizing both accuracy and efficiency. Some approaches improve efficiency at inference time, such as token-budget-aware reasoning, or prompting strategies like "reason-without-thinking" and chain-of-draft. Others apply post-training optimization via supervised fine-tuning (SFT), including TokenSkip, TwT, LightThinker, and C3oT. These SFT methods primarily construct high-quality compressed reasoning paths containing key information, and train the models on them. In parallel, RL-based approaches often improve efficiency by incorporating length controls or penalties into their reward functions. For instance, O1-Pruner uses offline length rewards comparing samples against mean lengths. Kimi k1.5 applies online penalties to correct samples only. Building on prior RL-based approaches, we advance length-based reward design to enable LLMs to balance reasoning accuracy with computational efficiency.