Demystifying Entropy-based Selection for Chain-of-Thought Compression in Large Reasoning Models

# Demystifying Entropy-based Selection for Chain-of-Thought Compression in Large Reasoning Models
Source: [https://arxiv.org/html/2607.28707](https://arxiv.org/html/2607.28707)
Sara Candussio1Daniel Scalena2,3 Luca Bortolussi1Elisabetta Fersini2Malvina Nissim3Gabriele Sarti4 1University of Trieste2University of Milano\-Bicocca 3CLCG, University of Groningen4Khoury College of Computer Sciences, Northeastern University sara\.candussio@phd\.units\.it

###### Abstract

Entropy\-based pruning has been proposed as an effective method for compressing Chain\-of\-Thought \(CoT\) reasoning with negligible accuracy loss\. We test the robustness of low\- and high\-entropy CoT step selection methods across various models and reasoning tasks, showing that entropy offers no advantage over random pruning in any evaluated setting\. Moving from sentences to tokens, we then show that retaining low\-entropy tokens seems effective only on mathematical benchmarks\. We find this is due to the inherently low\-entropy nature of numeric tokens, which also convey semantic content in such problems\. Finally, we demonstrate that patching a subset of a few CoT tokens with their original activations recovers near\-perfect full\-trace performance, providing causal evidence that task information is not concentrated in a small set of CoT tokens identifiable by heuristics, but rather distributed across the full reasoning chain\.111Code instructions and data at[https://anonymous\.4open\.science/r/Demystifying\-Entropy\-Selection/](https://anonymous.4open.science/r/Demystifying-Entropy-Selection/)\.

Demystifying Entropy\-based Selection for Chain\-of\-Thought Compression in Large Reasoning Models

Sara Candussio1Daniel Scalena2,3Luca Bortolussi1Elisabetta Fersini2Malvina Nissim3Gabriele Sarti41University of Trieste2University of Milano\-Bicocca3CLCG, University of Groningen4Khoury College of Computer Sciences, Northeastern Universitysara\.candussio@phd\.units\.it

## 1Introduction

Inference\-time scaling via Chain\-of\-Thought \(CoT\) reasoning has become the norm for improving the accuracy of large language models \(LLMs\) on complex multi\-step tasks\. However, these gains often come at the expense of unnecessarily long traces that quickly fill the models’ context windows\(Boppanaet al\.,[2026](https://arxiv.org/html/2607.28707#bib.bib21); Scalenaet al\.,[2026a](https://arxiv.org/html/2607.28707#bib.bib22); Qiaoet al\.,[2025](https://arxiv.org/html/2607.28707#bib.bib23)\), motivating the development of CoT compression methods reducing context size with limited performance loss\.

One such strategy involves treating token\- or sentence\-entropy as a proxy for relevance, pruning large fractions of a trace based on entropy value alone with little to no accuracy drop\. Opposite claims coexist in the literature, favoring either the pruning\(Wanget al\.,[2025](https://arxiv.org/html/2607.28707#bib.bib1); Liet al\.,[2026](https://arxiv.org/html/2607.28707#bib.bib2); Laaouach,[2025](https://arxiv.org/html/2607.28707#bib.bib11); Kimet al\.,[2026](https://arxiv.org/html/2607.28707#bib.bib12)\)or the retention\(Huanget al\.,[2026](https://arxiv.org/html/2607.28707#bib.bib4); Tonet al\.,[2025](https://arxiv.org/html/2607.28707#bib.bib7); Wanget al\.,[2026](https://arxiv.org/html/2607.28707#bib.bib8); Xionget al\.,[2026](https://arxiv.org/html/2607.28707#bib.bib9)\)of low\-entropy items\.

This disagreement suggests that the entropy signal might not be as precise or indicative as presented in the literature\. Given that previous works lack comparison to random baseline, one may wonder whether it is indeed entropy the reason for their reported success, or it is generic compression which is effective\. Additionally, since compression methods are mostly tested on mathematical benchmarks, we do not know whether the observed success deriving from entropy\-based compression strategies is a property of mathematical reasoning or generalises to non\-mathematical tasks, too\.

In this work, we test whether the claimed advantage of low\- and high\-entropy selection for CoT compression persists under both conditions, evaluating LLMs of various sizes from three model families \(gpt\-oss,Gemma\-4, andQwen3\) on mathematical, logical and commonsense reasoning tasks\. We show that while at the sentence level, entropy generally provides no advantage over random selection, at the token level, an apparent advantage of low\-entropy selection emerges only on mathematical benchmarks\. Our results demonstrate that this can be explained by the overlap between low\-entropy tokens and content\-rich number tokens, rather than by any special property of low\-entropy tokens\. Even for mathematical tasks, selecting number tokens is most effective only when paired with a direct patching of activations\. Taken together, our results suggest that the semantic content of a reasoning trace is not concentrated in entropy\-relevant locations\.

## 2Related work

A growing body of work treats token\- and sentence\-level entropy as a heuristic for identifying relevant content in a CoT trace, under the premise that low\-entropy tokens are predictable given the preceding context and therefore redundant, while high\-entropy tokens mark points of genuine deliberation and reasoning direction\.Wanget al\.\([2025](https://arxiv.org/html/2607.28707#bib.bib1)\)distinguish between the latter, dubbed as*forking tokens*, and the remaining vast majority of low\-entropy tokens that merely complete reasoning already initiated at such forks\.Liet al\.\([2026](https://arxiv.org/html/2607.28707#bib.bib2)\)operationalize the same intuition at sentences\-level, reporting that on 50 test samples removing up to 80% of low\-entropy steps \(equivalent to 45% of tokens\) leaves accuracy unchanged, while random or high\-entropy removal degrades it sharply\. They also argue that filtering at sentence\-level is preferable to filtering at token\-level, since removing individual low\-entropy tokens causes a sharp performance drop that they attribute to a loss of*syntactic coherence*\. On the other hand,Huanget al\.\([2026](https://arxiv.org/html/2607.28707#bib.bib4)\)show that discarding high\-entropy tokens can also improve accuracy; they exploit this finding and propose a reward mechanism to bias the model to avoid excessively high\-entropy tokens, leading to good compression without performance loss\.

On the role of these entropy\-related tokens inside the trace,Zhaoet al\.\([2026](https://arxiv.org/html/2607.28707#bib.bib6)\)propose the distinction between*structural tokens*, i\.e\. low\-entropy tokens exhibiting low semantic content and having a prevalent syntactic role, and*organic tokens*, i\.e\. the high\-entropy ones constituting instead the main skeleton of the reasoning\. This distinction is notable since low\-entropy tokens are elsewhere treated as bearing the semantic content of a sentenceWanget al\.\([2025](https://arxiv.org/html/2607.28707#bib.bib1)\); Huanget al\.\([2026](https://arxiv.org/html/2607.28707#bib.bib4)\)\.Tonet al\.\([2025](https://arxiv.org/html/2607.28707#bib.bib7)\)point out that a high\-entropy step may reflect either a genuine reasoning fork or noise arising from model under\-training on a given task, a failure mode that they term an*unidentifiable task*\. They also claim that low\-entropy steps may be anchor points for subsequent reasoning, such as the end of useful computation in the CoT\.Wanget al\.\([2026](https://arxiv.org/html/2607.28707#bib.bib8)\)push this observation further: when the model is unsure of the answer, entropy exhibits high variance across different rollouts, making it an unreliable signal of the trace’s convergence to the final output\. They also show that trace token entropies are not informative at all, preferring the entropy at the first answer token\.

Due to this instability of trace entropy as a pointwise signal,Xionget al\.\([2026](https://arxiv.org/html/2607.28707#bib.bib9)\)note that a downward entropy trend is typically associated with longer traces and show that a trend\-based reward signal outperforms a purely local, per\-token entropy compression criterion\.Chenet al\.\([2026](https://arxiv.org/html/2607.28707#bib.bib10)\)observe that, although others use per\-token entropy directly for threshold\-based early\-exit decoding\(Laaouach,[2025](https://arxiv.org/html/2607.28707#bib.bib11); Kimet al\.,[2026](https://arxiv.org/html/2607.28707#bib.bib12)\), this signal is too noisy to threshold in isolation\. As a consequence, they compute a moving average over a sliding window, treating these sustained local maxima as the critical points of a trace; reducing these points improves performance on easier problems but substantially degrades it on harder ones\.

These works offer no consensus on what entropy really captures in a reasoning trace: it is unclear whether high\-entropy tokens mark genuine forks or model uncertainty, and whether low\-entropy tokens carry syntax or semantics remains contested across studies\. This, combined with the near\-total focus on mathematical benchmarks\(Wanget al\.,[2025](https://arxiv.org/html/2607.28707#bib.bib1); Liet al\.,[2026](https://arxiv.org/html/2607.28707#bib.bib2); Huanget al\.,[2026](https://arxiv.org/html/2607.28707#bib.bib4); Tonet al\.,[2025](https://arxiv.org/html/2607.28707#bib.bib7); Xionget al\.,[2026](https://arxiv.org/html/2607.28707#bib.bib9)\)and the scarcity of random baseline comparisons \(present only inLiet al\.\([2026](https://arxiv.org/html/2607.28707#bib.bib2)\)\), motivates the robustness assessment provided by our work\.

## 3Method

#### Entropy definitions

Given a reasoning trace, we define the entropy of a token as the Shannon entropy of the model’s next\-token predictive distribution at that position, restricted to and renormalized over its top\-kklog\-probabilities returned at generation time\. The*token\-level entropy*isHi=−∑t=1kpi​\(t\)​log⁡pi​\(t\)H\_\{i\}=\-\\sum\_\{t=1\}^\{k\}p\_\{i\}\(t\)\\log p\_\{i\}\(t\), wherepi​\(t\)p\_\{i\}\(t\)is the normalized probability assigned to a vocabulary tokenttat sequence positionii\. We compute*sentence\-level entropy*as the mean token\-level entropy over the sentence, where sentence boundaries are identified via fixed heuristics \(Section[4](https://arxiv.org/html/2607.28707#S4)\)\.

#### Compressed CoT formatting

The thinking regionCCinside a reasoning trace is identified as the token span between the model’s beginning\- and end\-of\-thinking markers\. A selection criterionssranks the candidate units \(sentences or tokens\) withinCCand adds them in ranked order until a target token budgetb=max⁡\(1,min⁡\(r⋅\|C\|,\|C\|\)\)b=\\max\(1,\\min\(r\\cdot\|C\|,\|C\|\)\)is reached, wherer∈\(0,1\]r\\in\(0,1\]is the retention rate\. The selected items are recomposed in the order in which they appeared in the original trace, producing a compressed CoTC~rs\\tilde\{C\}^\{s\}\_\{r\}\.

The compressed trace is wrapped into a reconstructed inputP\+\[BOT\]\+C~rs\+\[EOT\]\+SP\+\\texttt\{\[BOT\]\}\+\\tilde\{C\}^\{s\}\_\{r\}\+\\texttt\{\[EOT\]\}\+SwherePPis the prompt,\[BOT\]and\[EOT\]are model\-dependent beginning\- and end\-of\-thinking markers,\+\+denotes sequence concatenation, andSSis a task\-specific suffix that elicits a direct answer\.222When left unspecified, we use the suffixTherefore, the answer is \\boxed\{\. Other variants are ablated in Appendix[A](https://arxiv.org/html/2607.28707#A1), producing near\-identical results\.The answer is then greedily decoded for a limited number of tokens, forcing the model to rely on a compressed reasoning trace only\.

#### Selection criteria

At sentence\-level, we define the following ranking criteria:*low\-entropy*and*high\-entropy*rank sentences by their mean token entropy, respectively retaining the lowest\- and the highest\-entropy sentences first;*numbers*ranks sentences by the fraction of tokens containing a digit,333This resembles the prompt\-basedOnlyNumbersbaseline ofXiaet al\.\([2025](https://arxiv.org/html/2607.28707#bib.bib5)\), applied here in a stricter setting\.as a proxy for essential content in mathematical tasks;*low\-entropy no numbers*ranks by mean entropy as before, excluding numeric tokens to disentangle their effect on entropy;444As a consequence, sentences with no non\-numeric tokens are excluded from the candidate pool\.*random*uniformly samples whole sentences from available candidates\.

At the token level, the same five criteria are recomputed at the token granularity, with two additional pattern\-based selectors:*newlines*and*end\-of\-sentence*markers, retaining only token categories that coincide with sentence boundaries\. This addition is motivated by the use of end\-of\-step token positions as probing locations for extracting information regarding the full step\(Bogdanet al\.,[2026](https://arxiv.org/html/2607.28707#bib.bib25)\)\.555Since numbers, newlines and end\-of\-sentence markers typically match few tokens per trace, we report the actual compression rate rather than the nominal one when their candidate pool is exhausted before the nominal token budget is reached\.

#### Compression protocols

We evaluate the compressed traces obtained through different selectorsssunder two protocols\. Following prior work presented in Section[2](https://arxiv.org/html/2607.28707#S2), the compressed contextC~s\\tilde\{C\}^\{s\}is passed to the model as a new prompt to obtain the resulting answer\. However, for the token\-level compression analysis, we also run a forward pass over the full, uncompressed trace, caching its per\-layer hidden states, and patching them into the corresponding retained positions when generating from the compressed sequence, yielding patched contextC^s\\hat\{C\}^\{s\}\. Given a fixed selectorss,C~s\\tilde\{C\}^\{s\}andC^s\\hat\{C\}^\{s\}have the same text, butC~s\\tilde\{C\}^\{s\}activations are recomputed from scratch, whileC^s\\hat\{C\}^\{s\}activations match those of the selected full\-CoT tokens across all layers\.

#### Baselines and evaluation

Each trace is bounded by the*full CoT*performance, a theoretical upper bound on achievable accuracy\. This is compared against the accuracy obtained underC~rs\\tilde\{C\}^\{s\}\_\{r\}andC^rs\\hat\{C\}^\{s\}\_\{r\}, for every selectorssand retention raterr\. We report the*relative performance retention*\(RPR\), i\.e\. the ratio of a certain setting’s accuracy to the*full CoT accuracy*as a general measure of compression effectiveness, using exact match with ground\-truth answer\.666Appendix[B](https://arxiv.org/html/2607.28707#A2)additionally reports results obtained with a semantic verifier, confirming that the trends discussed below are not an artifact of the matching criterion\.To summarize a selector’s behavior across the full RPR range, we additionally report the RPR*Area Under the Curve*\(AUC\) as a function of retention raterr\. We useΔAUC=AUCs−AUCrandom\\Delta\_\{\\text\{AUC\}\}=\\mathrm\{AUC\}\_\{s\}\-\\mathrm\{AUC\}\_\{\\text\{random\}\}to quantify a selectorss’s advantage relative to the random baseline\.

## 4Experimental setup

#### Models and datasets

We test the generality of entropy\-based compression across model families, parameter scales, and reasoning domains using six models spanning three families and multiple sizes:gpt\-oss\-20bandgpt\-oss\-120bOpenAIet al\.\([2025](https://arxiv.org/html/2607.28707#bib.bib13)\),gemma\-4\-E4B\-itandgemma\-4\-A26B\-itTeamet al\.\([2026](https://arxiv.org/html/2607.28707#bib.bib14)\), andQwen3\-4BandQwen3\-14BYanget al\.\([2025](https://arxiv.org/html/2607.28707#bib.bib15)\), all operating in reasoning mode\. We test these on mathematical reasoning \(AIME 2024, 2025, and 2026;Zhang and Math\-AI \([2024](https://arxiv.org/html/2607.28707#bib.bib29),[2025](https://arxiv.org/html/2607.28707#bib.bib30),[2026](https://arxiv.org/html/2607.28707#bib.bib31)\); and a subset of 100, 50, and 50 questions from MATH\-500;Lightmanet al\.\([2023](https://arxiv.org/html/2607.28707#bib.bib26)\)\), logical puzzles \(ZebraLogic;Linet al\.\([2025](https://arxiv.org/html/2607.28707#bib.bib27)\)\), and multiple\-choice science questions \(GPQA\-Diamond;Reinet al\.\([2024](https://arxiv.org/html/2607.28707#bib.bib28)\)\) to ensure our findings are widely applicable across various reasoning tasks\.

#### Trace sampling

For each prompt, 8 reasoning traces are generated with temperature0\.70\.7and top\-pp0\.90\.9, with a budget of 16384 max new tokens dictated by our compute availability\. Traces that do not reach the end\-of\-thinking marker\[EOT\]within this budget are discarded\. Trace compression is evaluated forr∈\{0\.01,0\.05,0\.1,0\.2,0\.3,0\.4,0\.5,0\.6,0\.7,0\.8,0\.9\}r\\in\\\{0\.01,0\.05,0\.1,0\.2,0\.3,0\.4,0\.5,0\.6,0\.7,0\.8,0\.9\\\}leaving up to6464tokens to complete the answer after the suffix\.

#### Entropy measures and sentence segmentation

Token\-level entropyHiH\_\{i\}is computed over the top\-k=20k=20log\-probabilities returned by the sampler at generation time\.777We usek=20k=20as a proxy of the full logit distribution due to vLLM constraints, followingScalenaet al\.\([2026b](https://arxiv.org/html/2607.28707#bib.bib24)\)\.Sentences are identified by splitting the thinking region at token\-level sentence\-ending marks: a token marks the end of a sentence if it contains a period followed by a whitespace or if it ends with a period and the following token begins with whitespace\.888Exclamation and question marks are consequently not treated as sentence boundaries\.

#### Activation extraction

We useNNsight\(Fiotto\-Kaufmanet al\.,[2025](https://arxiv.org/html/2607.28707#bib.bib16)\)to access and extract model internals and to intervene on them\. Residual stream activations are collected at each layer over the full, uncompressed trace and subsequently used to overwrite specific positions at all the layers during generation over the compressed CoTC^rs\\hat\{C\}^\{s\}\_\{r\}\. This approach requires no modification to model weights or architecture, and is compatible with all evaluated model families through a shared interface\.

## 5Results

We organize our findings around five questions:

#### Does entropy\-based sentence pruning outperform random selection?

Figure[1](https://arxiv.org/html/2607.28707#S5.F1)reports the result of*random*,*low\-entropy*and*high\-entropy*sentence\-level pruning as described in Section[3](https://arxiv.org/html/2607.28707#S3)ongpt\-oss\-20bon AIME25\. Across the entire compression range,*random*dominates both entropy\-based selection criteria: at moderate\-to\-high compressions \(≥0\.5\\geq 0\.5\),*random*retains7878\-90%90\\%of full CoT performance, while*low\-entropy*has already dropped to4040\-79%79\\%and*high\-entropy*to6060\-89%89\\%\. Entropy alone, in either direction, fails to isolate the content necessary for correct reasoning\.

![Refer to caption](https://arxiv.org/html/2607.28707v1/x1.png)Figure 1:*Random*dominates both entropy\-based selectors across the entire compression range\.RPRvs\. Compression Rate forgpt\-oss\-20bon AIME25 for various selection strategies\. Shaded bands are 95% bootstrap Confidence Intervals \(Appendix[C](https://arxiv.org/html/2607.28707#A3)\)\.Table 1:AUC of*random*andΔ\\DeltaAUC of*low*/*high*relative to it over the RPR ranger∈\(0,1\]r\\in\(0,1\], for a math\-reasoning dataset \(AIME25\) and a non\-mathematical one \(ZebraLogic\)\.*random*is bolded when it outperforms both entropy\-based selectors\.
#### Does this hold across model families, sizes and reasoning domains?

Table[1](https://arxiv.org/html/2607.28707#S5.T1)reports, for two representative datasets, the AUC of the*random*baseline and the delta of*low\-*and*high\-entropy*pruning relative to it, over the full RPR range\. Retaining*high\-entropy*sentences is a poor pruning strategy regardless of model: on AIME25,Δlow\\Delta\_\{\\text\{low\}\}is within noise of zero forQwen3\-14B,Qwen3\-4B,gemma\-4\-E4B\-itandgpt\-oss\-120b, but drops sharply forgpt\-oss\-20bandgemma\-4\-26B\-A4B\-it\. On ZebraLogic,Δlow\\Delta\_\{\\text\{low\}\}is negative for five of the six models, withgpt\-oss\-120bbeing the only exception \(full breakdown in Appendix[D\.1](https://arxiv.org/html/2607.28707#A4.SS1)\)\. These results suggest that the low\-entropy degradation depends on domain rather than model family, and that the entropy pruning assumption fails systematically: across different model families, scales, and domains, no selector significantly outperforms random sentence selection\.

#### Is entropy the right explanation at token\-level?

![Refer to caption](https://arxiv.org/html/2607.28707v1/x2.png)Figure 2:*Low\-entropy*token\-level compression outperforms random selection only on mathematical tasks when patching is performed, thanks to numerical tokens\.On MATH\-500,*low\-entropy no numbers*drops sharply below both*low\-entropy*and*random*, while*numbers*is the most performant selector even at extreme compressions\. The advantage of*low\-entropy*on mathematical reasoning is driven by the limited choice of numeric tokens\.Unlike sentence\-level pruning, token\-level compression for math reasoning tasks benefits from*low\-entropy*selection over*random*for most compression rates, while this gap vanishes on non\-mathematical tasks \(Figure[2](https://arxiv.org/html/2607.28707#S5.F2), top row\)\. We hypothesize that this asymmetry can be explained with the fact that numerical tokens exhibit significantly lower entropy in mathematical CoTs, in light of the limited vocabulary options for single\-token digits\.999I\.e\., when the context strongly implies a number, the model needs to choose only between 0\-9 rather than the full set of possible tokens\.To test whether this confounds the observed trend, we distinguish between*low\-entropy no numbers*and*numbers*settings: if numeric tokens are responsible for the gap, the latter should perform comparably to*low\-entropy*\.

On unpatched compressed tracesC~rs\\tilde\{C\}^\{s\}\_\{r\}, we find that*low\-entropy no numbers*and*numbers*both perform significantly worse than low\-entropy \(Figure[2](https://arxiv.org/html/2607.28707#S5.F2), top left; Table[6](https://arxiv.org/html/2607.28707#A5.T6)in Appendix\)\. We attribute this difference to the lack of context fluency produced by token\-level filtering, as also highlighted byLiet al\.\([2026](https://arxiv.org/html/2607.28707#bib.bib2)\), with the resulting context often containing bare digits and arithmetic operators, or syntactic connectives stripped of their numeric content\.101010A concrete example is shown in Appendix[D\.2](https://arxiv.org/html/2607.28707#A4.SS2)\.This behaviour is expected to disappear when token original activations are added to compressed forward pass: patched compressionC^rs\\hat\{C\}^\{s\}\_\{r\}results \(Figure[2](https://arxiv.org/html/2607.28707#S5.F2), bottom row\) indeed confirm that selecting and patching only numbers in the math domain \(*numbers*\) exceeds*low\-entropy*performance\. Selecting instead non\-numerical low\-entropy tokens \(*low\-entropy no numbers*\) leads to significant degradation, even worse than*random*\. On GPQA\-Diamond \(Figure[2](https://arxiv.org/html/2607.28707#S5.F2), bottom right\), all strategies instead perform on par with random selection\. When numerical tokens are not task\-relevant content,*numbers*and consequently*low\-entropy*selectors are comparable to*random*pruning\.

#### Does token activation patching performance recover generalize to other selectors?

Having shown that patching restores*numbers*’ advantage on mathematical tasks, we question whether this holds for other selectors across models and benchmarks\.

Figure[3](https://arxiv.org/html/2607.28707#S5.F3)reports the effect of activation patching on AIME25 for all six models:*low\-entropy*remains close to the full trace performance up to much higher compression rates than the unpatched setting, retaining≥0\.9\\geq 0\.9relative performance using2020\-30%30\\%of the original trace \(compared to5050\-70%70\\%without patching\)\.*Numbers*also retains most of the trace’s informativeness, almost recovering original performance using only1010\-20%20\\%of the tokens, while both*low\-entropy no numbers*and*high\-entropy*show no comparable improvement once patched\. Similarly, purely pattern\-based selectors \(*newlines*and*end of sentence*, that do not carry surface\-relevant content\) do not perform better under patching \(averageΔAUC\\Delta\_\{\\text\{AUC\}\}of−0\.74\-0\.74and−0\.73\-0\.73on AIME25 respectively, both worse than their unpatched versions; full results in Appendix[D\.3](https://arxiv.org/html/2607.28707#A4.SS3)\)\. Compression with patching works only when content\-rich tokens are retained: for mathematical tasks,*high\-entropy*tokens do not overlap with meaningful content, and*low\-entropy*ones owe their effectiveness to the numerical subset they include\.

![Refer to caption](https://arxiv.org/html/2607.28707v1/x3.png)Figure 3:Patching closes the gap for*low\-entropy*and*numbers*across all models on AIME25, while*high\-entropy*and*low\-entropy no numbers*remain largely unaffected or worse\.
#### Does discarding either high\- or low\-entropy elements improve accuracy?

Huanget al\.\([2026](https://arxiv.org/html/2607.28707#bib.bib4)\)observe that removing high\-entropy tokens improves original performance111111Paragraph 2\.3, Figure 3 ofHuanget al\.\([2026](https://arxiv.org/html/2607.28707#bib.bib4)\): averaged across GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.28707#bib.bib32)\), MATH\-500, AIME24 and AMC23\(Wang and others,[2025](https://arxiv.org/html/2607.28707#bib.bib33)\)and tested onQwen3\-4BandQwen3\-8B\., claiming that*their absence reduces noise in the reasoning process*\. We observe a superficially similar pattern \(see left panels of Figure[2](https://arxiv.org/html/2607.28707#S5.F2)and Appendix[E](https://arxiv.org/html/2607.28707#A5)\), but attribute it to the numeric\-token overlap discussed above \(Section[4](https://arxiv.org/html/2607.28707#S4)\), since activation\-patched*numbers*is linked to a larger increase in accuracy\.

On the opposite,Liet al\.\([2026](https://arxiv.org/html/2607.28707#bib.bib2)\)prune low\-entropy sentences121212Figure 1 left ofLiet al\.\([2026](https://arxiv.org/html/2607.28707#bib.bib2)\):DeepSeek\-R1\-7B\(DeepSeek\-AI,[2025](https://arxiv.org/html/2607.28707#bib.bib34)\)on 50 samples from DeepScaleR\(Nalaieet al\.,[2022](https://arxiv.org/html/2607.28707#bib.bib35)\);Qwen3\-8Bis also used throughout the experiments\.and conclude that*a vast majority of low\-entropy steps are indeed redundant*and thus can be discarded without accuracy degradation, while this is not the case for high\-entropy and random sentence selection\. Our results do also confirm that pruning strategies can lead to improved performances \(Appendix[E](https://arxiv.org/html/2607.28707#A5)\), but do also show that random selection achieves the same at comparable or even higher compression rates \(Figure[1](https://arxiv.org/html/2607.28707#S5.F1), Appendix[D\.1](https://arxiv.org/html/2607.28707#A4.SS1)\)\.

The claim that entropy values \(in both directions, either high\- or low\-\) are by their own a signal for reasoning content relevance is not supported by our experiments\.

## 6Conclusions

In this work, we revisit the widespread assumption that token\- or sentence\-level entropy are associated to the semantic content of a reasoning trace and therefore can be used as a signal to prune a trace without accuracy drops\. We test entropy\-based selection against a random baseline across six models \(spanning three families and two parameter scales\) on both mathematical and non\-mathematical reasoning tasks\.

At sentence\-level, the assumption fails consistently:*random*pruning dominates both*low\-*and*high\-entropy*selection across every model, scale and domain we tested, with no strategy reliably outperforming chance\. At token\-level, an apparent advantage for*low\-entropy*selection emerges on mathematical benchmarks only; we show this is not an entropy effect at all, but instead a spurious consequence of the overlap between low\-entropy and numeric tokens, which alone account for the same gain\.

Selecting surface\-text is however insufficient to effectively compress the trace, especially when the criterion produces an unreadable trace \(e\.g\. consisting in bare digits or syntactic connectives only\): the fresh forward pass performed by the model on this kind of context is uninformative of the reasoning process\. When paired with activations,*numbers*and consequently*low\-entropy*selectors recover near full\-trace performance on mathematical benchmarks at extreme compression rates \(using only1010\-20%20\\%of the original tokens\), while the others show no comparable recovery\. On non\-mathematical datasets, no selector performs significantly better than random\.

These results suggest that traces can be compressed without performance loss even with few patched tokens if they correspond to meaningful semantic content, such as numbers in mathematical tasks\. The fact that entropy\-based selectors do not outperform random pruning \(when purified by spurious signals such as numeric tokens on AIME and MATH\-500\) is an evidence that semantic content of a reasoning trace is not a property that entropy elements carry\.

## Limitations

Our study is restricted to six open\-weight models across three families; while this spans a discrete range of scales and training recipes, we cannot rule out that some closed, bigger \(we tested up to 120B, withgpt\-oss\-120b\), or simply belonging to another model family with different training strategies \(e\.g\. from the DeepSeek series\) would show a different relationship between entropy and reasoning relevant content\. Our benchmarks, while spanning both mathematical \(AIME, MATH\-500\) and non\-mathematical \(GPQA\-Diamond, ZebraLogic\) domains, remain focused on reasoning tasks\. We also limit in number and in the size of the non\-AIME subsets \(50\-100 questions each\) and the trace length \(16384 max new tokens\) which may understate the variance of some reported gaps, a choice made for efficient data collection\.

The activation patching intervention requires access to model internals and full\-precision forward passes over the uncompressed trace, which is not a deployable compression strategy on its own but rather a diagnostic tool used here to test whether adding contextual information to unreadable traces can help recover the full trace performance\.

Finally, our selection criteria operate at fixed retention rates that can only be determined post\-hoc and that are drastically different according to the original trace length\.

## References

- \[1\]Scaling test\-time compute with open models\.External Links:[Link](https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling-test-time-compute)Cited by:[Appendix B](https://arxiv.org/html/2607.28707#A2.p1.1)\.
- P\. C\. Bogdan, U\. Macar, N\. Nanda, and A\. Conmy \(2026\)Thought anchors: which LLM reasoning steps matter?\.External Links:[Link](https://openreview.net/forum?id=6NUtPO9PdV)Cited by:[§3](https://arxiv.org/html/2607.28707#S3.SS0.SSS0.Px3.p2.1)\.
- S\. Boppana, A\. Ma, M\. Loeffler, R\. Sarfati, E\. Bigelow, A\. Geiger, O\. Lewis, and J\. Merullo \(2026\)Reasoning theater: disentangling model beliefs from chain\-of\-thought\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=1lyagkzogH)Cited by:[§1](https://arxiv.org/html/2607.28707#S1.p1.1)\.
- S\. Chen, H\. Guo, Y\. Ye, S\. Huang, W\. Hu, J\. Chen, M\. Zhang, H\. Li, S\. Guo, and N\. Peng \(2026\)ARES: multimodal adaptive reasoning via difficulty\-aware token\-level entropy shaping\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=2g945Ngc7l)Cited by:[Appendix B](https://arxiv.org/html/2607.28707#A2.p1.1),[§2](https://arxiv.org/html/2607.28707#S2.p3.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[footnote 11](https://arxiv.org/html/2607.28707#footnote11)\.
- DeepSeek\-AI \(2025\)DeepSeek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[footnote 12](https://arxiv.org/html/2607.28707#footnote12)\.
- J\. Fiotto\-Kaufman, A\. R\. Loftus, E\. Todd, J\. Brinkmann, K\. Pal, D\. Troitskii, M\. Ripa, A\. Belfki, C\. Rager, C\. Juang, A\. Mueller, S\. Marks, A\. S\. Sharma, F\. Lucchetti, N\. Prakash, C\. Brodley, A\. Guha, J\. Bell, B\. C\. Wallace, and D\. Bau \(2025\)NNsight and ndif: democratizing access to open\-weight foundation model internals\.External Links:2407\.14561,[Link](https://arxiv.org/abs/2407.14561)Cited by:[§4](https://arxiv.org/html/2607.28707#S4.SS0.SSS0.Px4.p1.1)\.
- C\. Huang, W\. Lu, and W\. Zhang \(2026\)PEAR: phase entropy aware reward for efficient reasoning\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=HLc2igXEA3)Cited by:[Appendix B](https://arxiv.org/html/2607.28707#A2.p1.1),[§1](https://arxiv.org/html/2607.28707#S1.p2.1),[§2](https://arxiv.org/html/2607.28707#S2.p1.1),[§2](https://arxiv.org/html/2607.28707#S2.p2.1),[§2](https://arxiv.org/html/2607.28707#S2.p4.1),[§5](https://arxiv.org/html/2607.28707#S5.SS0.SSS0.Px5.p1.1),[footnote 11](https://arxiv.org/html/2607.28707#footnote11)\.
- J\. Kim, S\. Lee, and T\. Kim \(2026\)Think just enough: leveraging self\-assessed confidence for adaptive reasoning in language models\.InFindings of the Association for Computational Linguistics: EACL 2026,V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 5000–5006\.External Links:[Link](https://aclanthology.org/2026.findings-eacl.263/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-eacl.263),ISBN 979\-8\-89176\-386\-9Cited by:[§1](https://arxiv.org/html/2607.28707#S1.p2.1),[§2](https://arxiv.org/html/2607.28707#S2.p3.1)\.
- H\. Kydlíček \(2024\)Math\-verify: math verification library\.Note:[https://github\.com/huggingface/math\-verify](https://github.com/huggingface/math-verify)Version 0\.6\.1Cited by:[Appendix B](https://arxiv.org/html/2607.28707#A2.p2.1)\.
- Y\. Laaouach \(2025\)HALT\-cot: model\-agnostic early stopping for chain\-of\-thought reasoning via answer entropy\.In4th Muslims in ML Workshop co\-located with ICML 2025,External Links:[Link](https://openreview.net/forum?id=CX5c7C1CZa)Cited by:[§1](https://arxiv.org/html/2607.28707#S1.p2.1),[§2](https://arxiv.org/html/2607.28707#S2.p3.1)\.
- Z\. Li, J\. Zhong, Z\. Zheng, X\. Wen, Z\. Xu, Y\. Cheng, F\. Zhang, and Q\. Xu \(2026\)Making slow thinking faster: compressing LLM chain\-of\-thought via step entropy\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=cGLqQfS5wH)Cited by:[§1](https://arxiv.org/html/2607.28707#S1.p2.1),[§2](https://arxiv.org/html/2607.28707#S2.p1.1),[§2](https://arxiv.org/html/2607.28707#S2.p4.1),[§5](https://arxiv.org/html/2607.28707#S5.SS0.SSS0.Px3.p2.2),[§5](https://arxiv.org/html/2607.28707#S5.SS0.SSS0.Px5.p2.1),[footnote 12](https://arxiv.org/html/2607.28707#footnote12)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2023\)Let’s verify step by step\.External Links:2305\.20050Cited by:[§4](https://arxiv.org/html/2607.28707#S4.SS0.SSS0.Px1.p1.1)\.
- B\. Y\. Lin, R\. L\. Bras, K\. Richardson, A\. Sabharwal, R\. Poovendran, P\. Clark, and Y\. Choi \(2025\)ZebraLogic: on the scaling limits of LLMs for logical reasoning\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=sTAJ9QyA6l)Cited by:[§4](https://arxiv.org/html/2607.28707#S4.SS0.SSS0.Px1.p1.1)\.
- K\. Nalaie, R\. Xu, and R\. Zheng \(2022\)DeepScale: online frame size adaptation for multi\-object tracking on smart cameras and edge servers\.In2022 IEEE/ACM Seventh International Conference on Internet\-of\-Things Design and Implementation \(IoTDI\),pp\. 67–79\.External Links:[Document](https://dx.doi.org/10.1109/IoTDI54339.2022.00010)Cited by:[footnote 12](https://arxiv.org/html/2607.28707#footnote12)\.
- OpenAI, :, S\. Agarwal, L\. Ahmad, J\. Ai, S\. Altman, A\. Applebaum, E\. Arbus, R\. K\. Arora, Y\. Bai, B\. Baker, H\. Bao, B\. Barak, A\. Bennett, T\. Bertao, N\. Brett, E\. Brevdo, G\. Brockman, S\. Bubeck, C\. Chang, K\. Chen, M\. Chen, E\. Cheung, A\. Clark, D\. Cook, M\. Dukhan, C\. Dvorak, K\. Fives, V\. Fomenko, T\. Garipov, K\. Georgiev, M\. Glaese, T\. Gogineni, A\. Goucher, L\. Gross, K\. G\. Guzman, J\. Hallman, J\. Hehir, J\. Heidecke, A\. Helyar, H\. Hu, R\. Huet, J\. Huh, S\. Jain, Z\. Johnson, C\. Koch, I\. Kofman, D\. Kundel, J\. Kwon, V\. Kyrylov, E\. Y\. Le, G\. Leclerc, J\. P\. Lennon, S\. Lessans, M\. Lezcano\-Casado, Y\. Li, Z\. Li, J\. Lin, J\. Liss, Lily, Liu, J\. Liu, K\. Lu, C\. Lu, Z\. Martinovic, L\. McCallum, J\. McGrath, S\. McKinney, A\. McLaughlin, S\. Mei, S\. Mostovoy, T\. Mu, G\. Myles, A\. Neitz, A\. Nichol, J\. Pachocki, A\. Paino, D\. Palmie, A\. Pantuliano, G\. Parascandolo, J\. Park, L\. Pathak, C\. Paz, L\. Peran, D\. Pimenov, M\. Pokrass, E\. Proehl, H\. Qiu, G\. Raila, F\. Raso, H\. Ren, K\. Richardson, D\. Robinson, B\. Rotsted, H\. Salman, S\. Sanjeev, M\. Schwarzer, D\. Sculley, H\. Sikchi, K\. Simon, K\. Singhal, Y\. Song, D\. Stuckey, Z\. Sun, P\. Tillet, S\. Toizer, F\. Tsimpourlas, N\. Vyas, E\. Wallace, X\. Wang, M\. Wang, O\. Watkins, K\. Weil, A\. Wendling, K\. Whinnery, C\. Whitney, H\. Wong, L\. Yang, Y\. Yang, M\. Yasunaga, K\. Ying, W\. Zaremba, W\. Zhan, C\. Zhang, B\. Zhang, E\. Zhang, and S\. Zhao \(2025\)Gpt\-oss\-120b and gpt\-oss\-20b model card\.External Links:2508\.10925,[Link](https://arxiv.org/abs/2508.10925)Cited by:[§4](https://arxiv.org/html/2607.28707#S4.SS0.SSS0.Px1.p1.1)\.
- Z\. Qiao, Y\. Deng, J\. Zeng, D\. Wang, L\. Wei, F\. Meng, J\. Zhou, J\. Ren, and Y\. Zhang \(2025\)ConCISE: confidence\-guided compression in step\-by\-step efficient reasoning\.ArXivabs/2505\.04881\.External Links:[Link](https://api.semanticscholar.org/CorpusID:278394177)Cited by:[§1](https://arxiv.org/html/2607.28707#S1.p1.1)\.
- D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman \(2024\)GPQA: a graduate\-level google\-proof q&a benchmark\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=Ti67584b98)Cited by:[§4](https://arxiv.org/html/2607.28707#S4.SS0.SSS0.Px1.p1.1)\.
- D\. Scalena, S\. Candussio, L\. Bortolussi, E\. Fersini, M\. Nissim, and G\. Sarti \(2026a\)Beyond the commitment boundary: probing epiphenomenal chain\-of\-thought in large reasoning models\.External Links:2606\.13603,[Link](https://arxiv.org/abs/2606.13603)Cited by:[§1](https://arxiv.org/html/2607.28707#S1.p1.1)\.
- D\. Scalena, L\. Zotos, E\. Fersini, M\. Nissim, and A\. Üstün \(2026b\)EAGer: entropy\-aware GEneration for adaptive inference\-time scaling\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=G0Mw5MaPcR)Cited by:[footnote 7](https://arxiv.org/html/2607.28707#footnote7)\.
- C\. Snell, J\. Lee, K\. Xu, and A\. Kumar \(2024\)Scaling llm test\-time compute optimally can be more effective than scaling model parameters\.External Links:2408\.03314,[Link](https://arxiv.org/abs/2408.03314)Cited by:[Appendix B](https://arxiv.org/html/2607.28707#A2.p1.1)\.
- G\. Team, S\. E\. Abd, V\. Aggarwal, R\. Algayres, A\. Andreev, O\. Bachem, I\. Ballantyne, C\. Brick, V\. Cărbune, M\. Casbon, M\. Chaturvedi, V\. Cotruta, A\. Coucke, P\. Culliton, R\. Dadashi, L\. Dixon, M\. Elhawaty, U\. Evci, C\. Farabet, J\. Ferret, F\. Galgani, S\. Girgin, J\. Grill, M\. Grootendorst, J\. Guo, C\. Hardin, Y\. He, S\. M\. Hernandez, O\. Homburger, L\. Hussenot, J\. Ji, A\. Joulin, A\. Kamath, P\. Kassraie, O\. Lacombe, P\. Lahoti, G\. Liu, G\. Martins, L\. Martins, T\. Matejovicova, R\. Merhej, N\. Momchev, S\. Mondal, R\. Mullins, S\. R\. Panyam, S\. Pathak, S\. Perrin, A\. S\. Pinto, E\. Pot, A\. Pouget, A\. Ramé, S\. Ramos, D\. Reid, D\. Rim, M\. Rivière, K\. Roth, L\. Rouillard, O\. Sanseviero, P\. G\. Sessa, S\. Settle, D\. Sinopalnikov, S\. Smoot, P\. Stanczyk, A\. Steiner, L\. Stewart, I\. Tolstikhin, M\. Tschannen, A\. Tsitsulin, N\. Vieillard, R\. Wu, P\. Xu, H\. Yang, E\. Yvinec, L\. Zhang, J\. Zou, N\. Aagnes, A\. Abdelhamed, S\. Agrawal, S\. Agrawal, I\. Alabdulmohsin, J\. B\. Alayrac, U\. Alon, C\. Amarnath, A\. Anand, C\. Anastasiou, S\. Ariafar, F\. Aubet, K\. Axiotis, F\. Barbero, J\. Barral, A\. Bendebury, U\. Bergmann, S\. Bileschi, K\. Black, M\. Blondel, S\. Borgeaud, A\. Bražinskas, R\. Burnell, R\. Busa\-Fekete, M\. Cai, G\. Cameron, C\. Caucheteux, G\. Chadha, J\. Chan, A\. Chawla, B\. J\. Chen, J\. Chen, L\. Chen, X\. Chen, D\. Cheng, T\. Chien, N\. Chinaev, Y\. Chou, Z\. Chu, B\. Coleman, P\. Consul, S\. Conway\-Rahman, S\. Crowell, D\. Cutler, V\. Dani, S\. Daruki, A\. Das, D\. Deutsch, N\. Dikkala, L\. Ding, Q\. Ding, S\. Dodhia, K\. Donhauser, T\. Doshi, A\. Dragan, A\. Druinsky, S\. Dua, Z\. Egyed, D\. Eisenbud, D\. Eppens, C\. Fan, B\. Fatemi, Y\. Fathullah, V\. Feinberg, M\. Ferev, T\. Fujimoto, I\. Galatzer\-Levy, J\. Gante, S\. Geisler, S\. Ghosal, A\. M\. Girgis, A\. Go, A\. Gokhale, A\. Grills, Y\. Gu, P\. Gupta, G\. Guruganesh, R\. Hadsell, H\. Harkous, J\. Harlalka, D\. Hassabis, A\. Hauth, J\. Heyward, A\. Hosseini, C\. Hsia, I\. Hsu, X\. Huang, Y\. Huang, K\. Hui, A\. Hutter, T\. I, F\. Iliopoulos, A\. Jain, G\. Jawahar, Z\. Ji, Q\. Jin, M\. Johnson, K\. Joshi, A\. Kandoor, W\. Kang, K\. Kavukcuoglu, M\. Kazemi, K\. Kenealy, A\. Khalifa, P\. Kirk, S\. Kothawade, V\. Kovalev, N\. Kovelamudi, A\. Kraft, R\. Kumar, H\. Kuppam, J\. Lannin, C\. Lee, S\. Lee, D\. Lepikhin, D\. Li, Q\. Li, V\. Liévin, E\. Lin, Z\. Lin, C\. Liu, T\. Liu, T\. Liu, X\. Liu, M\. Lunayach, M\. Ma, G\. Madan, A\. Maksai, E\. Malmi, M\. Matuszak, D\. McDuff, G\. Menghani, D\. Mirylenka, K\. Misiunas, V\. Misra, A\. Mitran, K\. Mohamed, M\. Mukha, E\. Noland, J\. O’Donnell, K\. Olszewska, B\. Orlando, W\. Pan, R\. Panigrahy, U\. Parekh, C\. Park, E\. Paskie, L\. Peng, B\. Petrini, S\. Petrov, J\. Pfeiffer, B\. Piot, M\. Plomecka, S\. Poder, O\. Ponce, A\. Pramanik, D\. Racz, A\. Rajan, M\. Ramanovich, A\. Rao, M\. Ritter, V\. Rodrigues, E\. Rosen, M\. Rybiński, N\. Sachdeva, M\. E\. Sander, R\. Sathyanarayana, S\. Savla, S\. Schmidgall, T\. Schuster, B\. Seguin, A\. Sellergren, A\. Severyn, I\. Shafran, D\. Shah, Y\. Shangguan, A\. Shenoy, P\. Shenoy, R\. Shivanna, P\. Sho, L\. Spangher, W\. Stokowiec, T\. Strother, Y\. Su, Y\. Sun, M\. Sundararajan, A\. Tacchetti, M\. H\. Taege, P\. Tafti, C\. Tekur, R\. Thapa, M\. Traverse, L\. Treven, T\. Tu, C\. T\. Tung, P\. Veličković, M\. P\. Venkat, S\. G\. Venkatesh, V\. Venkiteswaran, F\. Visin, A\. Vitvitskyi, K\. Vodrahalli, W\. Wang, X\. Wang, T\. Warkentin, J\. Wassenberg, J\. Wieting, L\. Xiao, H\. Xu, Y\. Xu, F\. Xue, A\. Yadav, J\. Yan, A\. Yang, L\. Yang, M\. Yang, Z\. Ying, J\. H\. Yoo, S\. Zafar, F\. Zhang, J\. Zhang, J\. Zhang, X\. Zhang, C\. Zhao, D\. Zhou, and C\. Zou \(2026\)Gemma 4 technical report\.External Links:2607\.02770,[Link](https://arxiv.org/abs/2607.02770)Cited by:[§4](https://arxiv.org/html/2607.28707#S4.SS0.SSS0.Px1.p1.1)\.
- J\. Ton, M\. F\. Taufiq, and Y\. Liu \(2025\)Understanding chain\-of\-thought in LLMs through information theory\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=IjOWms0hrf)Cited by:[§1](https://arxiv.org/html/2607.28707#S1.p2.1),[§2](https://arxiv.org/html/2607.28707#S2.p2.1),[§2](https://arxiv.org/html/2607.28707#S2.p4.1)\.
- S\. Wang, L\. Yu, C\. Gao, C\. Zheng, S\. Liu, R\. Lu, K\. Dang, X\. Chen, J\. Yang, Z\. Zhang, Y\. Liu, A\. Yang, A\. Zhao, Y\. Yue, S\. Song, B\. Yu, G\. Huang, and J\. Lin \(2025\)Beyond the 80/20 rule: high\-entropy minority tokens drive effective reinforcement learning for llm reasoning\.External Links:2506\.01939,[Link](https://arxiv.org/abs/2506.01939)Cited by:[Appendix B](https://arxiv.org/html/2607.28707#A2.p1.1),[§1](https://arxiv.org/html/2607.28707#S1.p2.1),[§2](https://arxiv.org/html/2607.28707#S2.p1.1),[§2](https://arxiv.org/html/2607.28707#S2.p2.1),[§2](https://arxiv.org/html/2607.28707#S2.p4.1)\.
- X\. Wang, J\. McInerney, L\. Wang, and N\. Kallus \(2026\)EAT: entropy after $\\textlangle \\tt /think \\textrangle$ for reasoning model early exiting\.External Links:[Link](https://openreview.net/forum?id=hfEVqiJyF6)Cited by:[Appendix B](https://arxiv.org/html/2607.28707#A2.p1.1),[§1](https://arxiv.org/html/2607.28707#S1.p2.1),[§2](https://arxiv.org/html/2607.28707#S2.p2.1)\.
- Z\. Wanget al\.\(2025\)Step guided reasoning: improving mathematical reasoning using guidance generation and step reasoning\.InProceedings of the International Conference on Learning Representations \(ICLR\),Cited by:[footnote 11](https://arxiv.org/html/2607.28707#footnote11)\.
- H\. Xia, Y\. Li, C\. T\. Leong, W\. Wang, and W\. Li \(2025\)TokenSkip: controllable chain\-of\-thought compression in llms\.ArXivabs/2502\.12067\.External Links:[Link](https://api.semanticscholar.org/CorpusID:276421423)Cited by:[footnote 3](https://arxiv.org/html/2607.28707#footnote3)\.
- X\. Xiong, H\. Liu, L\. Gu, Z\. Chi, Y\. Qiu, Y\. Yu, and Y\. Wang \(2026\)ETR: entropy trend reward for efficient chain\-of\-thought reasoning\.External Links:2604\.05355,[Link](https://arxiv.org/abs/2604.05355)Cited by:[§1](https://arxiv.org/html/2607.28707#S1.p2.1),[§2](https://arxiv.org/html/2607.28707#S2.p3.1),[§2](https://arxiv.org/html/2607.28707#S2.p4.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4](https://arxiv.org/html/2607.28707#S4.SS0.SSS0.Px1.p1.1)\.
- A\. Yang, B\. Zhang, B\. Hui, B\. Gao, B\. Yu, C\. Li, D\. Liu, J\. Tu, J\. Zhou, J\. Lin, K\. Lu, M\. Xue, R\. Lin, T\. Liu, X\. Ren, and Z\. Zhang \(2024\)Qwen2\.5\-math technical report: toward mathematical expert model via self\-improvement\.External Links:2409\.12122,[Link](https://arxiv.org/abs/2409.12122)Cited by:[Appendix B](https://arxiv.org/html/2607.28707#A2.p1.1)\.
- Y\. Zhang and T\. Math\-AI \(2024\)American invitational mathematics examination \(AIME\) 2024\.External Links:[Link](https://huggingface.co/datasets/math-ai/aime24)Cited by:[§4](https://arxiv.org/html/2607.28707#S4.SS0.SSS0.Px1.p1.1)\.
- Y\. Zhang and T\. Math\-AI \(2025\)American invitational mathematics examination \(AIME\) 2025\.External Links:[Link](https://huggingface.co/datasets/math-ai/aime25)Cited by:[§4](https://arxiv.org/html/2607.28707#S4.SS0.SSS0.Px1.p1.1)\.
- Y\. Zhang and T\. Math\-AI \(2026\)American invitational mathematics examination \(AIME\) 2026\.External Links:[Link](https://huggingface.co/datasets/math-ai/aime26)Cited by:[§4](https://arxiv.org/html/2607.28707#S4.SS0.SSS0.Px1.p1.1)\.
- Z\. Zhao, S\. Land, D\. M\. Bikel, and W\. Alshikh \(2026\)Shorthand for thought: compressing llm reasoning via entropy\-guided supertokens\.External Links:2604\.26355,[Link](https://arxiv.org/abs/2604.26355)Cited by:[§2](https://arxiv.org/html/2607.28707#S2.p2.1)\.

## Appendix ASuffix ablation

To rule out that our choice of answer\-eliciting suffix \(Therefore the answer is \\boxed \{, footnote of Section[3](https://arxiv.org/html/2607.28707#S3)\) does not itself bias the reported accuracies, we run a small ablation on a subset ofN=25N=25questions \(we limit this test to one trace per question\) drawn from one mathematical benchmark \(AIME25\) and one non\-mathematical one \(ZebraLogic\) across all six models, comparing the default suffix againstK=3K=3alternative phrasings:\\boxed\{,Based only on the above, the best answer I can determine is \\boxed\{, andGiven the reasoning above, in one sentence, the answer is \\boxed\{\. We report accuracy under normal greedy decoding of the full trace, since the goal here is to rule out a suffix\-induced confound on the upper bound itself rather than to re\-run the full compression sweep for every suffix\.

Table[2](https://arxiv.org/html/2607.28707#A1.T2)shows the results: on ZebraLogic, all four suffixes yield identical mean accuracy across the six models \(93\.3%\); on AIME25, the four suffixes vary by at most 1\.4 percentage points \(67\.3\-68\.7%\), well within the cross\-model standard deviation \(13\.1 points\)\. This confirms that the suffix acts purely as a format cue for answer extraction rather than as a source of information that could bias the reported full\-CoT accuracies\.

Table 2:Full\-CoT accuracy by suffix using greedy decoding, averaged across the six evaluated models \(N=25N=25questions per model, one trace per question\)\. Differences across suffixes are negligible on both benchmarks\.
## Appendix BAnswer correctness assessment via a semantic verifier

Throughout the main text, correctness is assessed via exact match between the extracted answer and the ground truth\. Related works in this space adopt a range of different \(and often incompatible\) criteria for this same judgement, complicating any direct comparison of reported accuracies\.Wanget al\.\([2025](https://arxiv.org/html/2607.28707#bib.bib1)\)inherit the reward verifier of the underlying RL codebase \(verl/DAPO\), which applies symbolic normalization rather than raw string matching;Huanget al\.\([2026](https://arxiv.org/html/2607.28707#bib.bib4)\)adopt the extraction and verification pipeline ofYanget al\.\([2024](https://arxiv.org/html/2607.28707#bib.bib17)\); the entropy\-after\-</think\>early\-exiting method ofWanget al\.\([2026](https://arxiv.org/html/2607.28707#bib.bib8)\)relies on canonical\-form answer comparison viasal\.math, the math\-evaluation utilities of Hugging Face’s*search\-and\-learn*library\([Beechinget al\.,](https://arxiv.org/html/2607.28707#bib.bib19); Snellet al\.,[2024](https://arxiv.org/html/2607.28707#bib.bib20)\), itself adapted from the Qwen2\.5\-Math evaluation parser;Chenet al\.\([2026](https://arxiv.org/html/2607.28707#bib.bib10)\)employ a two\-stage criterion, i\.e\. normalized string matching on the extracted boxed answer followed by an LLM\-as\-judge fallback when no boxed answer is found or the normalized strings disagree\.

Given this variability \(and since we cover most of the models and datasets used across this body of work\), we recompute all reported curves usingmath\-verify\(a shared semantic verifier\(Kydlíček,[2024](https://arxiv.org/html/2607.28707#bib.bib18)\)\) in addition to the exact\-match criterion used in the main text, to check whether our conclusions are sensitive to the choice of matching criterion\. Figures[8](https://arxiv.org/html/2607.28707#A5.F8)and[9](https://arxiv.org/html/2607.28707#A5.F9)report the sentence\- and token\-level full grids recomputed under this semantic criterion\.

## Appendix CBootstrapped Confidence Intervals

Every accuracy we report in this work \(a single point on a compression curve or an AUC summary\) is itself an average over a noisy set of traces\. To quantify measure uncertainty we use a non\-parametric bootstrap: we resample the observed traces themselves and read the spread of the resulting statistic directly\.

Shaded bands in Figures[1](https://arxiv.org/html/2607.28707#S5.F1)and[3](https://arxiv.org/html/2607.28707#S5.F3)are obtained via non\-parametric bootstrap over traces, following the same procedure at every retention raterrand for every \(model, dataset, selector\) combination\.

For a given selectorss, letCi,rs∈\{0,1\}C^\{s\}\_\{i,r\}\\in\\\{0,1\\\}andCifull∈\{0,1\}C^\{\\text\{full\}\}\_\{i\}\\in\\\{0,1\\\}denote respectively whether traceiiis answered correctly under thess\-compressed CoT with retention rater∈\(0,1\]r\\in\(0,1\]and under the full CoT\. The point estimate of Relative Performance Retention is

RPRr=∑iCi,rs∑iCifull\\text\{RPR\}\_\{r\}=\\frac\{\\sum\_\{i\}C^\{s\}\_\{i,r\}\}\{\\sum\_\{i\}C^\{\\text\{full\}\}\_\{i\}\}
where the sum runs over allnntraces available for that \(model, dataset, selector\) combination, pooled across all evaluated questions\. Since up to88traces are sampled per question \(Section[4](https://arxiv.org/html/2607.28707#S4)\) and traces that do not reach\[EOT\]\[\\texttt\{EOT\}\]within the token budget are discarded \(Section[4](https://arxiv.org/html/2607.28707#S4)\),nnrepresents the upper bound of the considered traces\. We resample this set ofnntraces with replacementB=1000B=1000times; at each bootstrap iterationbb, we draw indices\{i1,…,in\}\\\{i\_\{1\},\\dots,i\_\{n\}\\\}uniformly with replacement from the originalnntraces and recomputeRPR\(b\)\\text\{RPR\}^\{\(b\)\}, discarding any resample with zero denominator\. The reported 95% confidence interval is the\[2\.5,97\.5\]\[2\.5,97\.5\]percentile range of the resulting bootstrapped distribution\{RPR\(b\)\}b=1B\\\{\\text\{RPR\}^\{\(b\)\}\\\}\_\{b=1\}^\{B\}\.

For AUC and consequentlyΔAUC\\Delta\_\{\\text\{AUC\}\}\(Tables[1](https://arxiv.org/html/2607.28707#S5.T1),[5](https://arxiv.org/html/2607.28707#A5.T5)\), the same per\-trace resampling is applied jointly across all retention rates within a given \(model, dataset, selector\) combination, before recomputing the trapezoidal AUC, rather than resampling independently at each retention rate\. Sincennis fixed acrossrrfor a given combination, the same resampled index set\{i1,…,in\}\\\{i\_\{1\},\\dots,i\_\{n\}\\\}is reused at every retention rate within a single bootstrap iterationbb\. This matters because accuracy at neighbouring retention rates is not independent: it is computed from the same underlying traces, and a trace answered correctly at one retention rate tends to remain so at higher rates too\. Resampling once per bootstrap iteration and reusing the same index set across all rates preserves this correlation, providing a more faithful interval than treating every point in the curve as an unrelated experiment\.

## Appendix DDetailed results

### D\.1Sentence\-level pruning

Table[5](https://arxiv.org/html/2607.28707#A5.T5)reports the full per\-model, per\-dataset breakdown ofΔAUC\\Delta\_\{\\text\{AUC\}\}for all four sentence\-level entropy\-based selectors compared to*random*, both over the full retention\-rate range and restricted to the aggressive\-compression regime \(r∈\[0\.01,0\.3\]r\\in\[0\.01,0\.3\]\), where any gap tends to widen\. Figure[5](https://arxiv.org/html/2607.28707#A5.F5)shows the corresponding curves for every model\-dataset pair across all six evaluated benchmarks\. Across the grid,*random*is at or above every entropy\-based selector in the majority of settings \(bold rows in Table[5](https://arxiv.org/html/2607.28707#A5.T5)\); the few exceptions are discussed in Appendix[E](https://arxiv.org/html/2607.28707#A5)\. The pattern is consistent across both mathematical and non\- datasets: none of the four entropy\-based selectors \(*low*,*low no numbers*,*numbers*,*high*\) reliably outperforms random selection at the sentence level, for any model family or scale we tested\.

### D\.2Token\-level pruning

Prompt:“What is the smallest positive perfect cube that can be written as the sum of three consecutive integers?”

Table 3:Tokens retained by each selector on a short MATH\-500 excerpt at retention rater=0\.10r=0\.10, and the model’s greedy\-decoded \(unpatched\) answer from each compressed text\.Greenmarks literal occurrences of the correct answer’s digits \(27\) within the reconstructed text\.*low\-entropy*generates the correct answer;*low\-entropy no numbers*and*numbers*both converge on the same incorrect answer,273=72927^\{3\}=729, despite*numbers*also containing the substring27multiple times, showing that lexical presence of the correct digits is not, by itself, sufficient without the surrounding context\.Similarly to the previous section, Table[6](https://arxiv.org/html/2607.28707#A5.T6)reportsΔAUC\\Delta\_\{\\text\{AUC\}\}for every token\-level selector relative to*random*, across all models and datasets \(patched results are reported separately in the next subsection\)\. Figure[6](https://arxiv.org/html/2607.28707#A5.F6)shows the corresponding full grid\.

Focusing on the three AIME benchmarks \(2024, 2025, 2026\) only, the*low\-entropy*advantage over random is consistent for four of the six models \(Qwen3\-14B,Qwen3\-4B,gemma\-4\-E4B\-it,gemma\-4\-26B\-A4B\-it\), while bothgpt\-ossmodel scales show negligible or reversed effect\. We attribute this recurring gap to the overlap between low\-entropy and numeric tokens rather than to entropy itself, as argued in Section[4](https://arxiv.org/html/2607.28707#S4)and further isolated in the worked example below\.

To make concrete how these three selectors diverge in practice, Table[3](https://arxiv.org/html/2607.28707#A4.T3)shows the tokens retained by each on an example of agpt\-oss\-20b’s MATH\-500 trace at a fixed retention rate \(r=0\.1r=0\.1\)\.*low\-entropy*retains a mix of numeric tokens and low\-information connectives; once numeric tokens are excluded,*low\-entropy no numbers*is left choosing almost exclusively syntactic filler, degrading the surface trace to a sequence of connectives stripped of computational content;*numbers*instead retains only digits and operators, producing a terse but computationally dense fragment\. This illustrates why*low\-entropy no numbers*tracks*numbers*so poorly once the numeric overlap is removed \(Section[4](https://arxiv.org/html/2607.28707#S4)\): what remains is syntax, not content\.

### D\.3Token\-level pruning with activation patching

Figure[4](https://arxiv.org/html/2607.28707#A4.F4)makes the picture from Table[7](https://arxiv.org/html/2607.28707#A5.T7)concrete in terms of retention budget, averaging the minimum retention rate needed to reachRPR≥0\.9\\text\{RPR\}\\geq 0\.9across all six models on AIME25\. Patching improves every selector to some degree, including*random*, whose own retention requirement drops from 73% to 32% of the trace once patched\. This is expected: patching restores full\-context activations at every retained position regardless of which positions were chosen, so it should help any selector to some extent, including one that discards content without regard to relevance\. The relevant question is therefore not whether patching helps, but how much, and for which selectors it changes the qualitative picture rather than merely shifting it\.

![Refer to caption](https://arxiv.org/html/2607.28707v1/x4.png)Figure 4:Minimum percentage of trace tokens needed to reach and sustain aRPRof at least0\.90\.9on AIME25, averaged across all six models, comparing patched \(full colour\) and unpatched \(lighter colour\)\. Hatched bars \(capped at 100%\) indicate that an unpatched compression using that selector is unable to retain the full trace performance\. Patching improves every selector to some degree \(including*random*, from 73% to 32%\), but the effect is markedly uneven:*numbers*moves from never reaching the threshold unpatched to the most efficient recovery of all once patched \(9%\), while*low\-entropy no numbers*only partially recovers \(70%\) and*high\-entropy*barely improves \(93% to 78%\)\.*numbers*and*low\-entropy no numbers*never reachRPR≥0\.9\\text\{RPR\}\\geq 0\.9at any tested retention rate without patching, on average \(their unpatched bars are capped at 100% in the figure, meaning that the full trace is required to close the gap\)\. Patching changes this qualitatively:*numbers*recovers the threshold using only 9% of the trace, the most efficient recovery among all tested selectors, while*low\-entropy no numbers*still requires 70%, only a partial recovery\.*low\-entropy*, whose unpatched advantage is itself largely inherited from its overlap with numeric tokens \(Section[4](https://arxiv.org/html/2607.28707#S4)\), needs 47% of the trace unpatched and 22% patched, a proportional improvement comparable to*random*’s, rather than the qualitative jump seen for*numbers*\.*high\-entropy*weakly improves \(93% to 78%\), remaining the least efficient selector in both conditions\.

## Appendix E*Numbers*selection exceeds full trace performance

Section[4](https://arxiv.org/html/2607.28707#S4)notes that there exists some token\-level cases in which the compressed CoT performance exceeds full\-CoT ones \(RPR\>1\\text\{RPR\}\>1\)\.

Table[4](https://arxiv.org/html/2607.28707#A5.T4)reports, separately for token\-level selection split by patching condition, the share of all tested \(model, dataset, compression rate\) points for which the point estimate exceeds 1\.0, and, among only those overshooting points, the share for which the 95% CI lies entirely above 1\.0\. These curves are visible directly in Figures[5](https://arxiv.org/html/2607.28707#A5.F5)\(sentence\-level\),[6](https://arxiv.org/html/2607.28707#A5.F6)\(token\-level\), and[7](https://arxiv.org/html/2607.28707#A5.F7)\(token\-level, patched\), where panels occasionally rise slightly above theRPR=1\\text\{RPR\}=1line\.

Table 4:Overshoot summary, per level, selector, and \(for token\-level\) patching condition, aggregated over all models and datasets\. %\>1\>1: share of all tested points whose point estimate exceeds full\-CoT accuracy\. % sig\. of these: among only the points exceeding 1\.0, the share whose 95% bootstrap CI lies entirely above 1\.0\.*newlines*\(\\n\) and*end of sentence*\(eos\) never overshoot under either condition\.The only selector that is both frequent and reliably above full\-trace performance is*numbers*under activation patching: numeric\-token activations occasionally carry enough task\-relevant information on their own to match or marginally exceed full\-trace performance, but only once patching supplies the context those activations would otherwise lack\. No selector improves original performance*without*patching and across models; the single isolated case of sentence\-level*high\-entropy*is limited toQwen3\-14Bon MATH\-500 and therefore cannot be used to argue that either entropy extreme uniquely identifies uninformative content\.

Table 5:AUC of*random*and delta of each entropy\-based selectorssrelative to it \(Δ=AUCs−AUCrandom\\Delta=\\mathrm\{AUC\}\_\{s\}\-\\mathrm\{AUC\}\_\{\\text\{random\}\}\), sentence\-level selection, shown for the full retention\-rate range and for the aggressive\-compression regime side by side\.Bold: all four deltas non\-positive \(random weakly outperforms or ties every selector\)\.![Refer to caption](https://arxiv.org/html/2607.28707v1/x5.png)Figure 5:Sentence\-levelfull grid: Relative Performance Retention vs\. compression rate for all six models \(rows\)×\\timessix datasets \(columns\), selectors*random*,*low\-entropy*,*high\-entropy*\. Shaded bands are 95% bootstrap CIs \(Appendix[C](https://arxiv.org/html/2607.28707#A3)\)\.*Random*is at or above both entropy\-based selectors across nearly every panel, consistent with Table[5](https://arxiv.org/html/2607.28707#A5.T5)\.Table 6:AUC of*random*and delta of eachtoken\-levelselector relative to it \(Δ=AUCselector−AUCrandom\\Delta=\\mathrm\{AUC\}\_\{\\text\{selector\}\}\-\\mathrm\{AUC\}\_\{\\text\{random\}\}\),no patching, full retention\-rate range and aggressive\-compression regime\.BoldonΔ\\Deltalow:*low\-entropy*strictly outperforms both*random*\(Δ\>0\\Delta\>0\) and every other selector in that setting\.![Refer to caption](https://arxiv.org/html/2607.28707v1/x6.png)Figure 6:Token\-level full grid, no patching:Relative Performance Retention vs\. compression rate, selectors*random*,*low\-entropy*,*high\-entropy*,*low\-entropy no numbers*,*numbers*\. Model ordering:gpt\-oss\-20b, gpt\-oss\-120b, gemma\-4\-E4B\-it, gemma\-4\-26B\-A4B\-it, Qwen3\-14B, Qwen3\-4B\.Table 7:AUC of*random*and delta of eachtoken\-levelselector relative to it \(Δ=AUCselector−AUCrandom\\Delta=\\mathrm\{AUC\}\_\{\\text\{selector\}\}\-\\mathrm\{AUC\}\_\{\\text\{random\}\}\),patched, full retention\-rate range\. Last column: newlines / end\-of\-sentence deltas\.Boldmarks the best\-performing selector \(highest resulting AUC\) per row\.![Refer to caption](https://arxiv.org/html/2607.28707v1/x7.png)Figure 7:Token\-level patched full grid:Relative Performance Retention vs\. compression rate, selectors*random*,*low\-entropy*,*high\-entropy*,*low\-entropy no numbers*,*numbers*\. Model ordering:gpt\-oss\-20b, gpt\-oss\-120b, gemma\-4\-E4B\-it, gemma\-4\-26B\-A4B\-it, Qwen3\-14B, Qwen3\-4B\.![Refer to caption](https://arxiv.org/html/2607.28707v1/x8.png)Figure 8:Sentence\-level full grid undermath\-verify: Relative Performance Retention vs\. compression rate for all six models \(rows\)×\\timessix datasets \(columns\), selectors*random*,*low\-entropy*,*high\-entropy*\. The pattern matches Figure[5](https://arxiv.org/html/2607.28707#A5.F5)\(exact\-match\):*random*is at or above both entropy\-based selectors in the large majority of panels, confirming that the sentence\-level results are not an artifact of the matching criterion\.![Refer to caption](https://arxiv.org/html/2607.28707v1/x9.png)Figure 9:Token\-level full grid undermath\-verify\(no patching\): Relative Performance Retention vs\. compression rate for all six models \(rows\)×\\timessix datasets \(columns\), selectors*random*,*low\-entropy*,*high\-entropy*\. The pattern matches Figure[6](https://arxiv.org/html/2607.28707#A5.F6)\(exact\-match\):*low\-entropy*exceeds*random*on mathematical benchmarks only\.