Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning

# Diagnosing and Training LLMs to Abort Futile Reasoning
Source: [https://arxiv.org/html/2607.29211](https://arxiv.org/html/2607.29211)
Xinyan Guan1,2,Jiali Zeng3,†\\dagger,Chunlei Xin1,2,Yaojie Lu1,†\\dagger, Hongyu Lin1,Xianpei Han1,Le Sun1,Fandong Meng3 1Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences 3Weixin AI, Tencent Inc, China \{guanxinyan2022,chunlei2021,luyaojie,hongyu,xianpei,sunle\}@iscas\.ac\.cn \{lemonzeng,fandongmeng\}@tencent\.com

###### Abstract

Large language models generate computationally expensive yet semantically void reasoning on beyond\-capability tasks, creating risks where plausible\-sounding but incorrect derivations mislead users\. We characterize thisfutile reasoningphenomenon through systematic analysis, revealing universal capability overreach and systematic miscalibration between capability and behavior\. The dominant failure mode is specious reasoning, which outputs look superficially valid but contain subtle errors, escalating with task difficulty\. To address this, we introduceCaRL\(Capability\-alignedReinforcementLearning\), which aligns model behavior with capability boundaries through reward shaping that incentivizes refusal over futile reasoning and hindsight refusal augmentation that converts failures into refusal supervision\. Experiments demonstrate a substantial reduction in futile reasoning while preserving performance across task difficulties, effectively achieving capability\-aligned behavior without sacrificing utility\.111[https://github\.com/icip\-cas/Knowing\-When\-to\-Quit](https://github.com/icip-cas/Knowing-When-to-Quit)

Knowing When to Quit: Diagnosing and Training LLMs to AbortFutile Reasoning

Xinyan Guan1,2, Jiali Zeng3,†\\dagger, Chunlei Xin1,2, Yaojie Lu1,†\\dagger,Hongyu Lin1,Xianpei Han1,Le Sun1,Fandong Meng31Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences2University of Chinese Academy of Sciences3Weixin AI, Tencent Inc, China\{guanxinyan2022,chunlei2021,luyaojie,hongyu,xianpei,sunle\}@iscas\.ac\.cn\{lemonzeng,fandongmeng\}@tencent\.com

00footnotetext:Corresponding author\.## 1Introduction

Large language models \(LLMs\), characterized by billions of parameters, have emerged as powerful tools for solving various tasksXu et al\. \([2025](https://arxiv.org/html/2607.29211#bib.bib29)\); Minaee et al\. \([2024](https://arxiv.org/html/2607.29211#bib.bib22)\); Zhao et al\. \([2023](https://arxiv.org/html/2607.29211#bib.bib34)\)\. Recent advances in generative reasoning models \(GRMs\) further amplify their reasoning capabilities through test\-time scalingXu et al\. \([2025](https://arxiv.org/html/2607.29211#bib.bib29)\); Huang and Chang \([2023](https://arxiv.org/html/2607.29211#bib.bib12)\), unlocking the ability to solve increasingly complex reasoning problems\.

However, when confronted with tasks that exceed their intrinsic capabilities, LLMs fail to recognize their boundaries\. Instead of acknowledging their ignorance, they often persist in generating output, producing outputs that superficially resemble valid solutions\. Due to the lengthy and convoluted nature of these generated traces, distinguishing valid reasoning from such hallucinations is notoriously difficult for humansBowman et al\. \([2022](https://arxiv.org/html/2607.29211#bib.bib3)\); Perez et al\. \([2023](https://arxiv.org/html/2607.29211#bib.bib25)\)\. Consequently, this creates a critical reliability risk: users may mistakenly trust plausible\-sounding but fundamentally incorrect derivations, severely limiting the adoption of LLMs in high\-reliability domainsLiu et al\. \([2023](https://arxiv.org/html/2607.29211#bib.bib21)\)\.

![Refer to caption](https://arxiv.org/html/2607.29211v1/x1.png)Figure 1:Illustration offutile reasoninggenerated by DeepSeek\-R1Guo et al\. \([2025](https://arxiv.org/html/2607.29211#bib.bib11)\)\. When faced with problems beyond its capability, the model generates plausible\-looking but fundamentally incorrect reasoning traces instead of refusing to answer\.We term this phenomenonfutile reasoning, referring to the plausibly\-looking but fundamentally incorrect reasoning attempts triggered when models face problems beyond their capabilities\. As illustrated in Figure[1](https://arxiv.org/html/2607.29211#S1.F1), the model exhibits two distinct behavioral regimes\. For problems within its capability boundary, it engages in productive reasoning and derives correct solutions\. However, when operating beyond this boundary, it fails to recognize its limitations and instead produces spurious outputs\. For instance, on a Countdown task requiring construction of target 0\.3 from numbers \[1,5,6,8,9\], the model attempts multiple approaches but ultimately fabricates an incorrect expression and falsely claims it evaluates to 0\.3 \(the actual result is 0\.6\)\. Instead, it should simply acknowledge “I cannot reach 0\.3 with the given numbers”\. Such failures typically manifest as specious logical steps or endless generation, creating a deceptive illusion of competence\. Ideally, models should attempt to reason on challenging problems, but abort and explicitly refuse when they recognize, after several reasoning steps, that a solution is unattainable\.

To quantify this phenomenon, we conduct a comprehensive empirical study using a difficulty\-graded Countdown task \(N=3N=3toN=8N=8\)\. We observe that models exhibit universal capability overreach: vanilla models never refuse \(0% across all difficulty levels\), and even explicit prompting fails to prevent over 80% of futile attempts on the hardest tasks\. The dominant failure mode isspecious reasoning, superficially valid outputs containing subtle hallucinations, whose prevalence escalates with task difficulty\. Moreover, by quantifying the alignment gap under standard prompting, we reveal systematic miscalibration with over\-confidence \(20%\) occurring 6× more frequently than over\-conservatism \(3\.4%\), and reducing refusal recall below 30% on hard tasks while causing 10% capability loss on easy ones, indicating that stronger interventions are required\.

Building on these insights, we introduceCaRL\(Capability\-alignedReinforcementLearning\), a framework designed to align model behavior with capability boundaries\. CaRL operates through two mechanisms: \(1\)Capability\-Calibrated Reward Shaping, which establishes a strict reward hierarchy to incentivize refusal over futile reasoning, and \(2\)Hindsight Refusal Augmentation, which enhances refusal samples by converting futile reasoning into refusal format\. Together, these mechanisms enable models to explore solution paths and abort when they detect insurmountable difficulties, rather than fabricating specious outputs\. CaRL can be applied to an arbitrary model to instill capability boundary awareness\. Experiments on Qwen3\-8B and Qwen3\-14B demonstrate that CaRL preserves performance on both in\-distribution and out\-of\-distribution tasks while reducing futile reasoning \(8B: 65\.5% → 7\.0%; 14B: 78\.6% → 1\.0%\), effectively calibrating the model’s behavioral boundary with its true competence\.

Our contributions are summarized as follows:

- •We identify and characterizefutile reasoning, demonstrating that models generate plausible\-looking but fundamentally incorrect reasoning when facing beyond\-capability tasks\.
- •We conduct a systematic empirical analysis, and reveal three critical phenomena: \(1\) universal capability overreach, \(2\) specious reasoning as the dominant failure mode escalating with difficulty, and \(3\) a substantial capability\-behavior alignment gap with 6× bias towards over\-confidence, demonstrating that implicit behavioral cues are insufficient\.
- •We proposeCaRL, combining capability\-calibrated reward shaping and hindsight refusal augmentation, achieving substantial reduction in futile reasoning while preserving task performance\.

## 2Related Work

#### Futile Reasoning\.

Recent analysis has identified specific pathologies in model reasoning\.Chen et al\. \([2024](https://arxiv.org/html/2607.29211#bib.bib7)\)demonstrates that a significant proportion of errors in LLMs stems from overthinking\. Furthermore,Yeo et al\. \([2025](https://arxiv.org/html/2607.29211#bib.bib31)\)highlights that a primary obstacle for long\-thinking models is the tendency towards repetition\. These findings corroborate our characterization offutile reasoning, where models engage in repetitive loops or generate specious justifications that appear plausible but are functionally void\.

#### Capability Alignment and Honesty\.

To mitigate overconfidence, prior works have explored the model’s ability to refuse and its self\-knowledgeCheng et al\. \([2024](https://arxiv.org/html/2607.29211#bib.bib8)\); Kadavath et al\. \([2022](https://arxiv.org/html/2607.29211#bib.bib16)\)\. However, these studies predominantly focus onknowledge boundaries\(i\.e\., factual retrieval\), largely overlooking thecapability boundariesessential for reasoning tasks\. More recently,Joglekar et al\. \([2025](https://arxiv.org/html/2607.29211#bib.bib15)\)emphasizes the importance of achieving honesty via confession, suggesting that models should be trained to recognize and correct their own errors\. Our work extends this by using reinforcement learning to internalize this repentance, teaching models to abort futile reasoning when intrinsic capability limits are reached\.

## 3Preliminary Experiments and Analyses

We conduct a systematic empirical analysis to characterize the nature offutile reasoning\. Our primary objective is to quantify the misalignment between LLMs’ generative behavior and their intrinsic competence boundaries\. Using a controlled reasoning testbed, we investigate three key questions: \(1\) Do models refuse beyond\-capability tasks, or exhibit universal capability overreach? \(2\) What are the characteristic failure patterns of futile reasoning? \(3\) Are models’ refusal decisions properly calibrated with their empirical capabilities?

### 3\.1Experimental Setup: The Countdown Task

To decouplereasoning capabilityfrom knowledge retrievalHuang et al\. \([2025a](https://arxiv.org/html/2607.29211#bib.bib13)\); Sahoo et al\. \([2024](https://arxiv.org/html/2607.29211#bib.bib27)\), we utilize the Countdown Task \(a variant of the Game of 24\) as our primary testbed, as it allows precise manipulation of difficulty without confounding factors of external knowledge\. A detailed example is shown in Appendix[A\.1](https://arxiv.org/html/2607.29211#A1.SS1)\.

### 3\.2Phenomenon I: Universal Capability Overreach

![Refer to caption](https://arxiv.org/html/2607.29211v1/x2.png)Figure 2:Universal Capability Overreach\.We first investigate whether models possess an intrinsic mechanism to acknowledge ignorance when facing unsolvable problems\. We conduct experiments on Qwen3\-8BYang et al\. \([2025](https://arxiv.org/html/2607.29211#bib.bib30)\), Qwen3\-32BYang et al\. \([2025](https://arxiv.org/html/2607.29211#bib.bib30)\), gpt\-oss\-120bOpenAI \([2025](https://arxiv.org/html/2607.29211#bib.bib23)\), Qwen3\-235B\-A22BYang et al\. \([2025](https://arxiv.org/html/2607.29211#bib.bib30)\), and DeepSeek\-V3\.2Liu et al\. \([2025](https://arxiv.org/html/2607.29211#bib.bib20)\)across difficulty levels \(N=3N=3toN=8N=8\)\. We experiment on two settings: “Baseline” refers to standard generation with long chain\-of\-thought reasoning, while “Prompted” employs explicit instructions encouraging the model to acknowledge ignorance when facing beyond\-capability tasks\.

As illustrated in Figure[2](https://arxiv.org/html/2607.29211#S3.F2), we report the error rate of each model while visualizing its refusal rate through the radius of each data point\. Ideally, a well\-calibrated model should exhibit a dynamic response: as the error rate increases, the radius of the point should expand significantly, indicating that the model correctly identifies the rising difficulty and opts to refuse rather than hallucinate\.

However, we observe a critical calibration failure\. Despite error rates surging with task difficulty \(e\.g\., Qwen3\-8B exceeds 65% error atN=8N=8\), the corresponding bubbles remain negligibly small across all models\. Even in the prompted setting, the marginal expansion in bubble size is disproportionate to the massive increase in errors\. The persistence of these tiny bubbles at high error rates reveals that models blindly attempt to reason through unsolvable queries despite explicit instructions to the contrary\.

Specifically, gpt\-oss\-120b remains obstinately non\-compliant, failing to refuse even under the prompted setting\. In contrast, Qwen3\-235B\-A22B and DeepSeek\-V3\.2 demonstrate latent refusal capabilities that can be effectively elicited through prompting\. Moreover, DeepSeek\-V3\.2 is the unique model that displays traces of spontaneous self\-doubt \(albeit in <1%\) under the baseline setting\.All these phenomena show that models lack intrinsic mechanisms to recognize capability boundaries, exhibiting universal capability overreach regardless of model scale or task difficulty\.

### 3\.3Phenomenon II: Failure Patterns of Futile Reasoning

![Refer to caption](https://arxiv.org/html/2607.29211v1/x3.png)Figure 3:Distribution of Futile Reasoning Patterns\.We categorize futile reasoning into three distinct patterns\.Specious Reasoningconstructs apparently coherent derivations that contain subtle errors \(e\.g\., arithmetic mistakes, reusing numbers\), creating an illusion of competence\.Endless Generationcontinuously proposes new attempts without recognizing futility, exhaustively exploring combinations\.Degenerate Repetitionenters recursive loops, repeating identical reasoning steps\.

Figure[3](https://arxiv.org/html/2607.29211#S3.F3)reveals that specious reasoning dominates \(57\-68% across levels\), while endless generation remains stable \(30\-40%\) and degenerate repetition decreases \(13%→2%\)\. Critically, as difficulty increases, models shift from simple loops to suspicious reasoning\.This indicates thatwhen pushed beyond capability, models do not degenerate randomly; instead, they fabricate increasingly sophisticated justifications to maintain a facade of correctness\.

### 3\.4Phenomenon III: Quantifying Capability\-Behavior Misalignment

We quantify the alignment gap between ideal capability\-aware behavior and actual model responses\. Following prior workLin et al\. \([2022](https://arxiv.org/html/2607.29211#bib.bib18)\); Kadavath et al\. \([2022](https://arxiv.org/html/2607.29211#bib.bib16)\); Cheng et al\. \([2024](https://arxiv.org/html/2607.29211#bib.bib8)\), we sample multiple responses per question and classify it as solvable if the model achieves an accuracy \> 0\.5, and unsolvable otherwise\. We then define theCapability Quadrants, categorizing model behaviors into four outcomes: A\.Ideal Answer\(solvable, answered\), B\.Over\-Conservative\(solvable, refused\), C\.Over\-Confidence\(unsolvable, answered\), and D\.Ideal Refusal\(unsolvable, refused\)\. We analyze Qwen3\-32B responses in this section

![Refer to caption](https://arxiv.org/html/2607.29211v1/x4.png)Figure 4:Distribution of Capability Quadrants\.#### Asymmetry of Misalignment\.

Figure[4](https://arxiv.org/html/2607.29211#S3.F4)reveals a striking 6× asymmetry: Over\-Confidence \(20%\) vastly exceeds Over\-Conservative \(3\.4%\)\. This directional bias indicatesmodels systematically over\-estimate capability rather than exhibiting random uncertainty\.

#### Degradation Across the Difficulty Spectrum\.

To understand how the capability quadrants shift with task difficulty, we examine two aggregate metrics derived from the quadrant distribution\.Refusal Recallmeasures safety by quantifying what fraction of unsolvable tasks fall into theIdeal Refusalquadrant \(versusOver\-Confidence\)\.Capability Lossmeasures utility by quantifying what fraction of solvable tasks fall into theOver\-Conservativequadrant \(versusIdeal Answer\)\. Figure[5](https://arxiv.org/html/2607.29211#S3.F5)shows that as difficulty increases, Refusal Recall collapses \(100%→30%\) while Capability Loss rises \(0%→10%\)\. This revealspoor calibration: models apply uniform shifts that underprotect hard tasks while harming solvable ones\.

![Refer to caption](https://arxiv.org/html/2607.29211v1/x5.png)Figure 5:The Alignment Trade\-off\.Naive prompting leads to a collapse in Refusal Recall \(Green\) on hard tasks while simultaneously increasing Capability Loss \(Pink\) on solvable tasks\.![Refer to caption](https://arxiv.org/html/2607.29211v1/x6.png)Figure 6:Reasoning Depth Distribution\.Refusal behaviors show decisive termination \(Peaks\), whereas Over\-Confidence exhibits a long\-tail distribution, confirming the high computational cost of futile reasoning\.
#### Computational Cost of Misalignment\.

Beyond correctness, misalignment imposes computational costs\. Figure[6](https://arxiv.org/html/2607.29211#S3.F6)shows token length distributions across behavioral quadrants based on KDEDavis et al\. \([2011](https://arxiv.org/html/2607.29211#bib.bib10)\); Parzen \([1962](https://arxiv.org/html/2607.29211#bib.bib24)\)\. Refusal behaviors \(Ideal Refusal and Over\-Conservative\) exhibit sharply peaked distributions, indicating decisive termination\. In contrast, Over\-Confidence shows a flat distribution with a pronounced long tail, generating 2\-3× more tokens than proper refusals\. This confirms that when models attempt unsolvable tasks, they engage in prolonged futile reasoning, consuming excessive computation without producing value\. Consequently, mitigating the misalignment may offer an additional benefit beyond correctness: reducing unnecessary computational overhead from futile reasoning attempts\.

![Refer to caption](https://arxiv.org/html/2607.29211v1/x7.png)Figure 7:Overview of CaRL\.Our framework combines \(1\)Capability\-Calibrated Reward Shapingthat establishes a preference hierarchy favoring refusal over hallucination, and \(2\)Hindsight Refusal Augmentationthat converts failed trajectories into refusal trajectories, enabling capability\-aligned behavior\.

## 4Capability\-aligned Reinforcement Learning

To alleviate futile reasoning, we proposeCaRL\(Capability\-alignedReinforcementLearning\), a framework that aligns model behavior with capability boundaries through explicit reinforcement learning\. As shown in Figure[7](https://arxiv.org/html/2607.29211#S3.F7), CaRL consists of two main components: \(1\)Capability\-Calibrated Reward Shaping\(§[4\.1](https://arxiv.org/html/2607.29211#S4.SS1)\) encodes capability boundaries into the reward structure through a hierarchical design that penalizes capability overreach while preserving the incentive for productive reasoning\. \(2\)Hindsight Refusal Augmentation\(§[4\.2](https://arxiv.org/html/2607.29211#S4.SS2)\) addresses the scarcity of refusal training data by retrospectively converting failed reasoning attempts into refusal demonstrations, densifying the sparse reward landscape for refusal\.

### 4\.1Capability\-Calibrated Reward Shaping

Standard reasoning RL typically assigns 1 for correct answers and 0 for incorrect answers\. This leads to uniform low rewards to both incorrect answers and refusals, treating them as equivalent failures\. This creates no incentive for the model to distinguish between attempting and refusing, encouraging futile reasoning since there is no penalty for generating plausible\-sounding but incorrect outputs\.

To rectify this, we reshape the reward function to explicitly encode the preference for safe refusal over futile attempts\. We define the rewardr​\(c\)r\(c\)for a generated solutionccas:

r​\(c\)=\{\+1if​c​is correct0if​c​is a valid refusal−1if​c​is incorrectr\(c\)=\\begin\{cases\}\+1&\\text\{if \}c\\text\{ is correct\}\\\\ \\phantom\{\+\}0&\\text\{if \}c\\text\{ is a valid refusal\}\\\\ \-1&\\text\{if \}c\\text\{ is incorrect\}\\end\{cases\}\(1\)where a valid refusal is identified by explicit refusal snippets \(e\.g\., “Sorry, I can’t solve the problem\.”\)\.

### 4\.2Hindsight Refusal Augmentation

While reward shaping provides the incentive for refusal, it does not solve the data scarcity problem\. Models fine\-tuned for helpfulness rarely generate refusals during on\-policy explorationKashyap et al\. \([2025](https://arxiv.org/html/2607.29211#bib.bib17)\); Huang et al\. \([2025b](https://arxiv.org/html/2607.29211#bib.bib14)\); Bai et al\. \([2022](https://arxiv.org/html/2607.29211#bib.bib2)\), and RL exploration is notoriously inefficient in unfamiliar behavioral regionsChen et al\. \([2025b](https://arxiv.org/html/2607.29211#bib.bib5),[a](https://arxiv.org/html/2607.29211#bib.bib4)\); Zhang et al\. \([2025](https://arxiv.org/html/2607.29211#bib.bib32)\)\. Our analysis confirms this challenge: as shown in §[3\.4](https://arxiv.org/html/2607.29211#S3.SS4), baseline models exhibit 0% refusal rate, indicating that appropriate refusal lies far outside the model’s natural behavioral distribution\. Consequently, on\-policy exploration would require prohibitively many samples before encountering sufficient refusal examples, rendering the reward structure alone insufficient for effective learning\.

We address this throughHindsight Refusal Augmentation \(HRA\)\. Rather than waiting for rare on\-policy refusals, we leverage a key observation:every failed reasoning attempt implicitly reveals a situation where refusal would have been the appropriate action\. By retrospectively converting these abundant failures into synthetic refusal demonstrations, HRA densifies the sparser=0r=0training signal, enabling the policy to learn refusal boundaries efficiently\.

For each on\-policy trajectoryτ=\(q,c\)\\tau=\(q,c\)that generates an incorrect answercc\(receivingr=−1r=\-1\) based on a queryqq, we construct an augmented refusal sampleτaug=\(q,crefuse\)\\tau\_\{\\text\{aug\}\}=\(q,c\_\{\\text\{refuse\}\}\)as follows\. We first retain the reasoning trace up to \(but excluding\) the final answer step, then insert a refusal prefix such as “Sorry, I cannot solve this problem\. Here is how far I got:”\. Next, we prompt the model to generate a brief summary of the attempted reasoning progress, creating a natural refusal response that explains what was tried\. Finally, we assign the refusal rewardr=0r=0to this modified trajectory and add it to the training batch alongside the original failure\.

This creates a contrastive learning signal: under the same reasoning context that led to failure, the model learns that choosing to refuse \(r=0r=0\) yields higher reward than persisting to an incorrect conclusion \(r=−1r=\-1\)\. By systematically augmenting every failure with its refusal counterpart, HRA densifies the sparse reward landscape, enabling the policy to learn appropriate refusal boundaries even when on\-policy exploration rarely produces refusals\.

### 4\.3Training Procedure

Algorithm[1](https://arxiv.org/html/2607.29211#alg1)details the complete training procedure\. At each iteration, we generate multiple outputs per question, compute capability\-calibrated rewards \(§[4\.1](https://arxiv.org/html/2607.29211#S4.SS1)\), and apply HRA to augment failed attempts with refusal samples \(§[4\.2](https://arxiv.org/html/2607.29211#S4.SS2)\)\. The policy is then updated using group\-relative advantages computed from the augmented dataset\. We employ the GRPOLiu et al\. \([2024](https://arxiv.org/html/2607.29211#bib.bib19)\); Guo et al\. \([2025](https://arxiv.org/html/2607.29211#bib.bib11)\)objective for optimization\.

Algorithm 1CaRL Training Procedure1:Input:Policy

πθ\\pi\_\{\\theta\}, training set

𝒬\\mathcal\{Q\}, group size

GG, batch size

BB
2:foriteration

t=1,…,Tt=1,\\ldots,Tdo

3:Sample batch:

\{qi\}i=1B∼𝒬\\\{q\_\{i\}\\\}\_\{i=1\}^\{B\}\\sim\\mathcal\{Q\}
4:// Rollout Generation

5:Generate

GGoutputs per question:

\{ci,j\}j=1G∼πθ\\\{c\_\{i,j\}\\\}\_\{j=1\}^\{G\}\\sim\\pi\_\{\\theta\}for each

qiq\_\{i\}
6:// Capability\-Calibrated Reward Shaping

7:Compute rewards via Eq\. \([1](https://arxiv.org/html/2607.29211#S4.E1)\):

ri,j←r\(ci,jr\_\{i,j\}\\leftarrow r\(c\_\{i,j\}\)

8:Collect rollouts:

𝒟t←\{\(τi,j,ri,j\)\}\\mathcal\{D\}\_\{t\}\\leftarrow\\\{\(\\tau\_\{i,j\},r\_\{i,j\}\)\\\}where

τi,j=\(qi,ci,j\)\\tau\_\{i,j\}=\(q\_\{i\},c\_\{i,j\}\)
9:// Hindsight Refusal Augmentation

10:foreach

\(τi,j,ri,j\)∈𝒟t\(\\tau\_\{i,j\},r\_\{i,j\}\)\\in\\mathcal\{D\}\_\{t\}where

ri,j=−1r\_\{i,j\}=\-1do

11:Construct refusal rollout:

τi,jaug\\tau\_\{i,j\}^\{\\text\{aug\}\}
12:Augment:

𝒟t←𝒟t∪\{\(τi,jaug,r=0\)\}\\mathcal\{D\}\_\{t\}\\leftarrow\\mathcal\{D\}\_\{t\}\\cup\\\{\(\\tau\_\{i,j\}^\{\\text\{aug\}\},r=0\)\\\}
13:// Policy update via GRPO

14:Compute group advantages and update

πθ\\pi\_\{\\theta\}
15:Output:

πθ\\pi\_\{\\theta\}

## 5Experiments

MethodIn\-Distribution \(Countdown\)Out\-of\-Distribution \(Sudoku\)Acc↑\\uparrowReliability↑\\uparrowRefusalFutile↓\\downarrowAcc↑\\uparrowReliability↑\\uparrowRefusalFutile↓\\downarrowQwen3\-8BVanilla59\.670\.666313\.9265\.5046\.880\.496910\.6289\.41Standard RL64\.08\(\+4\.4\)0\.6425\(\-\.02\)0\.3399\.00\(\+33\.5\)43\.25\(\-3\.6\)0\.4744\(\-\.02\)13\.1385\.00\(\-4\.4\)RLunk=062\.71\(\+3\.0\)0\.6296\(\-\.04\)0\.5099\.00\(\+33\.5\)44\.62\(\-2\.3\)0\.4850\(\-\.01\)12\.1286\.00\(\-3\.4\)RLunk=0\.563\.42\(\+3\.8\)0\.6371\(\-\.03\)0\.5898\.00\(\+32\.5\)45\.25\(\-1\.6\)0\.5131\(\+\.02\)16\.7578\.00\(\-11\.4\)RFT59\.13\(\-0\.5\)0\.7610\(\+\.09\)35\.2117\.00\(\-48\.5\)0\.00\(\-46\.9\)0\.4763\(\-\.02\)95\.255\.00†CaRL \(Ours\)61\.00\(\+1\.3\)0\.7915\(\+\.13\)37\.177\.00\(\-58\.5\)46\.25\(\-0\.6\)0\.6156\(\+\.12\)36\.6343\.02\(\-46\.4\)Qwen3\-14BVanilla63\.250\.67197\.8878\.5750\.750\.555618\.6280\.46Standard RL56\.42\(\-6\.8\)0\.5750\(\-\.10\)2\.1795\.03\(\+16\.5\)43\.63\(\-7\.1\)0\.4831\(\-\.07\)13\.1383\.00\(\+2\.5\)RLunk=068\.21\(\+5\.0\)0\.8050\(\+\.13\)24\.5823\.00\(\-55\.6\)48\.38\(\-2\.4\)0\.5369\(\-\.02\)14\.0079\.00\(\-1\.5\)RFT63\.12\(\-0\.1\)0\.7879\(\+\.12\)31\.3315\.00\(\-63\.6\)0\.00\(\-50\.8\)0\.4525\(\-\.10\)90\.5010\.00†CaRL \(Ours\)67\.25\(\+4\.0\)0\.8348\(\+\.16\)32\.501\.00\(\-77\.6\)44\.87\(\-5\.9\)0\.6262\(\+\.07\)38\.8736\.00\(\-44\.5\)†RFT’s low futile rate on OOD is a trivial result of collapsing into near\-total refusal \(Ref\>90%\>90\\%, Acc=0%=0\\%\)\.

Table 1:Main Results on In\-Distribution and Out\-of\-Distribution Tasks\.Values in parentheses show changes relative to Vanilla baseline\.Greenindicates improvement,redindicates degradation\.### 5\.1Datasets

We utilize thereasoning\-gymframework222[https://github\.com/open\-thought/reasoning\-gym](https://github.com/open-thought/reasoning-gym)to construct our benchmarks\. For the in\-distributionCountdowntask, we generate 1,000 training and 100 evaluation samples for each of the three difficulty levels \(N=4,6,8N=4,6,8\)\. For out\-of\-distribution \(OOD\) evaluation, we synthesize 100Sudokusamples, which require filling a partially completed 9×9 grid such that each row, column, and 3×3 subgrid contains all digits 1\-9 without repetition\. Notably, Sudoku puzzles necessitate significantly longer reasoning chains compared to Countdown tasks\. We leverage this characteristic as a stress test to verify whether the model maintains robust reasoning on complex queries, or if it simply collapses into over\-refusal when facing high computational costs\.

### 5\.2Baselines

We evaluate performance against the following baselines\.Vanillarefers to the raw reasoning model without further alignment\.Standard RLoptimizes solely for task accuracy using a binary reward structure \(r=1r=1for correct answers,r=−1r=\-1otherwise\)\. RFTZhang et al\. \([2023](https://arxiv.org/html/2607.29211#bib.bib33)\)\(Rejection Sampling Fine\-Tuning\) performs supervised fine\-tuning on a static dataset constructed by filtering for both correct reasoning paths and valid refusals\. RLunkadopts our capability\-calibrated reward shaping to decouple the impact of reward signals from HRA\. Specifically, we implement two strategies:RLunk=0assigns a neutral reward \(r=0r=0\) to refusal responses, whileRLunk=0\.5assigns a positive partial reward \(r=0\.5r=0\.5\)\.

### 5\.3Evaluation Metrics

To comprehensively assess the model’s performance and alignment boundaries, we utilize four quantitative metrics\. LetNcN\_\{c\},NrN\_\{r\}, andNiN\_\{i\}denote the number of correct, refusal, and incorrect responses, respectively, withN=Nc\+Nr\+NiN=N\_\{c\}\+N\_\{r\}\+N\_\{i\}as the total number of samples\.

#### Accuracy \(Acc\)\.

Success rate:Acc=Nc/N\\text\{Acc\}=N\_\{c\}/N\.

#### Reliability Score\.

Weighted metric assigning 1\.0 for correct, 0\.5 for refusal, 0 for incorrect:Reliability=\(Nc\+0\.5​Nr\)/N\\text\{Reliability\}=\(N\_\{c\}\+0\.5N\_\{r\}\)/N\.

#### Refusal Rate\.

Proportion of acknowledged ignorance:Refusal Rate=Nr/N\\text\{Refusal Rate\}=N\_\{r\}/N\.

#### Futile Rate\.

Proportion of failures where the model generates futile reasoning instead of refusing:Futile Rate=Ni/\(Ni\+Nr\)\\text\{Futile Rate\}=N\_\{i\}/\(N\_\{i\}\+N\_\{r\}\)\.

### 5\.4Main results

As presented in Table[1](https://arxiv.org/html/2607.29211#S5.T1), we evaluate Qwen3\-8B and Qwen3\-14B across in\-distribution \(Countdown\) and out\-of\-distribution \(Sudoku\) settings, comparing against the vanilla baseline\. The analysis reveals three critical findings\.

#### CaRL achieves optimal performance across scales\.

CaRL consistently attains the highest reliability and lowest futile rate across both model sizes\. For example, on Qwen3\-8B, CaRL achieves reliability 0\.7915 \(\+0\.13\) with 7\.00% futile rate\. On 14B, these improvements amplify: reliability 0\.8348 \(\+0\.16\) with merely 1\.00% futile rate, demonstrating that larger models benefit more from capability\-aligned training\. In contrast, Standard RL improves accuracy on 8B but catastrophically increases futile reasoning to 99%; on 14B, it degrades both accuracy and reliability, confirming that naive RL optimization exacerbates over\-confidence\. This pattern persists out\-of\-distribution: CaRL maintains the highest Reliability and lowest futile rate, whereas Standard RL consistently degrades across metrics\.

#### Reward shaping alone remains insufficient\.

The RLunkvariants show strikingly different behaviors across scales\. On 8B, both RLunk=0and RLunk=0\.5maintain 98\-99% futile rate, indicating a failure of calibrating capability boundary\. However, on 14B, RLunk=0achieves improvement, suggesting that larger models can benefit more from reward shaping\. Nevertheless, this remains substantially worse than CaRL, confirming our analysis in §[4\.2](https://arxiv.org/html/2607.29211#S4.SS2): without dense refusal supervision from HRA, models struggle to discover appropriate refusal boundaries through reward\-only signals\. The 14B results demonstrate that while model scale helps, HRA remains critical for efficient capability alignment\.

#### RFT fails to generalize\.

RFT shows reasonable in\-distribution results but catastrophic OOD collapse on both 8B and 14B\. In contrast, CaRL maintains consistent behavior \(ID Refusal: 37\.17% → OOD: 36\.63%\) with substantial futile reduction \(from 89\.41% to 43\.02%\), demonstrating genuine capability boundary learning versus pattern memorizationChu et al\. \([2025](https://arxiv.org/html/2607.29211#bib.bib9)\)\.

### 5\.5Generalization and Efficiency Analysis

We evaluate how capability awareness generalizes to different difficulty levels through futile rate and response length across levelsN=4,6,8N=4,6,8\(Table[2](https://arxiv.org/html/2607.29211#S5.T2)\)\.

![Refer to caption](https://arxiv.org/html/2607.29211v1/x8.png)Figure 8:Case study on the countdown task\.Table 2:Futile Rate\(%\) and response length across difficulty levels on Qwen3\-8B\.#### Generalization\.

RLunkmaintains \>95% futile rate across all levels, confirming that reward shaping alone cannot induce capability awareness\. RFT exhibitsgeneralization degradation: futile rate escalates 7× from 2\.82% \(Level 4\) to 20\.36% \(Level 8\), indicating supervised learning fails to extrapolate to harder tasks\. In contrast, CaRL maintains robust generalization \(1\.96% → 8\.12%\), demonstrating that reinforcement learning enables capability boundary internalization across difficulties\.

#### Efficiency\.

RFT produces the longest responses \(9133 tokens at Level 8\), exhibiting verbose failure from prolonged futile reasoning\. On the contrary, CaRL achieves nearly 33% token reduction by aborting futile attempts early through appropriate refusal, particularly critical in greater difficulties where computational waste compounds reliability costs\. This efficiency gain aligns with our discussion in §[3\.4](https://arxiv.org/html/2607.29211#S3.SS4)\.

### 5\.6Performance on General Tasks and Parameter Analysis

Table 3:Performance on General Tasks on Qwen3\-8B\.To demonstrate that CaRL does not compromise general reasoning capability, we evaluate the trained model on out\-of\-domain benchmarks: AIME 2024Art of Problem Solving \([n\.d\.](https://arxiv.org/html/2607.29211#bib.bib1)\)and GPQARein et al\. \([2024](https://arxiv.org/html/2607.29211#bib.bib26)\)\. Table[3](https://arxiv.org/html/2607.29211#S5.T3)shows that CaRL maintains competitive accuracy with marginal differences \(<<2%\) compared to the vanilla baseline\. Notably, CaRL achieves substantial improvements in reliability and computational efficiency, demonstrating that capability\-aligned training preserves general reasoning while improving reliability and reducing futile reasoning overhead\.

Moreover, by analyzing parameter changes before and after CaRL training on Qwen3\-8B, we find that blocks with a mean absolute difference\>10−5\>10^\{\-5\}are primarily concentrated in layers 31\-35\. This phenomenon aligns with recent findings that parameter changes in later layers are typically associated with task\-specific adaptationXin et al\. \([2025](https://arxiv.org/html/2607.29211#bib.bib28)\); Chen et al\. \([2023](https://arxiv.org/html/2607.29211#bib.bib6)\)\. This localized adaptation pattern suggests that CaRL modifies decision\-making mechanisms without disrupting the model’s foundational reasoning capabilities\.

### 5\.7Case Study

Figure[8](https://arxiv.org/html/2607.29211#S5.F8)illustrates contrasting behaviors on a countdown task\(level=8\)\.

The baseline \(left\) exhibits recursive hallucination: despite internally detecting errors, it repeatedly outputs "Final Correct Answer" followed by invalid expressions, revealing a disconnect between error detection and generation control\.

In contrast, CaRL \(right\) demonstrates capability awareness\. After exploring approaches and recognizing the closest achievable value 89 falls short of the target number 151, it appropriately refuses with explanation: "Sorry, I can’t solve this problem\. Here is my attempt…"\. This confirms that CaRL enables the model to assess solvability and abort futile attempts\.

## 6Conclusion

We identify and addressfutile reasoning, where models generate incorrect reasoning on beyond\-capability tasks rather than refusing\. Through systematic analysis, we reveal universal capability overreach, dominant specious reasoning patterns, and the insufficiency of prompt engineering\. Our proposed CaRL framework achieves a substantial reduction in futile reasoning through explicit capability\-aligned training while preserving task performance\.

## Limitations

To cleanly isolate capability boundaries from knowledge boundaries, we focus on Countdown and Sudoku, which are pure algorithmic reasoning tasks that require no external knowledge\. This ensures observed failures reflect reasoning limitations rather than missing factual information\. However, real\-world applications often involve knowledge\-intensive tasks where these boundaries intertwine\. In the future, we will extend CaRL to domains like open\-domain QA or mathematical reasoning to validate whether the refusal mechanism generalizes across all scenarios\.

## Acknowledgements

We sincerely thank the reviewers for their insightful comments and valuable suggestions\. This work was supported by the National Key R&D Program of China \(2024YFC3308000\), the Natural Science Foundation of China \(No\. 62476265, 62306303\)\.

## References

- Art of Problem Solving \(n\.d\.\)Art of Problem Solving\. n\.d\.AIME problems and solutions\.[https://artofproblemsolving\.com/wiki/index\.php/AIME\_Problems\_and\_Solutions](https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions)\.Accessed: 2026\-01\-06\.
- Bai et al\. \(2022\)Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others\. 2022\.Training a helpful and harmless assistant with reinforcement learning from human feedback\.*arXiv preprint arXiv:2204\.05862*\.
- Bowman et al\. \(2022\)Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilė Lukošiūtė, Amanda Askell, Andy Jones, Anna Chen, and 1 others\. 2022\.Measuring progress on scalable oversight for large language models\.*arXiv preprint arXiv:2211\.03540*\.
- Chen et al\. \(2025a\)Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie\. 2025a\.Sft or rl? an early investigation into training r1\-like reasoning large vision\-language models\.*arXiv preprint arXiv:2504\.11468*\.
- Chen et al\. \(2025b\)Liang Chen, Xueting Han, Li Shen, Jing Bai, and Kam\-Fai Wong\. 2025b\.Beyond two\-stage training: Cooperative sft and rl for llm reasoning\.*arXiv preprint arXiv:2509\.06948*\.
- Chen et al\. \(2023\)Nuo Chen, Ning Wu, Shining Liang, Ming Gong, Linjun Shou, Dongmei Zhang, and Jia Li\. 2023\.Is bigger and deeper always better? probing llama across scales and layers\.*arXiv preprint arXiv:2312\.04333*\.
- Chen et al\. \(2024\)Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, and 1 others\. 2024\.Do not think that much for 2\+ 3=? on the overthinking of o1\-like llms\.*arXiv preprint arXiv:2412\.21187*\.
- Cheng et al\. \(2024\)Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang, Zhangyue Yin, Shimin Li, Linyang Li, Zhengfu He, Kai Chen, and Xipeng Qiu\. 2024\.Can ai assistants know what they don’t know?*arXiv preprint arXiv:2401\.13275*\.
- Chu et al\. \(2025\)Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma\. 2025\.Sft memorizes, rl generalizes: A comparative study of foundation model post\-training\.*arXiv preprint arXiv:2501\.17161*\.
- Davis et al\. \(2011\)Richard A Davis, Keh\-Shin Lii, and Dimitris N Politis\. 2011\.Remarks on some nonparametric estimates of a density function\.In*Selected Works of Murray Rosenblatt*, pages 95–100\. Springer\.
- Guo et al\. \(2025\)Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others\. 2025\.Deepseek\-r1: Incentivizing reasoning capability in llms via reinforcement learning\.*arXiv preprint arXiv:2501\.12948*\.
- Huang and Chang \(2023\)Jie Huang and Kevin Chen\-Chuan Chang\. 2023\.Towards reasoning in large language models: A survey\.In*Findings of the association for computational linguistics: ACL 2023*, pages 1049–1065\.
- Huang et al\. \(2025a\)Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and 1 others\. 2025a\.A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions\.*ACM Transactions on Information Systems*, 43\(2\):1–55\.
- Huang et al\. \(2025b\)Yue Huang, Chujie Gao, Yujun Zhou, Kehan Guo, Xiangqi Wang, Or Cohen\-Sasson, Max Lamparth, and Xiangliang Zhang\. 2025b\.Position: We need an adaptive interpretation of helpful, honest, and harmless principles\.*arXiv preprint arXiv:2502\.06059*\.
- Joglekar et al\. \(2025\)Manas Joglekar, Jeremy Chen, Gabriel Wu, Jason Yosinski, Jasmine Wang, Boaz Barak, and Amelia Glaese\. 2025\.Training llms for honesty via confessions\.*arXiv preprint arXiv:2512\.08093*\.
- Kadavath et al\. \(2022\)Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield\-Dodds, Nova DasSarma, Eli Tran\-Johnson, and 1 others\. 2022\.Language models \(mostly\) know what they know\.*arXiv preprint arXiv:2207\.05221*\.
- Kashyap et al\. \(2025\)Gautam Siddharth Kashyap, Mark Dras, and Usman Naseem\. 2025\.Too helpful, too harmless, too honest or just right?In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 29711–29722\.
- Lin et al\. \(2022\)Stephanie Lin, Jacob Hilton, and Owain Evans\. 2022\.Teaching models to express their uncertainty in words\.*arXiv preprint arXiv:2205\.14334*\.
- Liu et al\. \(2024\)Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others\. 2024\.Deepseek\-v3 technical report\.*arXiv preprint arXiv:2412\.19437*\.
- Liu et al\. \(2025\)Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, and 1 others\. 2025\.Deepseek\-v3\. 2: Pushing the frontier of open large language models\.*arXiv preprint arXiv:2512\.02556*\.
- Liu et al\. \(2023\)Yang Liu, Yuanshun Yao, Jean\-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li\. 2023\.Trustworthy llms: a survey and guideline for evaluating large language models’ alignment\.*arXiv preprint arXiv:2308\.05374*\.
- Minaee et al\. \(2024\)Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao\. 2024\.Large language models: A survey\.*arXiv preprint arXiv:2402\.06196*\.
- OpenAI \(2025\)OpenAI\. 2025\.[gpt\-oss\-120b gpt\-oss\-20b model card](https://arxiv.org/abs/2508.10925)\.*Preprint*, arXiv:2508\.10925\.
- Parzen \(1962\)Emanuel Parzen\. 1962\.On estimation of a probability density function and mode\.*The annals of mathematical statistics*, 33\(3\):1065–1076\.
- Perez et al\. \(2023\)Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, and 1 others\. 2023\.Discovering language model behaviors with model\-written evaluations\.In*Findings of the association for computational linguistics: ACL 2023*, pages 13387–13434\.
- Rein et al\. \(2024\)David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman\. 2024\.Gpqa: A graduate\-level google\-proof q&a benchmark\.In*First Conference on Language Modeling*\.
- Sahoo et al\. \(2024\)Pranab Sahoo, Prabhash Meharia, Akash Ghosh, Sriparna Saha, Vinija Jain, and Aman Chadha\. 2024\.A comprehensive survey of hallucination in large language, image, video and audio foundation models\.*arXiv preprint arXiv:2405\.09589*\.
- Xin et al\. \(2025\)Chunlei Xin, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Xuanang Chen, Xinyan Guan, Yaojie Lu, Hongyu Lin, Xianpei Han, and Le Sun\. 2025\.[Sparse latents steer retrieval\-augmented generation](https://doi.org/10.18653/v1/2025.acl-long.228)\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 4547–4562, Vienna, Austria\. Association for Computational Linguistics\.
- Xu et al\. \(2025\)Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, and 1 others\. 2025\.Towards large reasoning models: A survey of reinforced reasoning with large language models\.*arXiv preprint arXiv:2501\.09686*\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others\. 2025\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*\.
- Yeo et al\. \(2025\)Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue\. 2025\.Demystifying long chain\-of\-thought reasoning in llms\.*arXiv preprint arXiv:2502\.03373*\.
- Zhang et al\. \(2025\)Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou\. 2025\.On\-policy rl meets off\-policy experts: Harmonizing supervised fine\-tuning and reinforcement learning via dynamic weighting\.*arXiv preprint arXiv:2508\.11408*\.
- Zhang et al\. \(2023\)Yifan Zhang, Jingqin Yang, Yang Yuan, and Andrew Chi\-Chih Yao\. 2023\.Cumulative reasoning with large language models\.*arXiv preprint arXiv:2308\.04371*\.
- Zhao et al\. \(2023\)Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, and 1 others\. 2023\.A survey of large language models\.*arXiv preprint arXiv:2303\.18223*, 1\(2\)\.

## Appendix ADataset Examples

### A\.1Countdown Task

The Countdown task requires constructing a target value using a given set of integers and basic arithmetic operations \(\+,−,×,÷\+,\-,\\times,\\div\)\. Each number can be used at most once\.

Countdown ExampleNumbers:\[3, 5, 10, 25\]Target:151Golden Answer:\(3\+10/5\)×25=125\(3\+10/5\)\\times 25=125

### A\.2Sudoku Task

The Sudoku task requires filling a 9×9 grid such that each row, column, and 3×3 subgrid contains all digits 1\-9 without repetition\.

Sudoku ExamplePuzzle:\(0 represents empty cells\)``` 5 3 0 | 0 7 0 | 0 0 0 6 0 0 | 1 9 5 | 0 0 0 0 9 8 | 0 0 0 | 0 6 0 ------+-------+------ 8 0 0 | 0 6 0 | 0 0 3 4 0 0 | 8 0 3 | 0 0 1 7 0 0 | 0 2 0 | 0 0 6 ------+-------+------ 0 6 0 | 0 0 0 | 2 8 0 0 0 0 | 4 1 9 | 0 0 5 0 0 0 | 0 8 0 | 0 7 9 ``` Golden Answer:``` 5 3 4 | 6 7 8 | 9 1 2 6 7 2 | 1 9 5 | 3 4 8 1 9 8 | 3 4 2 | 5 6 7 ------+-------+------ 8 5 9 | 7 6 1 | 4 2 3 4 2 6 | 8 5 3 | 7 9 1 7 1 3 | 9 2 4 | 8 5 6 ------+-------+------ 9 6 1 | 5 3 7 | 2 8 4 2 8 7 | 4 1 9 | 6 3 5 3 4 5 | 2 8 6 | 1 7 9 ```

## Appendix BFutile Reasoning Example

### B\.1Degenerate Repetition

The example is given numbers: \[37, 97, 3, 51, 3, 66, 38, 3\], with a target of 275\. Due to space constraints, we present excerpts from a representative failure case: the initial reasoning phase \(Figure[9](https://arxiv.org/html/2607.29211#A2.F9)\) and the final output \(Figure[10](https://arxiv.org/html/2607.29211#A2.F10)\)\. The model begins with systematic exploration, attempting multiple combinations\. However, after exhausting valid approaches without finding a solution, instead of acknowledging ignorance, the model resorts to exhaustive guessing—repeatedly outputting incorrect expressions while claiming they satisfy the constraints\.

![Refer to caption](https://arxiv.org/html/2607.29211v1/latex/figure/repeat1.png)Figure 9:Initial Reasoning Phase\.The model systematically explores combinations \(e\.g\.,97\+66=16397\+66=163,3×51=1533\\times 51=153\), attempting to construct the target value 275\. Early attempts show valid mathematical reasoning but fail to reach the exact target\.![Refer to caption](https://arxiv.org/html/2607.29211v1/latex/figure/repeat2.png)Figure 10:Final Output After Degenerate Repetition\.After 50\+ failed attempts, the model outputs\(97\+66\+51\+38\+37\)−\(3\+3\+3\)=280\(97\+66\+51\+38\+37\)\-\(3\+3\+3\)=280while incorrectly asserting it equals 275\. This exemplifies hallucination through exhaustive guessing rather than appropriate refusal\.
### B\.2Endless Generation

We illustrate this failure mode using a Countdown task with numbers \[58, 89, 15, 47, 37, 84, 86, 70\] and target 168\. The endless output is shown below:

Endless Generation ExampleWe are given numbers: 58, 89, 15, 47, 37, 84, 86, 70\. We need to use each exactly once in a single arithmetic expression using only \+, \-, \*, / and parentheses to get exactly 168\.We have 8 numbers\. The target is 168\. We need to combine them with operations\. Possibly we can group them into pairs or combine in some order\. Since 168 is not too large compared to some numbers, we might need to subtract or divide to reduce\.…I’m considering that maybe the solution is not coming easily\. Perhaps I should try to work backwards from 168\. Write 168 as a combination of some of the numbers with operations, then see if the remaining numbers can be used to adjust\.For example, 168 = 84\*2\. So if we can get 84 and 2 from separate groups using all numbers, but we already tried that\.168 = 86 \+ 82\. 82 can be 47\+35, but 35 not\.168 = 70 \+ 98\. 98 = 58\+40, no\.168 = 89 \+ 79\. 79 not\.168 = 58 \+ 110\. 110 = 70\+40, no\.168 = 37 \+ 131\. no\.168 = 15 \+ 153\. 153 = 86\+67, no\.168 = 47 \+ 121\. 121 = 89\+32, no\.168 = 84 \+ 84\. But only one 84\.168 = 86 \+ 84 \- 2\. We tried\.168 = 70 \+ 98, 98 = 84\+14, but 14 not\.168 = 89 \+ 84 \- 5\. 5 from 15/3, but 3 not\[ hit 32k tokens limitation with a sudden stop\]

### B\.3Specious Reasoning

We illustrate this failure mode using a Countdown task with numbers \[8, 12, 74, 88, 74, 67, 10, 6\] and target 17\. As shown in Figure[11](https://arxiv.org/html/2607.29211#A2.F11), the model constructs an expression using the number “6”twice, violating the constraint that each number may be used at most once\. However, during self\-verification, the model falsely claims that each number was used only once and asserts the solution is correct\. This exemplifiesspecious reasoning\. The derivation appears logically sound on the surface but contains a subtle yet critical constraint violation that the model fails to detect\.

Specious Example<think\> Okay, let’s tackle this problem\. I need to use each of the numbers \[8, 12, 74, 88, 74, 67, 10, 6\] exactly once with basic operations to get 17\. Hmm, that’s eight numbers\. Let me think about how to combine them\.First, maybe I can look for pairs or groups that can simplify\. Since there are so many numbers, perhaps combining some through multiplication or division might help reduce the total\. Let me list the numbers again: 8, 12, 74, 88, 74, 67, 10, 6\. Wait, there are two 74s\. That might be useful or maybe a red herring\. Let me see\.\[Sereval Attempts\] …For example, if I have \(67 \- \(8\*6 \+ \(12/6\)\)\) \+ \(\(74 \- 74\) \* \(88 \- 10\)\) ?Let me check\. 8\*6 = 48, 12/6 = 2, so 48 \+ 2 = 50\. 67 \- 50 = 17\. Then 74 \-74 = 0, 88 \-10 = 78\. 0 \* 78 = 0\. So total is 17 \+ 0 = 17\. But wait, I have used all numbers? Let’s check:Used numbers in first part: 67, 8, 6, 12\. Then in the second part: 74, 74, 88, 10\. Yes\! All numbers are used once\. So the expression would be:\(67 \- \(8 \* 6 \+ \(12 / 6\)\)\) \+ \(\(74 \- 74\) \* \(88 \- 10\)\)Let me check if that equals 17\.First part: 8\*6 = 48\. 12/6 = 2\. 48 \+ 2 = 50\. 67 \- 50 = 17\.Second part: 74 \-74 = 0\. 88 \-10 = 78\. 0 \* 78 = 0\.Total is 17 \+ 0 = 17\. Perfect\!Now, let me check if all numbers are used exactly once:Numbers used: 67, 8, 6, 12, 74, 74, 88, 10\. Yes\! All eight numbers are used once\.Now, I need to write this as a single arithmetic expression\. Let me make sure the parentheses are correct and the operations are allowed\.The expression is:\(67 \- \(8 \* 6 \+ \(12 / 6\)\)\) \+ \(\(74 \- 74\) \* \(88 \- 10\)\)But I need to make sure that all numbers are used once and that the operations are correct\. Let me check:\- 67 is used once\. \- 8 is used once\. \- 6 is used once\. \- 12 is used once\. \- 74 is used twice \(the two 74s\)\. \- 88 is used once\. \- 10 is used once\.Yes, all numbers are used exactly once\.Now, let me write this expression without any extra spaces, using only the allowed symbols\. Also, need to ensure that the operations are properly ordered\.The expression would be:\(67 \- \(8 \* 6 \+ \(12 / 6\)\)\) \+ \(\(74 \- 74\) \* \(88 \- 10\)\)But I need to check if there are any issues with the order of operations\. Let me verify:Inside the first part: 8 \* 6 is 48, 12 / 6 is 2, then adding them gives 50\. 67 \- 50 = 17\.In the second part: 74 \- 74 = 0, 88 \-10 = 78, 0 \* 78 = 0\.Adding them gives 17 \+ 0 = 17\. Correct\.So this expression works\. Now, I need to write it in the required format, wrapped in \.But let me check for any possible errors\. For example, are there any parentheses that need to be adjusted? Let me see:The expression is:\(67 \- \(8 \* 6 \+ \(12 / 6\)\)\) \+ \(\(74 \- 74\) \* \(88 \- 10\)\)Yes, parentheses are correctly placed to ensure the operations are performed in the right order\.So the final answer should be this expression\. Let me write it without any extra spaces, as per the rules\. Also, ensure that all numbers are used once and that the operations are correct\.So the boxed expression would be:\(67−\(8∗6\+\(12/6\)\)\)\+\(\(74−74\)∗\(88−10\)\)Yes, this should be correct\. </think\>\(67−\(8∗6\+\(12/6\)\)\)\+\(\(74−74\)∗\(88−10\)\)Figure 11:Supicous Reasoning Example

## Appendix CImplementation Details

### C\.1Training Details

We adopt Group Relative Policy Optimization \(GRPO\)Guo et al\. \([2025](https://arxiv.org/html/2607.29211#bib.bib11)\)as our underlying reinforcement learning algorithm\. For RL \(including Standard RL and CaRL variants\), we use a learning rate of1×10−61\\times 10^\{\-6\}and train for 3 epochs\. During exploration, we employ a group rollout size of 16 with a sampling temperature of1\.01\.0, maintaining a training batch size of 32\. For the RFT baseline, we fine\-tune the model for 3 epochs with a learning rate of8×10−58\\times 10^\{\-5\}and a global batch size of 32\. For all the evaluations, we conduct inference with a temperature of0\.60\.6\. To ensure robustness, we sample 8 responses for each query and report the averaged performance metrics\.

### C\.2Evaluation Details

For all evaluations on Countdown and Sudoku tasks, we sample 8 responses per question and compute the average score\. For general capability evaluation \(§[5\.6](https://arxiv.org/html/2607.29211#S5.SS6)\), we sample 8 responses per question for AIME 2024 and 4 responses per question for GPQA\.