Capability Self-Assessment: Teaching LLMs to Know Their Limits

# Teaching LLMs to Know Their Limits
Source: [https://arxiv.org/html/2606.00251](https://arxiv.org/html/2606.00251)
## Capability Self\-Assessment: Teaching LLMs to Know Their Limits

Haoyan Yang1,Reza Shirkavand211footnotemark:1,Yukai Jin3, Jiawei Zhou1,Shangqian Gao3222We train for 1 epoch on Math and 3 epochs on Science\. The only exception is Qwen3\-0\.6B, for which longer training causes the model to collapse into always selecting DELEGATE; therefore we apply early stopping and train it for 60 steps\.,Heng Huang2222We train for 1 epoch on Math and 3 epochs on Science\. The only exception is Qwen3\-0\.6B, for which longer training causes the model to collapse into always selecting DELEGATE; therefore we apply early stopping and train it for 60 steps\. 1Stony Brook University2University of Maryland3Florida State UniversityEqual Contributions\.Correspondence to: Jiawei Zhou \(jiawei\.zhou\.1@stonybrook\.edu\), Shangqian Gao \(sgao@cs\.fsu\.edu\), Heng Huang \(heng@cs\.umd\.edu\)\.

###### Abstract

The ability to recognize one’s own limitations and decide whether to solve a problem or delegate is fundamental for reliable intelligent systems\. Yet we show that modern large language models systematically lack this ability: across diverse model families and scales, they overestimate their competence and attempt queries they cannot solve\. We refer to this ability as Capability Self\-Assessment \(CSA\) and formulate it as a policy\-learning problem, aiming to improve self\-assessment while preserving the model’s original capabilities\. Our results show that reinforcement learning teaches CSA effectively, significantly outperforming supervised fine\-tuning while preserving original capabilities\. In contrast, supervised fine\-tuning severely degrades the capabilities the model is meant to assess\. Moreover, learned self\-assessment behavior generalizes well out of distribution, suggesting that CSA is a transferable model trait\. Finally, CSA is practically useful: it improves local–cloud decision making at inference time and provides a signal for targeted data selection during training\.111Code is available at[https://github\.com/Joyyang158/llm\-csa](https://github.com/Joyyang158/llm-csa)\.

## 1Introduction

An ideal intelligent system not only solves problems well, but also recognizes its limits\. It asks for help when a problem exceeds its ability or uses failure to identify what it should learn next, as humans do[madras2018predictresponsiblyimprovingfairness](https://arxiv.org/html/2606.00251#bib.bib43)\. However, large language models \(LLMs\) are trained to produce answers, not to decide whether they should answer\. We empirically find systematic capability overestimation across diverse model families and scales: LLMs choose to answer even on queries they cannot solve reliably\. The consequences are concrete\. Models answer beyond their capability boundary, producing hallucinations[farquhar2024detecting\-hallucinations\-entropy](https://arxiv.org/html/2606.00251#bib.bib16)and wasted compute[shirkavand2025costaware](https://arxiv.org/html/2606.00251#bib.bib55)\. This decision of whether to answer is often as important as the answer itself: before attempting a query, a model should assess whether the problem lies within its own competence or whether it should delegate to a stronger model or a human\.

We define Capability Self\-Assessment \(CSA\) as a model’s ability to judge whether a query falls within its solvable set, and use it as our central lens\. Prior work has studied related problems through calibration[kadavath2022llms\-mostly\-know](https://arxiv.org/html/2606.00251#bib.bib34), confidence estimation[yin2023do\-llms\-know\-what\-they\-dont\-know](https://arxiv.org/html/2606.00251#bib.bib72);[kapoor2024llms\-must\-be\-taugh](https://arxiv.org/html/2606.00251#bib.bib35), instruction tuning[zhang2024r](https://arxiv.org/html/2606.00251#bib.bib74), and abstention[wen2025abstention\-survey](https://arxiv.org/html/2606.00251#bib.bib64)\. However, these lines of work do not treat self\-assessment as an independent capability\-learning problem\. Calibration and confidence estimation mainly extract scores tied to model predictions, while instruction tuning and abstention typically mix solve\-or\-defer behavior into specific tasks\. To isolate capability assessment from execution and directly test whether a model can learn its own capability boundary, we formulate an actionable CSA task in a simple format: a binary choice betweenSelf\-SOLVE, attempting the query with the model itself, andDELEGATE, handing off the query to a stronger model\.

We view this task as a*policy learning*problem, not as an ordinary supervised classification\. The goal is not to train a separate judge, nor merely to elicit a calibrated confidence score\. Rather, we want the model itself to acquire CSA as a behavioral capability while retaining the underlying abilities whose limits it must assess\. This makes reinforcement learning \(RL\) particularly natural\. The object of interest is an action, not a label in isolation, and correctness is outcome\-based\. ASelf\-SOLVEdecision is correct if the model can in fact solve the query, andDELEGATEis correct otherwise\. This gives a clean verifiable reward[lambert2024tulu](https://arxiv.org/html/2606.00251#bib.bib39);[shao2024deepseekmath](https://arxiv.org/html/2606.00251#bib.bib54);[guo2025deepseek\-r1](https://arxiv.org/html/2606.00251#bib.bib28), making CSA a natural instance of policy optimization with outcome\-based feedback\. By contrast, Supervised Finetuning \(SFT\) on decision labels or rationales can teach the model to imitate the desired outputs without preserving the competence those outputs are meant to reflect\.

Across a broad range of model families, scales, and domains, we show that CSA is a teachable trait\. RL substantially improves this capability, outperforming supervised alternatives while preserving the model’s original problem\-solving ability\. The resulting behavior also transfers across domains, suggesting that CSA is not just a domain\-specific heuristic, but a more general and reusable capability\-assessment skill\.

We demonstrate the broader view of CSA as a learnable model capability in two concrete applications\. At inference time, learned CSA improves local–cloud routing by identifying which queries should be routed to a stronger model, yielding a better cost–accuracy tradeoff\. Moreover, becauseDELEGATEsamples lie near or beyond the model’s current capability boundary, CSA\-based signal can be used during training for targeted data selection\. In this way, learned CSA can play a dual role analogous to competence assessment in effective problem solving, guiding when to act or delegate at inference time, and guiding what to learn next during training\.

In summary, our contributions are:

1. 1\.We formalize Capability Self\-Assessment \(CSA\) as a policy learning problem, where the model decides betweenSelf\-SOLVEandDELEGATE, motivated by the severe capability overestimation we observe in current LLMs\.
2. 2\.We show that CSA is a teachable trait: unlike SFT, RL improves capability assessment across model scales and domains without degrading task ability\. The learned policy further generalizes to out\-of\-distribution settings\.
3. 3\.We demonstrate that learned CSA is useful in practice for both local–cloud routing and capability\-aware data selection\.

## 2Related Work and Motivation

### 2\.1Related Work

Self\-evaluation, calibration, and selective prediction and abstention\.A central line of work studies whether language models can assess the correctness of their own outputs\. Kadavath et al\.[kadavath2022llms\-mostly\-know](https://arxiv.org/html/2606.00251#bib.bib34)show that LLMs can produce useful self\-evaluation signals such asP​\(True\)P\(\\mathrm\{True\}\)andP​\(IK\)P\(\\mathrm\{IK\}\), motivating a view of self\-knowledge as a calibration problem[gneiting2007calibration1](https://arxiv.org/html/2606.00251#bib.bib20);[guo2017calibration2](https://arxiv.org/html/2606.00251#bib.bib27)\. Follow\-up work studies unanswerable questions and uncertainty\-aware fine\-tuning, showing that uncertainty estimation can improve with explicit training rather than prompting alone[yin2023do\-llms\-know\-what\-they\-dont\-know](https://arxiv.org/html/2606.00251#bib.bib72);[kapoor2024llms\-must\-be\-taugh](https://arxiv.org/html/2606.00251#bib.bib35)\. Closely related work on selective prediction and abstention asks when a model should answer versus refrain from answering, often through confidence thresholding or risk–coverage tradeoffs[chow2003optimum](https://arxiv.org/html/2606.00251#bib.bib12);[el2010foundations](https://arxiv.org/html/2606.00251#bib.bib15);[bartlett2008classification](https://arxiv.org/html/2606.00251#bib.bib5);[geifman2019selectivenet](https://arxiv.org/html/2606.00251#bib.bib18);[xin2021art\-of\-abstetion](https://arxiv.org/html/2606.00251#bib.bib66);[gu2023evaluation](https://arxiv.org/html/2606.00251#bib.bib25);[wen2025abstention\-survey](https://arxiv.org/html/2606.00251#bib.bib64)\. Together, these lines of work largely treat self\-knowledge as a*scalar*confidence estimation problem\.

Uncertainty signals and RL\-trained decision behavior\.A related literature studies uncertainty and hallucination detection through self\-consistency, semantic uncertainty, or hidden\-state probes[manakul2023selfcheckgpt](https://arxiv.org/html/2606.00251#bib.bib44);[farquhar2024detecting\-hallucinations\-entropy](https://arxiv.org/html/2606.00251#bib.bib16);[kossen2024semantic\-entropy](https://arxiv.org/html/2606.00251#bib.bib36);[azaria2023internal](https://arxiv.org/html/2606.00251#bib.bib3);[slobodkin2023curious](https://arxiv.org/html/2606.00251#bib.bib56);[han2025simple\-factuality\-probes](https://arxiv.org/html/2606.00251#bib.bib30);[cencerrado2025no\-answer\-needed](https://arxiv.org/html/2606.00251#bib.bib8)\. These works suggest that models contain useful signals about their own unreliability, but they are largely aimed at*post\-hoc detection*\. On the optimization side, DeepSeek\-R1[guo2025deepseek\-r1](https://arxiv.org/html/2606.00251#bib.bib28)shows that RL can induce useful behavioral traits such as self\-reflection and verification, while recent RL\-based help\-seeking work studies when models should rely on external tools[feng2025retool](https://arxiv.org/html/2606.00251#bib.bib17);[gul2025mash](https://arxiv.org/html/2606.00251#bib.bib26)\.

Positioning our contribution\.Rather than treating self\-assessment as a*scalar*confidence estimate or a*detection*method, we study whether the model itself can learn an explicitSelf\-Solve/Delegate*policy*\. The goal is to instill CSA as an internal capability of the model, not to train a separate router or judge\. This makes policy learning more appropriate than supervised classification: a confidence score or delegation label specifies only the desired output, while RL lets the model develop its own internal strategy for capability assessment\. Our experiments confirm this distinction is consequential \(Section[4\.2](https://arxiv.org/html/2606.00251#S4.SS2)\)\. A full discussion of related work is provided in Appendix[A](https://arxiv.org/html/2606.00251#A1)\.

### 2\.2Empirical Study: Current Models Lack CSA

![Refer to caption](https://arxiv.org/html/2606.00251v1/Figures/overconfidence_across_models.png)Figure 1:Current models lack CSA, across model families and sizes, evaluated on a combined math test set\.▼\\blacktriangledownmarks each model’s*self\-solve rate*and▲\\blacktrianglemarks its*actual accuracy*\. The shaded bar spans the gap between the two, with a larger gap indicating weaker CSA\.Similar to prior work showing that LLMs overestimate their likelihood of success or answer correctness[barkan2026do](https://arxiv.org/html/2606.00251#bib.bib4);[xiong2023can\-llm\-express\-uncertainty](https://arxiv.org/html/2606.00251#bib.bib67);[sun2025large](https://arxiv.org/html/2606.00251#bib.bib58);[chhikara2025mind](https://arxiv.org/html/2606.00251#bib.bib11), we empirically find that most current models lack reliable CSA\. Our evaluation spans the Qwen3[yang2025qwen3technicalreport](https://arxiv.org/html/2606.00251#bib.bib69), Gemma3[gemmateam2025gemma3technicalreport](https://arxiv.org/html/2606.00251#bib.bib59), Llama3[grattafiori2024llama3herdmodels](https://arxiv.org/html/2606.00251#bib.bib22), OLMo2[olmo20252olmo2furious](https://arxiv.org/html/2606.00251#bib.bib47), GPT\-5[openai2025gpt5](https://arxiv.org/html/2606.00251#bib.bib48), and Gemini\-3\.1[google2026gemini31](https://arxiv.org/html/2606.00251#bib.bib21)families, covering both open\-weight and frontier closed\-weight models\. For each model, we use queries from a combined math test set and ask it to choose betweenSelf\-SOLVEandDELEGATE\. We record the*self\-solve rate*, the percentage of samples on which it choosesSelf\-SOLVEand compare it with*actual accuracy*, the percentage it answers correctly when solving\.

As shown in Figure[1](https://arxiv.org/html/2606.00251#S2.F1), comparing these two metrics reveals that across all evaluated models the self\-solve rate is substantially higher than the actual accuracy\. This indicates that current models systematically over\-estimate their own capability\. See Appendices[B\.1](https://arxiv.org/html/2606.00251#A2.SS1),[D\.2](https://arxiv.org/html/2606.00251#A4.SS2), and[D\.3](https://arxiv.org/html/2606.00251#A4.SS3)for the prompt template, Math test set details, and inference setup\.

## 3CSA through Training

Motivated by the finding that current models lack reliable CSA \(Section[2\.2](https://arxiv.org/html/2606.00251#S2.SS2)\), in this section we investigate whether*CSA can be acquired through training*\. Figure[2](https://arxiv.org/html/2606.00251#S3.F2)illustrates our approach\.

![Refer to caption](https://arxiv.org/html/2606.00251v1/x1.png)Figure 2:Overview of our framework for instilling CSA into a language model\.\(1\) CSA Label Construction: the model attempts each queryKKtimes; predictions are compared with ground truth and aggregated to assign labelsyi∈\{Self\-Solve,Delegate\}y\_\{i\}\\in\\\{\\textsc\{Self\-Solve\},\\textsc\{Delegate\}\\\}\.\(2\) CSA Training Strategies: we explore four approaches:*\(a\)*SFTlabel\{\}\_\{\\text\{label\}\}uses bare labels,*\(b\)*SFTself\{\}\_\{\\text\{self\}\}uses self\-generated rationales,*\(c\)*SFTteacher\{\}\_\{\\text\{teacher\}\}uses teacher rationales, and*\(d\)*RLVR is a two\-stage GRPO with Diversity\-Filtered Warm\-up \(DFW\) followed by full GRPO training\.\(3\) CSA Inference & Evaluation: the trained modelℳ′\\mathcal\{M\}^\{\\prime\}decides whether to self\-solve or delegate on each new query; for evaluation,ℳ′\\mathcal\{M\}^\{\\prime\}is re\-probed to obtainy~i\\tilde\{y\}\_\{i\}, and CSA performance is measured by comparingy^i\\hat\{y\}\_\{i\}againsty~i\\tilde\{y\}\_\{i\}\.### 3\.1CSA Label Construction

Teaching a model CSA requires ground\-truth labels, which we construct by probing the model itself\. Given a dataset𝒟=\{\(qi,ai∗\)\}i=1N\\mathcal\{D\}=\\\{\(q\_\{i\},a\_\{i\}^\{\*\}\)\\\}\_\{i=1\}^\{N\}of queries and answers, we run modelℳ\\mathcal\{M\}on each queryKKtimes to obtain predictions\{a^i\(k\)\}k=1K\\\{\\hat\{a\}\_\{i\}^\{\(k\)\}\\\}\_\{k=1\}^\{K\}\. SamplingKKtimes reduces decoding noise, so the label reflects the model’s underlying capability\. Then assign labels by aggregating their comparisons to ground truth:

yi=\{Self\-Solve,if​g​\(\{𝟙​\[a^i\(k\)=ai∗\]\}k=1K\)=1,Delegate,otherwise,y\_\{i\}=\\begin\{cases\}\\textsc\{Self\-Solve\},&\\text\{if \}g\\big\(\\\{\\mathbbm\{1\}\[\\hat\{a\}\_\{i\}^\{\(k\)\}=a\_\{i\}^\{\*\}\]\\\}\_\{k=1\}^\{K\}\\big\)=1,\\\\ \\textsc\{Delegate\},&\\text\{otherwise,\}\\end\{cases\}\(1\)whereg​\(⋅\)g\(\\cdot\)is an aggregation function \(e\.g\., pass@KKor majority vote\)\. The resulting labels capture which queries the model can and cannot solve, and are shared across all training strategies\.

Importantly, these labels are constructed before training and remain fixed throughout\. Their validity therefore rests on a key assumption: that acquiring CSA does not degrade the model’s underlying problem\-solving ability, so that the trained model’s predictions still match the one captured by\{yi\}\\\{y\_\{i\}\\\}\. This assumption also reflects how we view CSA itself: as a capability layered on top of the model’s existing competence, rather than a reduction of the model to a binary classifier\.

### 3\.2CSA Training Strategies

We introduce four strategies for training a model to acquire CSA\. All four take the same input: a CSA prompt templatepp\(shown in Appendix[B\.1](https://arxiv.org/html/2606.00251#A2.SS1)\) concatenated with the queryqiq\_\{i\}, formingxi=\[p;qi\]x\_\{i\}=\[p;q\_\{i\}\]\. Strategies \(a\)–\(c\) are based onSFT, Strategy \(d\) adoptsReinforcement Learning with Verifiable Rewards \(RLVR\)approach\.

Strategies \(a\)–\(d\) are chosen to represent major post\-training techniques for LLMs, spanning from simple label\-based supervised fine\-tuning to chain\-of\-thought supervision and reinforcement learning with verifiable rewards\.

\(a\) SFTLabel[zhang2024r](https://arxiv.org/html/2606.00251#bib.bib74);[kapoor2024llms\-must\-be\-taugh](https://arxiv.org/html/2606.00251#bib.bib35)\.The target is the bare CSA label,oilabel=yio\_\{i\}^\{\\text\{label\}\}=y\_\{i\}\. The model is fine\-tuned to directly emit the decision without any intermediate reasoning\.

\(b\) SFTSelf[kadavath2022llms\-mostly\-know](https://arxiv.org/html/2606.00251#bib.bib34);[chaudhry2024finetuninglanguagemodelsemit](https://arxiv.org/html/2606.00251#bib.bib9)\.The target is a rationale followed by the label,oiself=\[riself;yi\]o\_\{i\}^\{\\text\{self\}\}=\[r\_\{i\}^\{\\text\{self\}\};y\_\{i\}\]\. The rationaleriselfr\_\{i\}^\{\\text\{self\}\}is generated by the training model itself: since equipping CSA is fundamentally a self\-assessment problem, having the model produce its own rationale aligns the supervision with how it natively reasons about its own capability\. To ensure correctness of the rationale, we condition the generation on the ground\-truth label,riself∼ℳ\(⋅∣xi,yi\)r\_\{i\}^\{\\text\{self\}\}\\sim\\mathcal\{M\}\(\\cdot\\mid x\_\{i\},y\_\{i\}\), so that the rationale justifies the correct decision\. Compared to SFTLabel, this can be viewed as a chain\-of\-thought form of CSA training\. The prompt template used for self\-rationale generation is provided in Appendix[B\.2](https://arxiv.org/html/2606.00251#A2.SS2)\.

\(c\) SFTTeacher[hager2025uncertaintydistillationteachinglanguage](https://arxiv.org/html/2606.00251#bib.bib29)\.The target has the same rationale\-then\-label form as SFTSelf:oiteacher=\[riteacher;yi\]o\_\{i\}^\{\\text\{teacher\}\}=\[r\_\{i\}^\{\\text\{teacher\}\};y\_\{i\}\], but the rationale is generated by a stronger teacher modelℳ∗\\mathcal\{M\}^\{\*\}rather than the training model\. This resembles teacher–student distillation: the student mimics the teacher’s reasoning style alongside the correct decision\. As in SFTSelf, generation is conditioned on the ground\-truth label,riteacher∼ℳ∗\(⋅∣xi,yi\)r\_\{i\}^\{\\text\{teacher\}\}\\sim\\mathcal\{M\}^\{\*\}\(\\cdot\\mid x\_\{i\},y\_\{i\}\)\. The teacher\-rationale prompt template is in Appendix[B\.3](https://arxiv.org/html/2606.00251#A2.SS3)\.

For the model training, all three SFT variants share the same supervised log\-likelihood objective:

ℒSFT\(θ\)=−∑i=1Nlogpθ\(oi\|xi\),oi∈\{oilabel,oiself,oiteacher\},\\mathcal\{L\}\_\{\\text\{SFT\}\}\(\\theta\)=\-\\sum\_\{i=1\}^\{N\}\\log p\_\{\\theta\}\\\!\\left\(o\_\{i\}\\,\\middle\|\\,x\_\{i\}\\right\),\\qquad o\_\{i\}\\in\\\{o\_\{i\}^\{\\text\{label\}\},\\,o\_\{i\}^\{\\text\{self\}\},\\,o\_\{i\}^\{\\text\{teacher\}\}\\\},\(2\)
\(d\) RLVR\.Unlike the SFT variants imitating fixed targets, RLVR lets the model discover effective CSA behavior through reward feedback using Group Relative Policy Optimization \(GRPO\)[shao2024deepseekmath](https://arxiv.org/html/2606.00251#bib.bib54)\. We use a binary verifiable reward:Ri\(g\)=\+1R\_\{i\}^\{\(g\)\}=\+1if the rollout’s predicted labely^i\(g\)\\hat\{y\}\_\{i\}^\{\(g\)\}in groupggmatches the ground\-truth CSA labelyiy\_\{i\}, and−1\-1otherwise\. However, since we observe in Section[2\.2](https://arxiv.org/html/2606.00251#S2.SS2)that base models predictSelf\-Solveon nearly every query regardless of their actual ability, applying GRPO directly is likely ineffective, because rollouts within a group share the same label, leading to vanishing reward variance and a policy gradient that carries no learning signal\. To address this, we add a Diversity\-Filtered Warm\-up \(DFW\) phase before the standard GRPO training, resulting in a two\-stage procedure summarized in Algorithm[1](https://arxiv.org/html/2606.00251#alg1)in Appendix[C](https://arxiv.org/html/2606.00251#A3)\.

*Stage 1: Diversity\-Filtered Warm\-up \(DFW\)\.*We construct a diversified subset𝒟div⊆𝒟\\mathcal\{D\}\_\{\\text\{div\}\}\\subseteq\\mathcal\{D\}by retaining only queries whoseGGrollouts fromπθ0\\pi\_\{\\theta\_\{0\}\}contain bothSelf\-SolveandDelegate\. On this subset, every group contains both labels, so reward variance is non\-zero and GRPO produces meaningful gradients when updating\. After this filtering, we runTwarmT\_\{\\text\{warm\}\}GRPO steps on𝒟div\\mathcal\{D\}\_\{\\text\{div\}\}to break the initialSelf\-Solveprior, producing a calibrated checkpointθwarm\\theta\_\{\\text\{warm\}\}\. Other methods tackle GRPO’s vanishing\-variance issue at the per\-query level\. For example, DAPO[yu2025dapoopensourcellmreinforcement](https://arxiv.org/html/2606.00251#bib.bib73)draws additional rollouts to rescue the variance within a group, but such methods fail here: the base model is so biased towardSelf\-Solvethat no amount of resampling on a single query is likely to produceDelegate\. DFW sidesteps this by diversifying across queries instead of within them\.

*Stage 2: Full GRPO Training\.*Starting fromθwarm\\theta\_\{\\text\{warm\}\}, we train for anotherTfullT\_\{\\text\{full\}\}steps on the full dataset𝒟\\mathcal\{D\}, where the now\-informative advantages drive learning across both solvable and unsolvable queries\.

In both stages of GRPO training, we follow the following objective:

ℒGRPO​\(θ;ℬ\)=−𝔼xi∼ℬ\{oi\(g\)\}∼πθold​\[1G​∑g=1Gmin⁡\(ρi\(g\)​\(θ\)​Ai\(g\),clip​\(ρi\(g\)​\(θ\),1−ϵ,1\+ϵ\)​Ai\(g\)\)\]\+β⋅KL​\[πθ∥πref\],\\footnotesize\\mathcal\{L\}\_\{\\text\{GRPO\}\}\(\\theta;\\mathcal\{B\}\)=\-\\,\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}x\_\{i\}\\sim\\mathcal\{B\}\\\\ \\\{o\_\{i\}^\{\(g\)\}\\\}\\sim\\pi\_\{\\theta\_\{\\text\{old\}\}\}\\end\{subarray\}\}\\\!\\left\[\\frac\{1\}\{G\}\\sum\_\{g=1\}^\{G\}\\min\\\!\\Big\(\\rho\_\{i\}^\{\(g\)\}\(\\theta\)\\,A\_\{i\}^\{\(g\)\},\\;\\mathrm\{clip\}\\\!\\big\(\\rho\_\{i\}^\{\(g\)\}\(\\theta\),\\,1\{\-\}\\epsilon,\\,1\{\+\}\\epsilon\\big\)A\_\{i\}^\{\(g\)\}\\Big\)\\right\]\+\\beta\\cdot\\mathrm\{KL\}\\\!\\left\[\\pi\_\{\\theta\}\\,\\\|\\,\\pi\_\{\\text\{ref\}\}\\right\],\(3\)whereρi\(g\)​\(θ\)=πθ​\(oi\(g\)∣xi\)/πθold​\(oi\(g\)∣xi\)\\rho\_\{i\}^\{\(g\)\}\(\\theta\)=\\pi\_\{\\theta\}\(o\_\{i\}^\{\(g\)\}\\mid x\_\{i\}\)/\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(o\_\{i\}^\{\(g\)\}\\mid x\_\{i\}\),ϵ\\epsilonis the clipping threshold,β\\betais the KL strength, andAi\(g\)=\(Ri\(g\)−meang′​Ri\(g′\)\)/stdg′​Ri\(g′\)A\_\{i\}^\{\(g\)\}=\(R\_\{i\}^\{\(g\)\}\-\\mathrm\{mean\}\_\{g^\{\\prime\}\}R\_\{i\}^\{\(g^\{\\prime\}\)\}\)/\\mathrm\{std\}\_\{g^\{\\prime\}\}R\_\{i\}^\{\(g^\{\\prime\}\)\}, which is the group\-normalized advantage within each rollout group\.ℬ\\mathcal\{B\}is a mini\-batch from𝒟div\\mathcal\{D\}\_\{\\text\{div\}\}\(Stage 1\) or𝒟\\mathcal\{D\}\(Stage 2\)\.

### 3\.3CSA Inference & Evaluation

Inference\.At test time, given a queryqq, the trained modelℳ′\\mathcal\{M\}^\{\\prime\}takes the same inputx=\[p;q\]x=\[p;q\]as during training and produces

ℳ′​\(x\)=\{y^,for SFTLabel,\[r^;y^\],for SFTSelf, SFTTeacher, and RLVR,\\mathcal\{M\}^\{\\prime\}\(x\)\\;=\\;\\begin\{cases\}\\hat\{y\},&\\text\{for SFT\\textsubscript\{Label\}\{\}\},\\\\ \[\\hat\{r\};\\,\\hat\{y\}\],&\\text\{for SFT\\textsubscript\{Self\}\{\}, SFT\\textsubscript\{Teacher\}\{\}, and RLVR\},\\end\{cases\}\(4\)wherey^∈\{Self\-Solve,Delegate\}\\hat\{y\}\\in\\\{\\textsc\{Self\-Solve\},\\textsc\{Delegate\}\\\}is the CSA decision andr^\\hat\{r\}is a rationale justifying it\. Only SFTLabelemits the bare decision; the other three first analyze the query relative to their own competence and then commit to a decision\.

Evaluation\.After training, the original modelℳ\\mathcal\{M\}becomes a new modelℳ′\\mathcal\{M\}^\{\\prime\}\. Under the assumption stated in Section[3\.1](https://arxiv.org/html/2606.00251#S3.SS1), that acquiring CSA does not alter the model’s underlying problem\-solving ability, the training labels\{yi\}\\\{y\_\{i\}\\\}probed fromℳ\\mathcal\{M\}should remain valid forℳ′\\mathcal\{M\}^\{\\prime\}\. To verify this assumption, we re\-probeℳ′\\mathcal\{M\}^\{\\prime\}on each evaluation query using the same procedure as Eq\. \([1](https://arxiv.org/html/2606.00251#S3.E1)\), yielding fresh labels\{y~i\}\\\{\\tilde\{y\}\_\{i\}\\\}, and evaluate CSA by comparing the model’s decisiony^i\\hat\{y\}\_\{i\}againsty~i\\tilde\{y\}\_\{i\}\. This not only measures CSA performance against the most up\-to\-date labels but also tells us whether the trained model has genuinely acquired CSA without degrading its original capability\.

## 4Experiments and Analysis

### 4\.1Setup

We conduct experiments on theQwen3[yang2025qwen3technicalreport](https://arxiv.org/html/2606.00251#bib.bib69)family \(0\.6B, 1\.7B, 4B, 8B, 14B\), as well asOLMo2[olmo20252olmo2furious](https://arxiv.org/html/2606.00251#bib.bib47)\(7B, 13B\)\. Evaluation spans two domains:Math\(GSM8K[cobbe2021trainingverifierssolvemath](https://arxiv.org/html/2606.00251#bib.bib13), MATH500[lightman2023lets](https://arxiv.org/html/2606.00251#bib.bib41), and AIME[aime\_1983\_2024](https://arxiv.org/html/2606.00251#bib.bib61)\) andScience\(MMLU\-Pro[wang2024mmluprorobustchallengingmultitask](https://arxiv.org/html/2606.00251#bib.bib63)across the biology, chemistry, health, and physics categories\)\. For each query, we instantiate Eq\. \([1](https://arxiv.org/html/2606.00251#S3.E1)\) withK=5K=5and aggregationgg:*any\-correct*for math \(open\-ended\) and*majority\-correct*for science \(to discount lucky guesses\)\. For evaluation, we design two classes of metrics to assess our training strategies\. First, we measure CSA quality viaCapability Discrimination Score \(CDS\)andMacro F1 \(M\-F1\)\. Second, we useCapability Ratio \(CR\)to check whether the model retains its original task ability after CSA training\. Full details of the experimental setup are in Appendix[D](https://arxiv.org/html/2606.00251#A4)\.

### 4\.2Main Results

Table 1:CSA performance across training strategies and domains\. Higher is better for all metrics; see Appendix[D\.5](https://arxiv.org/html/2606.00251#A4.SS5)for detailed definitions\.Boldmarks the best andunderlinethe second best\. Vanilla CR is 100 by definition and serves as the reference, excluded from comparison\.NAindicates an overly imbalanced vanilla output distribution \(e\.g\.≤10\\leq 10delegated samples\), where CDS is not applicable\.MathScienceModelMetricVanillaSFTLabelSFTSelfSFTTeacherRLVRVanillaSFTLabelSFTSelfSFTTeacherRLVRQwen3\-0\.6BCDS0\.662\.956\.879\.7515\.26−\-0\.330\.003\.973\.63−\-0\.03M\-F10\.4260\.4560\.6470\.7030\.7610\.2950\.4490\.5860\.5740\.487CR10013\.870\.276\.798\.01002\.061\.869\.486\.8black\!25blackQwen3\-1\.7BCDS5\.153\.417\.677\.8617\.763\.025\.306\.392\.7510\.12M\-F10\.5320\.4040\.6770\.6890\.8040\.4640\.4920\.6110\.5470\.674CR10010\.492\.073\.198\.810037\.389\.681\.2102\.2black\!25blackQwen3\-4BCDSNA5\.516\.145\.1517\.17NA4\.475\.927\.388\.07M\-F10\.5070\.4060\.6480\.6420\.8010\.4220\.3150\.5980\.6360\.644CR10013\.395\.096\.4100\.710020\.886\.887\.6101\.8black\!25blackQwen3\-8BCDSNA11\.106\.086\.3514\.13NA10\.734\.196\.346\.72M\-F10\.4880\.7600\.6560\.6770\.7720\.4500\.5020\.5800\.6230\.624CR10089\.298\.7102\.3101\.810049\.191\.594\.0101\.5black\!25blackQwen3\-14BCDSNA9\.317\.053\.4310\.94NA2\.255\.575\.125\.25M\-F10\.5060\.7270\.6720\.5980\.7530\.4500\.2050\.6100\.6040\.566CR10084\.398\.2106\.3100\.81006\.792\.595\.2100\.4black\!25blackOLMo2\-7BCDS10\.873\.6219\.0718\.2626\.603\.440\.007\.837\.0810\.95M\-F10\.7360\.2960\.8420\.8320\.8860\.5510\.4440\.6630\.6540\.691CR1002\.196\.692\.7101\.91000\.788\.376\.9106\.5black\!25blackOLMo2\-13BCDS7\.240\.9116\.0615\.0222\.736\.130\.005\.706\.998\.40M\-F10\.6680\.2840\.7790\.7890\.8860\.5440\.4200\.6060\.6360\.657CR1003\.149\.772\.2100\.91005\.076\.261\.199\.5AverageCDS5\.985\.269\.859\.4017\.803\.063\.255\.655\.617\.07M\-F10\.5520\.4760\.7030\.7040\.8090\.4540\.4040\.6080\.6110\.620CR10030\.985\.888\.5100\.410017\.483\.880\.899\.8

Table[4\.2](https://arxiv.org/html/2606.00251#S4.SS2)compares all four training strategies across models and domains along two axes: how effectively each injects CSA, and how well it preserves the model’s original task ability\.

##### RLVR substantially improves CSA across scales and domains\.

RLVR outperforms all SFT variants, achieving the highest CDS and M\-F1 for most models across Math and Science\. On Math, RLVR reaches an average CDS of 17\.80 versus 9\.85 for the best SFT variant \(SFTSelf\), and an average M\-F1 of 0\.809 versus 0\.704\. Science shows the same pattern, with CDS of 7\.07 vs\. 5\.65 and M\-F1 of 0\.620 vs\. 0\.611\. Overall CSA scores are lower on Science because it is the harder setting: Math combines multiple datasets with heterogeneous formats and difficulty, providing rich surface\-level cues for capability self\-assessment, whereas Science comes from a single source with a uniform multiple\-choice format, offering no such cues\. A separate failure mode appears on Qwen3\-0\.6B on Science, suggesting that RLVR needs a minimum capability floor to inject meaningful CSA\. When we continue training on this small model, it eventually converges to delegating every query, which may be a reasonable shortcut given its weak capability\.

##### RLVR preserves original task ability while injecting CSA\.

Comparing CR against vanilla counterparts reveals a clear contrast\. RLVR preserves task ability across the board, retaining nearly 100% of vanilla’s ability on both Math \(100\.4%\) and Science \(99\.8%\)\. In contrast, all three SFT variants catastrophically erode problem\-solving ability\. SFTLabelretains only about 30% of vanilla’s ability on Math and under 20% on Science, and even the reasoning\-augmented SFTSelfand SFTTeacherretain only 80–90% on average across both domains\.

### 4\.3Cross\-Domain Generalization

Table[2](https://arxiv.org/html/2606.00251#S4.T2)evaluates Qwen3 models trained on one domain and tested on the other without any domain\-specific adaptation\. RLVR exhibits the strongest cross\-domain transfer in both directions\. From Science to Math, RLVR reaches an average CDS of 9\.98 versus 9\.02 for the best SFT variant\. The gap is even more pronounced from Math to Science, where RLVR leads on CDS for most models and lifts the average from 1\.66 \(best SFT\) to 4\.19, more than doubling the SFT baselines\. These results indicate that RLVR teaches a transferable self\-assessment skill, not a domain\-specific shortcut\.

Comparing RLVR’s out\-of\-domain numbers against the in\-domain numbers in Table[4\.2](https://arxiv.org/html/2606.00251#S4.SS2), we find that Science\-to\-Math transfers better than Math\-to\-Science\. This is intuitive, as Science is the harder of the two domains, so transferring from an easier source to a harder target loses more signal than the reverse\. The same asymmetry appears in the SFT variants, most strikingly in SFTLabel, whose CDS even goes negative on Math\-to\-Science, indicating a complete breakdown of self\-assessment under this transfer direction\. Finally, mirroring the discussion in Section[4\.2](https://arxiv.org/html/2606.00251#S4.SS2), we find that the smallest 0\.6B model generalizes poorly in both directions, suggesting that very small models lack the basic capacity needed for transferable CSA\.

Table 2:Cross\-domain transfer CDS scores\. Models are trained on one domain and evaluated on the other, e\.g\. “Science→\\rightarrowMath” denotes training on Science and evaluation on Math\.Science→\\rightarrowMathMath→\\rightarrowScienceModelSFTLabelSFTSelfSFTTeacherRLVRSFTLabelSFTSelfSFTTeacherRLVRQwen3\-0\.6B3\.254\.386\.790\.72−\-3\.191\.263\.791\.47Qwen3\-1\.7B8\.517\.3510\.4614\.17−\-1\.511\.481\.447\.09Qwen3\-4B10\.747\.9211\.0813\.86−\-3\.541\.880\.863\.47Qwen3\-8B11\.969\.607\.6711\.351\.132\.431\.023\.87Qwen3\-14B10\.628\.178\.139\.78−\-2\.351\.151\.195\.04Average9\.027\.488\.839\.98−\-1\.891\.641\.664\.19

### 4\.4Why is RL better suited for this task?

We’ve shown thatRLVR more effectively injects CSA, preserves task ability, and generalizes across domains\. We further support this with two analyses of what each training paradigm actually teaches the model\.

![Refer to caption](https://arxiv.org/html/2606.00251v1/Figures/sft_label_vs_rlvr_dynamics.png)Figure 3:Qwen3\-4B trained over three epochs on Science\. We track*problem\-solving accuracy*on the original task, and M\-F1 computed against two label sets:*original labels*probed from the base model and used during training, and*re\-probed labels*freshly generated by the current checkpoint at each epoch\.Left:SFTLabel\.Right:RLVR\.##### SFTLabelmemorizes a fixed rule; RLVR learns CSA\.

As shown in Figure[3](https://arxiv.org/html/2606.00251#S4.F3), we track this process on Qwen3\-4B over three training epochs on Science, and the two methods exhibit sharply different trends\. For SFTLabel, M\-F1 against the original labels keeps rising while M\-F1 against the re\-probed labels drops, and problem\-solving accuracy collapses sharply after the first epoch\. This means the original labels no longer reflect the current model’s capability: as training proceeds, the model is gradually taught a fixed delegation rule frozen at the base\-model snapshot, while its underlying problem\-solving ability is destroyed in the process\. For RLVR, the two M\-F1 curves rise together throughout training, and problem\-solving accuracy stays at the base level\. This shows that RLVR’s self\-assessment remains aligned with the model’s current capability rather than locked to a stale snapshot\. This contrast highlights the importance of the assumption stated in Section[3\.1](https://arxiv.org/html/2606.00251#S3.SS1): that acquiring CSA should not come at the cost of the model’s underlying problem\-solving ability\. Once violated, the model is not genuinely taught CSA but instead memorizes a fixed delegation rule tied to the base model, as in SFTLabel\. RLVR preserves this assumption and genuinely teaches CSA\.

SFTSelfand SFTTeachermatch the ratio; RLVR makes the decisions\.We further contrast RLVR with SFTSelfand SFTTeacher, which share RLVR’s reasoning\-augmented output format and do not exhibit such severe degradation as SFTLabel\.

![Refer to caption](https://arxiv.org/html/2606.00251v1/Figures/decision_accuracy_calibration.png)Figure 4:Comparison of RLVR with SFT variants on Math\.Left:decision accuracy, i\.e\., correctSelf\-SOLVE/DELEGATErate, and calibration error, i\.e\., gap between predicted and trueSelf\-SOLVErates\.Right:Qwen3\-4B case study comparing each method’s ground\-truth, predicted, and correctness distributions\.Figure[4](https://arxiv.org/html/2606.00251#S4.F4)shows RLVR achieves higher decision accuracy across all scales, indicating more accurate per\-query decisions, while the two SFT variants exhibit lower calibration error, indicating a closer fit to theSelf\-SOLVE/DELEGATEratio\. This points to a deeper difference in what each method actually learns\. The Qwen3\-4B case on the right makes this clear: the ground truth is 18\.2% delegate / 81\.8% self\-solve, and SFTTeacherpredicts a near\-identical 17\.6% / 82\.4%, which is a near\-perfect aggregate match, hence its low calibration error\. But matching the ratio is not the same as making the right decisions: among the queries SFTTeacherself\-solves, many are unsolvable, and among those it delegates, many it could have solved\. The fragmented correctness bar reflects this, and explains why SFTTeacher’s decision accuracy falls well short of RLVR’s\. SFTSelfshows the same pattern, less pronounced\. In short, SFTSelfand SFTTeacherlearn how often to delegate, not which queries to delegate, which is what CSA truly requires\.

### 4\.5Ablation Study of DFW

Table 3:Ablation on the DFW phase on Math\.nS/nDn\_\{S\}/n\_\{D\}: counts ofSelf\-SOLVEvs\.DELEGATEpredictions; Reward std: standard deviation of rewards during training\.Qwen3\-4BQwen3\-8BMetricw/ow/w/ow/CDS12\.1017\.174\.0814\.13M\-F10\.7040\.8010\.4860\.772nS/ nD441/49417/73484/6426/64Reward std0\.1330\.2140\.1060\.232Table[3](https://arxiv.org/html/2606.00251#S4.T3)confirms the importance of DFW \(introduced in Section[3\.2](https://arxiv.org/html/2606.00251#S3.SS2)RLVR\): without DFW, the policy collapses to almost always predictingSelf\-SOLVE\(441/490 at 4B and 484/490 at 8B\), and both CDS and M\-F1 drop substantially \(e\.g\., CDS drops from 17\.17 to 12\.10 at 4B and from 14\.13 to 4\.08 at 8B\)\. The training reward standard deviation also falls sharply \(e\.g\.,0\.232→0\.1060\.232\\to 0\.106at 8B\)\. These changes show that without DFW, the reward signal across rollouts is too uniform to drive meaningful policy updates, highlighting DFW’s role in providing the variance needed for effective exploration and performance gains\.

In addition to verifying DFW’s effectiveness, we uncover two findings about the diversified subsets during DFW: \(1\)*stronger models yield smaller diversified subsets*, as they are more confident and produce more consistent rollouts; \(2\)*rollout diversity emerges primarily on queries the model should delegate*, indicating that uncertainty in rollouts arises when problems exceed the model’s capability\. We discuss both in detail in Appendix[E\.1](https://arxiv.org/html/2606.00251#A5.SS1)\.

![Refer to caption](https://arxiv.org/html/2606.00251v1/Figures/self_solve_rate_by_difficulty.png)Figure 5:Ground\-truth and predictedSelf\-SOLVErate of RLVR\-trained Qwen3 models on Math across datasets of increasing difficulty\.
![Refer to caption](https://arxiv.org/html/2606.00251v1/Figures/vanilla_vs_rlvr_distribution.png)Figure 6:Self\-SOLVEandDELEGATErate distributions of Qwen3 models on Science\. Vanilla vs RLVR\.

### 4\.6RLVR’s CSA Aligns with Difficulty and Capability

Figure[6](https://arxiv.org/html/2606.00251#S4.F6)compares the predicted and ground\-truthSelf\-SOLVErates of RLVR\-trained Qwen3 models \(1\.7B, 4B, 8B\) on Math, to test whether RLVR’s decisions adapt to query difficulty\. We find that the predicted rate decreases monotonically as difficulty rises and closely tracks the ground truth, showing that RLVR’s decisions align with query difficulty rather than applying a uniform policy\. Figure[6](https://arxiv.org/html/2606.00251#S4.F6)shows that RLVR decisions are calibrated to model capability: weaker modelsDELEGATEmore, while stronger modelsSelf\-SOLVEmore\. In addition, we provide two qualitative case studies in Appendix[E\.2](https://arxiv.org/html/2606.00251#A5.SS2)comparing RLVR\-trained and vanilla models\.

## 5Applications

Table 4:CSA\-based self\-routing with RLVR\. Local: Qwen3\-0\.6B/1\.7B/4B; Cloud: Qwen3\-235B\-A22B\.0\.6B1\.7B4BAccuracyLocal\-Only0\.3590\.6000\.694Cloud\-Only0\.857Local \+ CloudVanilla0\.3960\.6410\.702CSA\-Based0\.5730\.7330\.800Efficiency\# Cloud Calls1569673Avg\. Time \(s\)0\.191\.010\.66Avg\. Tokens79320137

Table 5:Data selection results onMedMCQAscience dataset\.Qwen3\-1\.7BQwen3\-4BMethod5%10%20%5%10%20%Eval only0\.4000\.422Train on all0\.4440\.476Random0\.4000\.4460\.4380\.3920\.3020\.326Dev Similarity0\.3960\.3900\.3980\.4340\.4460\.336High Loss0\.2680\.2640\.2740\.2360\.2040\.202Low Loss0\.4080\.4280\.4160\.4940\.5100\.486CSA\-Based0\.4300\.4520\.4480\.5740\.5580\.562

### 5\.1Self\-Routing

A model with accurate CSA is a natural router in a local\-cloud inference system: when the local model chooses to self\-solve, the query is answered on\-device; when it chooses to delegate, the query is routed to a powerful cloud model\. The local model serves as its own router, offering a practical cost\-accuracy tradeoff between local\-only and cloud\-only inference\. See Appendix[E\.3\.1](https://arxiv.org/html/2606.00251#A5.SS3.SSS1)for a detailed illustration\. Table[4](https://arxiv.org/html/2606.00251#S5.T4)presents the routing results\.Accuracy\.RLVR routing consistently outperforms both the local\-only baseline and vanilla routing across all three local model sizes, with the 4B local model reaching 0\.800, approaching the cloud\-only upper bound of 0\.857\.Efficiency\.RLVR routing delegates only 156, 96, and 73 out of 490 queries to the cloud for the 0\.6B, 1\.7B, and 4B local models, with low average latency and token overhead per query\.

### 5\.2Self\-Guided Data Selection

We further test whether CSA can serve as a data\-selection signal\. Starting from a base modelM0M\_\{0\}, we train an RL\-based CSA modelMRLVRM\_\{\\mathrm\{RLVR\}\}on MMLU\-Pro[wang2024mmluprorobustchallengingmultitask](https://arxiv.org/html/2606.00251#bib.bib63)and apply it to a fresh candidate pool from MedMCQA[pal2022medmcqa](https://arxiv.org/html/2606.00251#bib.bib49)\. We select examples based on a metric which prioritizes examples that the CSA model is likely toDelegatewhile filtering for cases that are uncertain but not extreme loss outliers, and fine\-tune the model on this subset\. We then compare against standard selection baselines, including random, loss\-based, and dev\-similarity selection, to assess whether CSA identifies examples that more effectively improve the model’s downstream performance\.

Table[5](https://arxiv.org/html/2606.00251#S5.T5)shows that CSA\-based selection consistently outperforms the baselines across both model sizes and all selection budgets\. For Qwen3\-1\.7B, CSA selection improves over the eval\-only model at every budget and slightly exceeds training on the full 50K candidate pool when using only 10–20% of the data\. The gains are especially pronounced for Qwen3\-4B\. CSA\-selected subsets achieve 55\.8–57\.4% accuracy, substantially outperforming both the eval\-only baseline \(42\.2%\) and training on all examples \(47\.6%\)\. In contrast, high\-loss selection performs poorly across all settings, indicating that simply selecting the hardest examples is not sufficient and can over\-emphasize noisy or unlearnable samples\. Low\-loss selection is the strongest baseline for Qwen3\-4B, but remains well below CSA\-based selection\. These results suggest that the learned CSA signal identifies examples that are not merely difficult or similar to the development set, but are especially useful for improving the model under a limited fine\-tuning budget\. See Appendix[E\.3\.2](https://arxiv.org/html/2606.00251#A5.SS3.SSS2)for the full details\.

## 6Conclusion

We introduced CSA as a learnable trait that enables a model to recognize when a query exceeds its capabilities and delegate accordingly\. Through a systematic study, we showed that reinforcement learning is uniquely suited for injecting CSA\. We believe CSA is a promising building block for more trustworthy and responsible AI systems\. Ultimately, intelligence is not only about solving problems, but also about recognizing the limits of one’s own competence\.

## References

- \[1\]Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al\.A survey on data selection for language models\.arXiv preprint arXiv:2402\.16827, 2024\.
- \[2\]Anastasios N Angelopoulos and Stephen Bates\.Conformal prediction: A gentle introduction\.Foundations and Trends in Machine Learning, 16\(4\):494–591, 2023\.
- \[3\]Amos Azaria and Tom Mitchell\.The internal state of an llm knows when it’s lying\.InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 967–976, 2023\.
- \[4\]Casey O\. Barkan, Sidney Black, and Oliver Sourbut\.Do large language models know what they are capable of?InThe Fourteenth International Conference on Learning Representations, 2026\.
- \[5\]Peter L Bartlett and Marten H Wegkamp\.Classification with a reject option using a hinge loss\.Journal of Machine Learning Research, 9\(8\), 2008\.
- \[6\]Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston\.Curriculum learning\.InProceedings of the 26th annual international conference on machine learning, pages 41–48, 2009\.
- \[7\]Margarida Campos, António Farinhas, Chrysoula Zerva, Mário AT Figueiredo, and André FT Martins\.Conformal prediction for natural language processing: A survey\.Transactions of the Association for Computational Linguistics, 12:1497–1516, 2024\.
- \[8\]Iván Vicente Moreno Cencerrado, Arnau Padrés Masdemont, Anton Gonzalvez Hawthorne, David Demitri Africa, and Lorenzo Pacchiardi\.No answer needed: Predicting llm answer accuracy from question\-only linear probes\.arXiv preprint arXiv:2509\.10625, 2025\.
- \[9\]Arslan Chaudhry, Sridhar Thiagarajan, and Dilan Gorur\.Finetuning language models to emit linguistic expressions of uncertainty, 2024\.
- \[10\]John J Cherian, Isaac Gibbs, and Emmanuel J Candès\.Large language model validity via enhanced conformal prediction methods\.Advances in Neural Information Processing Systems, 37:114812–114842, 2024\.
- \[11\]Prateek Chhikara\.Mind the confidence gap: Overconfidence, calibration, and distractor effects in large language models\.arXiv preprint arXiv:2502\.11028, 2025\.
- \[12\]C Chow\.On optimum recognition error and reject tradeoff\.IEEE Transactions on information theory, 16\(1\):41–46, 2003\.
- \[13\]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman\.Training verifiers to solve math word problems, 2021\.
- \[14\]Yu Du, Fangyun Wei, and Hongyang Zhang\.Anytool: Self\-reflective, hierarchical agents for large\-scale api calls\.arXiv preprint arXiv:2402\.04253, 2024\.
- \[15\]Ran El\-Yaniv et al\.On the foundations of noise\-free selective classification\.Journal of Machine Learning Research, 11\(5\), 2010\.
- \[16\]Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal\.Detecting hallucinations in large language models using semantic entropy\.Nature, 630\(8017\):625–630, 2024\.
- \[17\]Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong\.Retool: Reinforcement learning for strategic tool use in llms\.arXiv preprint arXiv:2504\.11536, 2025\.
- \[18\]Yonatan Geifman and Ran El\-Yaniv\.Selectivenet: A deep neural network with an integrated reject option\.InInternational conference on machine learning, pages 2151–2159\. PMLR, 2019\.
- \[19\]Isaac Gibbs, John J Cherian, and Emmanuel J Candès\.Conformal prediction with conditional guarantees\.Journal of the Royal Statistical Society Series B: Statistical Methodology, 87\(4\):1100–1126, 2025\.
- \[20\]Tilmann Gneiting and Adrian E Raftery\.Strictly proper scoring rules, prediction, and estimation\.Journal of the American statistical Association, 102\(477\):359–378, 2007\.
- \[21\]Google DeepMind\.Gemini 3\.1 Pro model card\.Technical report, Google DeepMind, 2 2026\.
- \[22\]Aaron Grattafiori et al\.The llama 3 herd of models, 2024\.
- \[23\]Tobias Groot and Matias Valdenegro\-Toro\.Overconfidence is key: Verbalized uncertainty evaluation in large language and vision\-language models\.InProceedings of the 4th Workshop on Trustworthy Natural Language Processing \(TrustNLP 2024\), pages 145–171, 2024\.
- \[24\]Yuxian Gu, Li Dong, Hongning Wang, Yaru Hao, Qingxiu Dong, Furu Wei, and Minlie Huang\.Data selection via optimal control for language models\.arXiv preprint arXiv:2410\.07064, 2024\.
- \[25\]Zhengyao Gu and Mark Hopkins\.On the evaluation of neural selective prediction methods for natural language processing\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), pages 7888–7899, 2023\.
- \[26\]Mustafa Omer Gul, Claire Cardie, and Tanya Goyal\.Pay\-per\-search models are abstention models\.arXiv preprint arXiv:2510\.01152, 2025\.
- \[27\]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger\.On calibration of modern neural networks\.InInternational conference on machine learning, pages 1321–1330\. PMLR, 2017\.
- \[28\]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al\.Deepseek\-r1: Incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948, 2025\.
- \[29\]Sophia Hager, David Mueller, Kevin Duh, and Nicholas Andrews\.Uncertainty distillation: Teaching language models to express semantic confidence, 2025\.
- \[30\]Jiatong Han, Neil Band, Muhammed Razzak, Jannik Kossen, Tim GJ Rudner, and Yarin Gal\.Simple factuality probes detect hallucinations in long\-form natural language generation\.Findings of the Association for Computational Linguistics: EMNLP, pages 16209–16226, 2025\.
- \[31\]Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al\.A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions\.ACM Transactions on Information Systems, 43\(2\):1–55, 2025\.
- \[32\]Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, et al\.Metatool benchmark for large language models: Deciding whether to use tools and which to use\.arXiv preprint arXiv:2310\.03128, 2023\.
- \[33\]Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung\.Survey of hallucination in natural language generation\.ACM computing surveys, 55\(12\):1–38, 2023\.
- \[34\]Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield\-Dodds, Nova DasSarma, Eli Tran\-Johnson, et al\.Language models \(mostly\) know what they know\.arXiv preprint arXiv:2207\.05221, 2022\.
- \[35\]Sanyam Kapoor, Nate Gruver, Manley Roberts, Katherine Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew G Wilson\.Large language models must be taught to know what they don’t know\.Advances in Neural Information Processing Systems, 37:85932–85972, 2024\.
- \[36\]Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal\.Semantic entropy probes: Robust and cheap hallucination detection in llms\.arXiv preprint arXiv:2406\.15927, 2024\.
- \[37\]Bhawesh Kumar, Charlie Lu, Gauri Gupta, Anil Palepu, David Bellamy, Ramesh Raskar, and Andrew Beam\.Conformal prediction with large language models for multi\-choice question answering\.arXiv preprint arXiv:2305\.18404, 2023\.
- \[38\]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E\. Gonzalez, Hao Zhang, and Ion Stoica\.Efficient memory management for large language model serving with pagedattention, 2023\.
- \[39\]Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al\.Tulu 3: Pushing frontiers in open language model post\-training\.arXiv preprint arXiv:2411\.15124, 2024\.
- \[40\]Minjae Lee, Kyungmin Kim, Taesoo Kim, and Sangdon Park\.Selective generation for controllable language models\.Advances in Neural Information Processing Systems, 37:50494–50527, 2024\.
- \[41\]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe\.Let’s verify step by step\.arXiv preprint arXiv:2305\.20050, 2023\.
- \[42\]Ziyin Liu, Zhikang Wang, Paul Pu Liang, Russ R Salakhutdinov, Louis\-Philippe Morency, and Masahito Ueda\.Deep gamblers: Learning to abstain with portfolio theory\.Advances in Neural Information Processing Systems, 32, 2019\.
- \[43\]David Madras, Toniann Pitassi, and Richard Zemel\.Predict responsibly: Improving fairness and accuracy by learning to defer, 2018\.
- \[44\]Potsawee Manakul, Adian Liusie, and Mark Gales\.Selfcheckgpt: Zero\-resource black\-box hallucination detection for generative large language models\.InProceedings of the 2023 conference on empirical methods in natural language processing, pages 9004–9017, 2023\.
- \[45\]Sabrina J Mielke, Arthur Szlam, Emily Dinan, and Y\-Lan Boureau\.Reducing conversational agents’ overconfidence through linguistic calibration\.Transactions of the Association for Computational Linguistics, 10:857–872, 2022\.
- \[46\]Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al\.Webgpt: Browser\-assisted question\-answering with human feedback\.arXiv preprint arXiv:2112\.09332, 2021\.
- \[47\]Team OLMo et al\.2 olmo 2 furious, 2025\.
- \[48\]OpenAI\.GPT\-5 system card\.Technical report, OpenAI, 8 2025\.
- \[49\]Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu\.Medmcqa: A large\-scale multi\-subject multi\-choice dataset for medical domain question answering\.InConference on health, inference, and learning, pages 248–260\. PMLR, 2022\.
- \[50\]Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S Jaakkola, and Regina Barzilay\.Conformal language modeling\.arXiv preprint arXiv:2306\.10193, 2023\.
- \[51\]Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He\.Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters\.InProceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3505–3506, 2020\.
- \[52\]Timo Schick, Jane Dwivedi\-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom\.Toolformer: Language models can teach themselves to use tools\.Advances in neural information processing systems, 36:68539–68551, 2023\.
- \[53\]Glenn Shafer and Vladimir Vovk\.A tutorial on conformal prediction\.Journal of machine learning research, 9\(3\), 2008\.
- \[54\]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al\.Deepseekmath: Pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300, 2024\.
- \[55\]Reza Shirkavand, Shangqian Gao, Peiran Yu, and Heng Huang\.Cost\-aware contrastive routing for LLMs\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems, 2025\.
- \[56\]Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfogel\.The curious case of hallucinatory \(un\) answerability: Finding truths in the hidden states of over\-confident large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3607–3625, 2023\.
- \[57\]Jiayuan Su, Jing Luo, Hongwei Wang, and Lu Cheng\.Api is enough: Conformal prediction for large language models without logit\-access\.InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 979–995, 2024\.
- \[58\]Fengfei Sun, Ningke Li, Kailong Wang, and Lorenz Goette\.Large language models are overconfident and amplify human bias\.arXiv preprint arXiv:2505\.02151, 2025\.
- \[59\]Gemma Team et al\.Gemma 3 technical report, 2025\.
- \[60\]Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning\.Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5433–5442, 2023\.
- \[61\]Hemish Veeraboina\.Aime problem set 1983\-2024, 2023\.
- \[62\]Vladimir Vovk, Alexander Gammerman, and Glenn Shafer\.Algorithmic learning in a random world\.Springer, 2005\.
- \[63\]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen\.Mmlu\-pro: A more robust and challenging multi\-task language understanding benchmark, 2024\.
- \[64\]Bingbing Wen, Jihan Yao, Shangbin Feng, Chenjun Xu, Yulia Tsvetkov, Bill Howe, and Lucy Lu Wang\.Know your limits: A survey of abstention in large language models\.Transactions of the Association for Computational Linguistics, 13:529–556, 2025\.
- \[65\]Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu\.Doremi: Optimizing data mixtures speeds up language model pretraining\.Advances in Neural Information Processing Systems, 36:69798–69818, 2023\.
- \[66\]Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin\.The art of abstention: Selective prediction and error regularization for natural language processing\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\), pages 1040–1051, 2021\.
- \[67\]Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi\.Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms\.arXiv preprint arXiv:2306\.13063, 2023\.
- \[68\]Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang\.On the tool manipulation capability of open\-source large language models\.arXiv preprint arXiv:2305\.16504, 2023\.
- \[69\]An Yang et al\.Qwen3 technical report, 2025\.
- \[70\]Daniel Yang, Yao\-Hung Hubert Tsai, and Makoto Yamada\.On verbalized confidence scores for llms\.arXiv preprint arXiv:2412\.14737, 2024\.
- \[71\]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao\.React: Synergizing reasoning and acting in language models\.InThe eleventh international conference on learning representations, 2022\.
- \[72\]Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuan\-Jing Huang\.Do large language models know what they don’t know?InFindings of the association for Computational Linguistics: ACL 2023, pages 8653–8665, 2023\.
- \[73\]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei\-Ying Ma, Ya\-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang\.Dapo: An open\-source llm reinforcement learning system at scale, 2025\.
- \[74\]Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang\.R\-tuning: Instructing large language models to say ‘i don’t know’\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\), pages 7113–7139, 2024\.

Appendix Contents

## Appendix ARelated Work

Our work is most closely related to research on self\-evaluation and calibration in LLMs, selective prediction and abstention, uncertainty estimation and hallucination detection, and reinforcement learning for decision\-making\. Across these areas prior work has shown that LLMs can sometimes estimate the correctness of their own outputs, expose useful internal uncertainty signals, or abstain when uncertain\. However, most existing work treats this behavior as either a scalar confidence estimation problem or a binary answer\-versus\-abstain problem\. In contrast, our paper studies self\-solve versus delegate as a trainable policy, and shows experimentally that this behavior can be improved with RL, that the gains increase with model size, and that the learned routing behavior transfers across domains\.

### A\.1Self\-Evaluation and Calibration of Correctness in LLMS

Kadavath et al\.\[[34](https://arxiv.org/html/2606.00251#bib.bib34)\]study whether models can estimate the correctness of their own outputs through quantities such as P\(True\) and P\(IK\)\. That work shows that larger models can exhibit useful self\-evaluation signals when queried in the right format, and it established the modern framing of LLM self\-knowledge as a calibration\[[20](https://arxiv.org/html/2606.00251#bib.bib20),[27](https://arxiv.org/html/2606.00251#bib.bib27)\]problem\. Follow\-up work has stressed both the promise and limits of this perspective\. Yin et al\.\[[72](https://arxiv.org/html/2606.00251#bib.bib72)\]ask whether LLMs know what they do not know, focusing on unanswerable questions and explicit uncertainty recognition, while Kapoor et al\.\[[35](https://arxiv.org/html/2606.00251#bib.bib35)\]argue that prompting alone is not enough and show that uncertainty estimation benefits substantially from explicit fine\-tuning\. Zhang et al\.\[[74](https://arxiv.org/html/2606.00251#bib.bib74)\]show models can be instruction\-tuned to say I don’t know\. Barkan et al\.\[[4](https://arxiv.org/html/2606.00251#bib.bib4)\]studies whether LLMs can predict their own task success and finds all tested LLMs are overconfident\. Together, these papers suggest that uncertainty awareness is at least partly learnable, but they still largely treat the object of interest as a score or belief rather than an explicit action policy\.

A large practical literature studies how to extract usable confidence from aligned systems\. Tian et al\.\[[60](https://arxiv.org/html/2606.00251#bib.bib60)\]show that for RLHF\-tuned models, verbalized confidences \(numbers or linguistic markers produced as tokens\) can be more calibrated than raw conditional probabilities on several QA benchmarks, and they catalog prompting/temperature\-scaling strategies for eliciting confidence\. Mielke et al\.\[[45](https://arxiv.org/html/2606.00251#bib.bib45)\]propose linguistic calibration for dialogue agents to better align the style of uncertainty expression with correctness, highlighting that calibration is not only numeric but also communicative\.

A parallel line scrutinizes verbalized uncertainty and its failure modes\. Sun et al\.\[[58](https://arxiv.org/html/2606.00251#bib.bib58)\]finds five LLMs overestimate the probability that their answer is correct by 20–60% on reasoning problems\. Chhikara\[[11](https://arxiv.org/html/2606.00251#bib.bib11)\]analyzes overconfidence as mismatch between predicted confidence and true correctness across QA datasets\. Xiong et al\.\[[67](https://arxiv.org/html/2606.00251#bib.bib67)\]show that LLMs can be systematically overconfident when asked to verbalize confidence, though calibration and error prediction improve with scale and with specific elicitation strategies \(e\.g\., consistency\-based approaches\)\. Groot et al\.\[[23](https://arxiv.org/html/2606.00251#bib.bib23)\]further evaluate verbalized uncertainty across LLMs/VLMs and emphasize overconfidence as a recurring concern\. Yang et al\.\[[70](https://arxiv.org/html/2606.00251#bib.bib70)\]synthesize why empirical findings about verbalized confidence disagree across papers \(prompting styles, evaluation setups\), reinforcing the need for decision\-centric evaluation rather than relying on any single elicitation recipe\.

This distinction between*scalar*uncertainty estimation and*policy learning*matters for our setting\. A scalar confidence estimate can support thresholding after the fact, but it does not by itself define the policy we care about: whether the model should attempt to solve the problem itself or delegate\. Our contribution is therefore not another method for extracting confidence from a pretrained model\. Instead, we train the model to make a structured decision under reward, and evaluate whether that decision\-making behavior itself is teachable\. In that sense, our work is closer to learning a meta\-capability than to post\-hoc calibration alone\. The cross\-domain transfer result is especially important here: if training on science improves delegation behavior on math and vice versa, then the learned behavior is not just memorization of domain\-specific uncertainty cues, but evidence for a more general decision policy\.

### A\.2Selective Prediction, Abstention, and Risk\-Coverage Evaluation

The classical foundation for abstention\[[42](https://arxiv.org/html/2606.00251#bib.bib42)\]is the reject option, often traced to Chow\[[12](https://arxiv.org/html/2606.00251#bib.bib12)\], who derives optimal decision rules trading misclassification error against rejection cost\. Modern selective prediction\[[15](https://arxiv.org/html/2606.00251#bib.bib15)\]formalizes this as optimizing a risk–coverage tradeoff—achieving low error on the subset of instances the model elects to answer\.

In deep learning, Geifman et al\.\[[18](https://arxiv.org/html/2606.00251#bib.bib18)\]treat selective classification as choosing a subset to predict on to satisfy a desired risk constraint\. The broader “learning with rejection” literature develops surrogate losses and theoretical analyses for reject option learning\[[5](https://arxiv.org/html/2606.00251#bib.bib5)\]\.

Within NLP specifically, Xin et al\.\[[66](https://arxiv.org/html/2606.00251#bib.bib66)\]revisit selective prediction for language tasks, comparing confidence estimators and introducing regularization tricks to improve confidence without heavy compute\. Gu & Hopkins\[[25](https://arxiv.org/html/2606.00251#bib.bib25)\]provide a methodological audit of selective prediction evaluation in NLP and propose additional metrics \(e\.g\., refinement\) to separate the quality of confidence scores from downstream accuracy\.

For LLMs, a current synthesis is Wen et al\.\[[64](https://arxiv.org/html/2606.00251#bib.bib64)\], which organizes abstention methods by lifecycle stage \(pretraining, alignment, inference\), and emphasizes that abstention must be evaluated not only technically but also relative to user values and deployment context\.

### A\.3Hallucination Detection

Hallucinations\[[33](https://arxiv.org/html/2606.00251#bib.bib33),[31](https://arxiv.org/html/2606.00251#bib.bib31)\]are precisely failures where the model acts as if it knows \(answers confidently\) when it does not\. A prominent black\-box detection approach is SelfCheckGPT\[[44](https://arxiv.org/html/2606.00251#bib.bib44)\]which detects hallucinated sentences by sampling multiple outputs and measuring disagreement/contradiction—motivated by the intuition that knowledgeable generations are more self\-consistent\. A complementary statistically grounded approach is semantic entropy: Farquhar et al\.\[[16](https://arxiv.org/html/2606.00251#bib.bib16)\]propose uncertainty estimators at the level of meaning, showing that entropy over semantic equivalence classes can detect a subset of hallucinations \(confabulations\) and generalize to unseen questions\. Kossen et al\.\[[36](https://arxiv.org/html/2606.00251#bib.bib36)\]then introduce semantic entropy probes designed to make such uncertainty signals cheaper while retaining detection quality\.

Internal representations can carry correctness/answerability signals even when outputs are overconfident\. Azaria & Mitchell\[[3](https://arxiv.org/html/2606.00251#bib.bib3)\]provide evidence that linear classifiers on hidden states can predict statement truthfulness better than using sequence probability alone, suggesting that "the model knows when it’s lying" at the representation level\. Slobodkin et al\.\[[56](https://arxiv.org/html/2606.00251#bib.bib56)\]focus on unanswerability: they find indications that models encode answerability, even when they still generate hallucinated answers for unanswerable questions\. Newer probe\-based hallucination detectors\[[30](https://arxiv.org/html/2606.00251#bib.bib30)\]push this toward practical deployment \(e\.g\., linear probes for long\-form generation\) while reducing the cost relative to multi\-sampling\. A related direction\[[8](https://arxiv.org/html/2606.00251#bib.bib8)\]predicts answer correctness from question\-only activations, indicating that some correctness signals may be present before decoding the full answer—useful for early deferral/abstention\.

### A\.4Conformal Prediction and Statistical Risk Control for LLMs

Conformal prediction \(CP\) methods usually provide post\-hoc risk control \(thresholding, filtering, prediction sets\) rather than training an LLM to internalize the decision rule\. CP provides distribution\-free coverage guarantees under exchangeability assumptions\[[2](https://arxiv.org/html/2606.00251#bib.bib2),[53](https://arxiv.org/html/2606.00251#bib.bib53),[62](https://arxiv.org/html/2606.00251#bib.bib62)\]\. Because exact conditional coverage is generally impossible in finite samples, modern work develops controlled relaxations and conditional targets; Gibbs et al\.\[[19](https://arxiv.org/html/2606.00251#bib.bib19)\]formalize a spectrum between marginal and conditional validity that is useful when risk differs by topic or subgroup\.

Several recent works adapt CP to LMs and text\. Quach et al\.\[[50](https://arxiv.org/html/2606.00251#bib.bib50)\]propose Conformal Language Modeling, producing sets/subsets of outputs \(or filtered segments of text\) with validity guarantees for language model generations\. Kumar et al\.\[[37](https://arxiv.org/html/2606.00251#bib.bib37)\]apply conformal prediction to multiple\-choice QA with LLMs, arguing CP produces useful uncertainty sets correlated with accuracy\. Because many deployed LLMs are API\-only, Su et al\.\[[57](https://arxiv.org/html/2606.00251#bib.bib57)\]propose CP methods that do not require access to logits, improving applicability in closed models\.

For open\-ended generation\[[7](https://arxiv.org/html/2606.00251#bib.bib7)\], correctness\-labeling and scoring functions become the bottleneck: CP guarantees depend on a nonconformity score that must detect errors in generated text\. Lee et al\.\[[40](https://arxiv.org/html/2606.00251#bib.bib40)\]address this by using textual entailment to define correctness of generated sequences and propose selective generation algorithms controlling false discovery rates \(FDR\) over entailment relations\. A closely related contribution is\[[10](https://arxiv.org/html/2606.00251#bib.bib10)\], which proposes enhanced conformal methods for validity guarantees on LLM outputs, explicitly noting issues of topic\-dependent reliability and the utility loss from overly aggressive filtering; they propose conditional\-conformal\-inspired adaptations and methods to improve the scoring function\.

### A\.5Learning Decision Policies with RL

DeepSeek\-R1’s report\[[28](https://arxiv.org/html/2606.00251#bib.bib28)\]argues that pure RL can incentivize reasoning patterns such as self\-reflection and verification, illustrating that useful behavioral traits can emerge from on\-policy training\. Closely related work studies RL\-trained help\-seeking in LLMs through tool use, API calling, and external search/retrieval\[[71](https://arxiv.org/html/2606.00251#bib.bib71),[52](https://arxiv.org/html/2606.00251#bib.bib52),[46](https://arxiv.org/html/2606.00251#bib.bib46),[32](https://arxiv.org/html/2606.00251#bib.bib32),[68](https://arxiv.org/html/2606.00251#bib.bib68),[17](https://arxiv.org/html/2606.00251#bib.bib17),[14](https://arxiv.org/html/2606.00251#bib.bib14),[26](https://arxiv.org/html/2606.00251#bib.bib26)\]\. While these methods learn when an external resource may improve task performance, they do not target or evaluate an internalized understanding of the model’s own capability boundary independent of the tool itself\. By contrast, we focus on Capability Self\-Assessment as an inherent property of the model: whether it can assess if a query is within its own solvable set or should be delegated\.

### A\.6Data Selection and Curriculum Learning

Albalak et al\. \(2024\)\[[1](https://arxiv.org/html/2606.00251#bib.bib1)\]survey data selection for language models across goals such as quality, diversity, difficulty, and efficiency, and highlights that naively training on all available data may be suboptimal or infeasible\. A prominent practical method is DoReMi\[[65](https://arxiv.org/html/2606.00251#bib.bib65)\], which optimizes domain mixture weights via distributionally robust optimization on a proxy model and then scales training on a reweighted mixture, improving efficiency and average downstream performance\. More recently, data selection has been formulated as an optimal control problem over training dynamics\[[24](https://arxiv.org/html/2606.00251#bib.bib24)\], providing language to treat data scheduling as a sequential decision problem\.

Classical curriculum learning\[[6](https://arxiv.org/html/2606.00251#bib.bib6)\]orders data from easier to harder can improve optimization and generalization\. Our RL\-trained delegation policy can itself be seen as a curriculum mechanism: it chooses when to attempt a solution vs defer, which changes the distribution of attempts the model experiences\. Second, CSA\-derived scores \(probability of success if self\-solving; expected utility of delegating\) can operationalize difficulty in a task\- and model\-specific way\.

## Appendix BPrompt Templates

### B\.1CSA Prompt Template

We use the following prompt template to elicit the model’sSelf\-Solve/Delegatedecision on each query\. The template instructs the model to assess its own capability, and to emit its decision in a structured format consisting of an<analysis\>block followed by a<decision\>block\. The same template is used for all subsequent CSA inference throughout the paper\.

CSA Prompt Template``` Decide whether you can reliably and correctly answer the user’s query. - Choose SELF_SOLVE if you believe you can solve it by yourself. - Choose DELEGATE if you believe it requires a more powerful model. # Output Format <analysis> Explain why you chose SELF_SOLVE or DELEGATE. </analysis> <decision> SELF_SOLVE or DELEGATE </decision> # Query: {query} ```

We parse the model’s response by extracting the content of the<decision\>block and matching it againstSelf\-SolveorDelegate\.

### B\.2Self\-Rationale Prompt \(SFTSelf\)

The training model itself is prompted to produce a self\-assessment justifying the ground\-truth label\. The placeholders\{query\}and\{decision\}are replaced with the queryqiq\_\{i\}and the labelyi∈\{Self\_Solve,Delegate\}y\_\{i\}\\in\\\{\\textsc\{Self\\\_Solve\},\\textsc\{Delegate\}\\\}, respectively\.

Self\-Rationale Prompt Template``` You are given a user query, along with a **ground-truth label** that correctly reflects your capability. The label is: - SELF_SOLVE: you can solve the query by yourself. - DELEGATE: the query requires a more powerful model. For the following query: {query} The ground-truth label is: {decision} Write a coherent self-assessment explaining **why this query should be classified as {decision}**. You do NOT need to actually solve the query. ```

### B\.3Teacher\-Rationale Prompt \(SFTTeacher\)

A stronger teacher modelℳ∗\\mathcal\{M\}^\{\*\}is prompted to produce a rationale on behalf of the target model\. The teacher is instructed to write in the first\-person perspective of the target model so that the resulting rationale stylistically resembles a self\-assessment, even though it is generated externally\.

Teacher\-Rationale Prompt Template``` You are analyzing the behavior of a target language model. You are given a user query, along with a **ground-truth label** that correctly reflects the target model’s capability: - SELF_SOLVE: the target model can solve the query by itself. - DELEGATE: the query requires a more powerful model. For the following user query: {query} The correct label is: {decision} Your goal is to explain **why this query should be classified as {decision} for the target model**. Write the explanation from the **first-person perspective of the target model**, as if it is assessing its own capability. You do NOT need to actually solve the query. ```

## Appendix CRLVR Training Algorithm

We provide the full pseudocode of our two\-stage GRPO procedure in Algorithm[1](https://arxiv.org/html/2606.00251#alg1)\. Stage 1 \(DFW\) trains on a diversity\-filtered subset𝒟div\\mathcal\{D\}\_\{\\text\{div\}\}to break the model’s initialSelf\-Solveprior and produce a calibrated checkpointθwarm\\theta\_\{\\text\{warm\}\}; Stage 2 then continues GRPO training on the full dataset𝒟\\mathcal\{D\}, starting fromθwarm\\theta\_\{\\text\{warm\}\}\.

Algorithm 1Two\-Stage GRPO with Diversity\-Filtered Warm\-up\.1:Dataset

𝒟=\{\(xi,yi\)\}i=1N\\mathcal\{D\}=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{N\}, initial policy

πθ0\\pi\_\{\\theta\_\{0\}\}, reference policy

πref\\pi\_\{\\text\{ref\}\}, group size

GG, warm\-up steps

TwarmT\_\{\\text\{warm\}\}, full\-phase steps

TfullT\_\{\\text\{full\}\}
2:Trained policy

πθfinal\\pi\_\{\\theta\_\{\\text\{final\}\}\}
3:

4:// Stage 1: Diversity\-Filtered Warm\-up \(DFW\)

5:

𝒟div←\{xi∈𝒟:G​rollouts from​πθ0​yield bothSelf\-SolveandDelegate\}\\mathcal\{D\}\_\{\\text\{div\}\}\\leftarrow\\\{x\_\{i\}\\in\\mathcal\{D\}:G\\text\{ rollouts from \}\\pi\_\{\\theta\_\{0\}\}\\text\{ yield both \}\\textsc\{Self\-Solve\}\\text\{ and \}\\textsc\{Delegate\}\\\}⊳\\trianglerightfilter on initial policy

6:

θwarm←Train\-GRPO​\(θ0,𝒟div,Twarm\)\\theta\_\{\\text\{warm\}\}\\leftarrow\\textsc\{Train\-GRPO\}\(\\theta\_\{0\},\\mathcal\{D\}\_\{\\text\{div\}\},T\_\{\\text\{warm\}\}\)
7:

8:// Stage 2: Full GRPO Training

9:

θfinal←Train\-GRPO​\(θwarm,𝒟,Tfull\)\\theta\_\{\\text\{final\}\}\\leftarrow\\textsc\{Train\-GRPO\}\(\\theta\_\{\\text\{warm\}\},\\mathcal\{D\},T\_\{\\text\{full\}\}\)⊳\\trianglerightcontinue on full data

10:return

πθfinal\\pi\_\{\\theta\_\{\\text\{final\}\}\}
11:

12:procedureTrain\-GRPO\(

θinit,ℬsrc,T\\theta\_\{\\text\{init\}\},\\mathcal\{B\}\_\{\\text\{src\}\},T\)

13:

θ←θinit\\theta\\leftarrow\\theta\_\{\\text\{init\}\}
14:for

t=1,…,Tt=1,\\ldots,Tdo

15:

θold←θ\\theta\_\{\\text\{old\}\}\\leftarrow\\theta
16:Sample mini\-batch

ℬ⊂ℬsrc\\mathcal\{B\}\\subset\\mathcal\{B\}\_\{\\text\{src\}\}
17:foreach

xi∈ℬx\_\{i\}\\in\\mathcal\{B\}do

18:Draw

GGcandidates

\{oi\(g\)\}g=1G∼πθold\(⋅∣xi\)\\\{o\_\{i\}^\{\(g\)\}\\\}\_\{g=1\}^\{G\}\\sim\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\cdot\\mid x\_\{i\}\), each with predicted label

y^i\(g\)\\hat\{y\}\_\{i\}^\{\(g\)\}
19:Compute reward

Ri\(g\)=\+1R\_\{i\}^\{\(g\)\}=\+1if

y^i\(g\)=yi\\hat\{y\}\_\{i\}^\{\(g\)\}=y\_\{i\}, else

−1\-1
20:Compute advantage

Ai\(g\)=\(Ri\(g\)−meang′​Ri\(g′\)\)/stdg′​Ri\(g′\)A\_\{i\}^\{\(g\)\}=\\big\(R\_\{i\}^\{\(g\)\}\-\\mathrm\{mean\}\_\{g^\{\\prime\}\}R\_\{i\}^\{\(g^\{\\prime\}\)\}\\big\)/\\mathrm\{std\}\_\{g^\{\\prime\}\}\\,R\_\{i\}^\{\(g^\{\\prime\}\)\}
21:endfor

22:Update

θ\\thetaby minimizing

ℒGRPO​\(θ;ℬ\)\\mathcal\{L\}\_\{\\text\{GRPO\}\}\(\\theta;\\mathcal\{B\}\)⊳\\trianglerightEq\.[3](https://arxiv.org/html/2606.00251#S3.E3)

23:endfor

24:return

θ\\theta
25:endprocedure

## Appendix DExperimental Setup

### D\.1Models

We experiment with two model families\. From the Qwen3 family\[[69](https://arxiv.org/html/2606.00251#bib.bib69)\], we use five scales: 0\.6B, 1\.7B, 4B, 8B, and 14B parameters, to study how CSA varies with model capacity within a single family\. We additionally include OLMo2\[[47](https://arxiv.org/html/2606.00251#bib.bib47)\]at 7B and 13B parameters to assess generalization across model families\. For the SFT Teacher strategy, we use Qwen3\-235B\-A22B as the teacher model\.

### D\.2Datasets

We evaluate on two domains:MathandScience\. Section[2\.2](https://arxiv.org/html/2606.00251#S2.SS2)uses the Math test set\. Our main results in Section[4](https://arxiv.org/html/2606.00251#S4)cover both domains\.

##### Math\.

We aggregate problems from three sources spanning a wide difficulty spectrum: GSM8K\[[13](https://arxiv.org/html/2606.00251#bib.bib13)\]\(grade\-school arithmetic word problems\), MATH500\[[41](https://arxiv.org/html/2606.00251#bib.bib41)\]\(competition\-style problems across algebra, geometry, number theory, etc\.\), and AIME\[[61](https://arxiv.org/html/2606.00251#bib.bib61)\]\(American Invitational Mathematics Examination problems from 1983 to 2024\)\. All problems have open\-ended numerical or symbolic answers\. Table[6\(a\)](https://arxiv.org/html/2606.00251#A4.T6.st1)summarizes the size of each source and the train/test split used in our experiments\.

##### Science\.

We use multiple\-choice questions from MMLU\-Pro\[[63](https://arxiv.org/html/2606.00251#bib.bib63)\], restricted to four science\-relevant categories:biology,chemistry,health, andphysics\. Each question has about 10 answer options, and we feed the question together with all options as the query to the model\. Table[6\(b\)](https://arxiv.org/html/2606.00251#A4.T6.st2)reports the per\-category counts and the overall train/test split\.

Table 6:Dataset composition\.\(a\)MathSource\#Train\#Test\#TotalGSM8K10003001300MATH500400100500AIME86090950Total22604902750
\(b\)ScienceCategory\#Train\#Test\#TotalBiology590127717Chemistry9322001132Health674144818Physics10702291299Total32667003966

### D\.3CSA Inference Setup

We use vLLM\[[38](https://arxiv.org/html/2606.00251#bib.bib38)\]for inference across all evaluated models\. Sampling parameters are kept consistent across models:

- •Temperature: 1\.0
- •Top\-pp: 1\.0
- •Top\-kk: \-1
- •Number of samples per query: 1

For closed\-weight models \(GPT\-5, Gemini 3\.1\), we query the official APIs provided by[OpenAI](https://platform.openai.com/docs/api-reference)and[Google](https://ai.google.dev/api)\.

### D\.4Label Construction

Because label quality is critical for both SFT supervision and RL reward computation\. For each query, we draw 5 independent samples from the target model \(the model being trained\) and decide itsSelf\-SOLVE/DELEGATElabel as follows\.

##### Math: any\-correct\.

We assignyi=Self\-SOLVEy\_\{i\}=\\textsc\{Self\-SOLVE\}if*any*of the 5 responses matches the ground\-truth answer; otherwiseyi=DELEGATEy\_\{i\}=\\textsc\{DELEGATE\}\. The intuition is that for open\-ended answers, even a single correct response indicates the model possesses the requisite knowledge\. The probability of producing a correct open\-ended answer by chance is negligible, so this lenient criterion does not introduce false positives\.

##### Science: majority\-correct\.

For multiple\-choice items, the model can get the correct answer purely by chance\. We therefore require a stricter criterion:yi=Self\-SOLVEy\_\{i\}=\\textsc\{Self\-SOLVE\}only if the model answers correctly in*at least 3 out of 5*samples\. To further guard against position bias and pattern\-matching on option order, we independently shuffle the answer options for each of the 5 samples\. Together, these choices ensure that the label reflects genuine understanding rather than chance\.

### D\.5Evaluation Metrics

##### Capability Discrimination Score \(CDS\)\.

The intuition behind CDS is: a model with stronger CSA should produce predictions whose two groups,Self\-SOLVEandDELEGATE, exhibit a larger gap in problem\-solving accuracy\. In other words, if CSA is well\-calibrated, queries the model labelsSelf\-SOLVEshould be answered correctly with high accuracy, while those it labelsDELEGATEshould be answered correctly with low accuracy\. Concretely, we partition the test set by the model’s predicted label and letpSp\_\{S\}andpDp\_\{D\}denote the empirical solve accuracy within theSelf\-SOLVEandDELEGATEgroups, respectively, with group sizesnSn\_\{S\}andnDn\_\{D\}\. CDS is the \(unpooled\) two\-proportionzz\-statistic

C​D​S=pS−pDpS​\(1−pS\)nS\+pD​\(1−pD\)nD,CDS=\\frac\{p\_\{S\}\-p\_\{D\}\}\{\\sqrt\{\\dfrac\{p\_\{S\}\(1\-p\_\{S\}\)\}\{n\_\{S\}\}\+\\dfrac\{p\_\{D\}\(1\-p\_\{D\}\)\}\{n\_\{D\}\}\}\},\(5\)which normalizes the accuracy gap by its standard error\. A larger CDS indicates a sharper, more statistically reliable separation between what the model can and cannot solve\.

*Example:*suppose the model labelsnS=200n\_\{S\}=200queries asSelf\-SOLVE, correctly answers 170 of them \(pS=170/200=0\.85p\_\{S\}=170/200=0\.85\), and labelsnD=100n\_\{D\}=100queries asDELEGATE, correctly answers 20 of them \(pD=20/100=0\.20p\_\{D\}=20/100=0\.20\)\. Then

C​D​S=0\.85−0\.200\.85⋅0\.15200\+0\.20⋅0\.80100≈13\.7\.CDS=\\frac\{0\.85\-0\.20\}\{\\sqrt\{\\dfrac\{0\.85\\cdot 0\.15\}\{200\}\+\\dfrac\{0\.20\\cdot 0\.80\}\{100\}\}\}\\approx 13\.7\.

##### Macro F1 \(M\-F1\)\.

M\-F1 is the unweighted mean of the per\-class F1 scores for theSelf\-SOLVEandDELEGATEclasses,

M−F​1=12​\(F1Self\-SOLVE\+F1DELEGATE\),M\-F1=\\tfrac\{1\}\{2\}\\left\(\\mathrm\{F1\}\_\{\\textsc\{Self\-SOLVE\}\}\+\\mathrm\{F1\}\_\{\\textsc\{DELEGATE\}\}\\right\),\(6\)where each per\-class F1 is the harmonic mean of precision and recall on that class:

F1c=2​Precc⋅ReccPrecc\+Recc,c∈\{Self\-SOLVE,DELEGATE\}\.\\mathrm\{F1\}\_\{c\}=\\frac\{2\\,\\mathrm\{Prec\}\_\{c\}\\cdot\\mathrm\{Rec\}\_\{c\}\}\{\\mathrm\{Prec\}\_\{c\}\+\\mathrm\{Rec\}\_\{c\}\},\\qquad c\\in\\\{\\textsc\{Self\-SOLVE\},\\textsc\{DELEGATE\}\\\}\.\(7\)Averaging the two per\-class F1 scores with equal weight gives a balanced aggregate that is robust to class imbalance betweenSelf\-SOLVEandDELEGATE\.

*Example:*on a test set of 100 queries with 60 trueSelf\-SOLVEand 40 trueDELEGATE, suppose the model predictsSelf\-SOLVEon 70 queries \(50 correct, 20 false\) andDELEGATEon 30 queries \(20 correct, 10 false\)\. ThenPrecSelf\-SOLVE=50/70≈0\.71\\mathrm\{Prec\}\_\{\\textsc\{Self\-SOLVE\}\}=50/70\\approx 0\.71,RecSelf\-SOLVE=50/60≈0\.83\\mathrm\{Rec\}\_\{\\textsc\{Self\-SOLVE\}\}=50/60\\approx 0\.83, givingF1Self\-SOLVE≈0\.77\\mathrm\{F1\}\_\{\\textsc\{Self\-SOLVE\}\}\\approx 0\.77\. Similarly,PrecDELEGATE=20/30≈0\.67\\mathrm\{Prec\}\_\{\\textsc\{DELEGATE\}\}=20/30\\approx 0\.67,RecDELEGATE=20/40=0\.50\\mathrm\{Rec\}\_\{\\textsc\{DELEGATE\}\}=20/40=0\.50, givingF1DELEGATE≈0\.57\\mathrm\{F1\}\_\{\\textsc\{DELEGATE\}\}\\approx 0\.57\. ThusM−F​1=12​\(0\.77\+0\.57\)≈0\.67M\-F1=\\tfrac\{1\}\{2\}\(0\.77\+0\.57\)\\approx 0\.67\.

##### Capability Ratio \(CR\)\.

To verify that CSA training preserves the model’s original capabilities, we report CR, defined as the percentage of the original model’s problem\-solving accuracy retained after CSA training:

C​R=AccpostAccpre×100%,CR=\\frac\{\\text\{Acc\}\_\{\\text\{post\}\}\}\{\\text\{Acc\}\_\{\\text\{pre\}\}\}\\times 100\\%,\(8\)whereAccpre\\text\{Acc\}\_\{\\text\{pre\}\}andAccpost\\text\{Acc\}\_\{\\text\{post\}\}denote the model’s problem\-solving accuracy before and after CSA training, respectively\. To reduce variance from stochastic decoding, bothAccpre\\text\{Acc\}\_\{\\text\{pre\}\}andAccpost\\text\{Acc\}\_\{\\text\{post\}\}are averaged over 5 independent runs\.

*Example:*if the model solves 60% of the test queries before CSA training \(Accpre=0\.60\\text\{Acc\}\_\{\\text\{pre\}\}=0\.60\) and 57% after \(Accpost=0\.57\\text\{Acc\}\_\{\\text\{post\}\}=0\.57\), thenC​R=0\.57/0\.60×100%=95%CR=0\.57/0\.60\\times 100\\%=95\\%\.

### D\.6Training Settings

Table[7](https://arxiv.org/html/2606.00251#A4.T7)reports the hyperparameters for both SFT and GRPO\. SFT and GRPO both use full\-parameter updates, implemented with the TRL library222[https://github\.com/huggingface/trl](https://github.com/huggingface/trl)on top of DeepSpeed ZeRO\-3\[[51](https://arxiv.org/html/2606.00251#bib.bib51)\]\.

Table 7:Training hyperparameters for SFT and GRPO\. Both use full\-parameter updates\.HyperparameterSFTGRPO*Optimization*OptimizerAdamWAdamWLearning rate1×10−51\\times 10^\{\-5\}1×10−61\\times 10^\{\-6\}LR schedulecosinecosineWarmup ratio0\.10\.1Weight decay0\.010\.01Epochs1 / 3222We train for 1 epoch on Math and 3 epochs on Science\. The only exception is Qwen3\-0\.6B, for which longer training causes the model to collapse into always selecting DELEGATE; therefore we apply early stopping and train it for 60 steps\.1 / 3222We train for 1 epoch on Math and 3 epochs on Science\. The only exception is Qwen3\-0\.6B, for which longer training causes the model to collapse into always selecting DELEGATE; therefore we apply early stopping and train it for 60 steps\.Per\-device batch size816Gradient accumulation11Precisionbf16bf16*GRPO\-specific*Rollouts per prompt \(GG\)—16KL coefficient \(β\\beta\)—0\.01Clipping threshold \(ϵ\\epsilon\)—0\.2Sampling temperature—1\.0Top\-pp—1\.0Rollout backend—vLLM*Training framework*FrameworkTRLTRLDistributed strategyDeepSpeed ZeRO\-3DeepSpeed ZeRO\-3CPU offloadFalseFalse##### DFW Hyperparameters\.

For DFW training, we use the same hyperparameter setup as in Table[7](https://arxiv.org/html/2606.00251#A4.T7), except for the number of training steps, which we report in Table[8](https://arxiv.org/html/2606.00251#A4.T8)for each model\. Because the size of the diversified subset varies across models, the training schedule differs accordingly: smaller models yield larger subsets and are typically early\-stopped before a full epoch, while larger models, with smaller subsets, are trained for 1–2 epochs\. This keeps the total number of training steps roughly comparable across model sizes\. The exception is Qwen\-0\.6B on Science, where training collapses early; as noted in the epoch footnote of Table[7](https://arxiv.org/html/2606.00251#A4.T7), we also early\-stop the warm\-up phase, leaving only 10 steps\.

##### Training Pipeline across Model Families\.

Another point worth noting is the difference in training pipelines across model families\. For Qwen3 models, we follow the full two\-stage RLVR pipeline described in Section[3\.2](https://arxiv.org/html/2606.00251#S3.SS2)\. For OLMo2 models, we omit the DFW stage, since they already produce sufficiently diverse responses during rollout as shown in Table[8](https://arxiv.org/html/2606.00251#A4.T8)\. This is consistent with Figure[1](https://arxiv.org/html/2606.00251#S2.F1)in Section[2\.2](https://arxiv.org/html/2606.00251#S2.SS2): while Qwen3 models exhibitSelf\-SOLVErates close to 100%, OLMo2 models show noticeably lowerSelf\-SOLVErates, indicating greater rollout diversity without explicit intervention\.

Table 8:Subset sizes and training steps for DFW across different models on Math and Science, where*Subset Size*denotes the number of samples selected by DFW from the full training set\.MathScienceModelSubset SizeTraining StepsSubset SizeTraining StepsQwen3\-0\.6B1528150276410Qwen3\-1\.7B5441501781100Qwen3\-4B18346111150Qwen3\-8B16542302152Qwen3\-14B14857171129OLMo2\-7B1832\-3131\-OLMo2\-13B2035\-2977\-
##### Hardware\.

All experiments are run on NVIDIA B200 GPUs\. SFT runs use a single B200, except for Qwen3\-14B and OLMo2\-13B, which use 2×\\timesB200s\. All GRPO runs use 4×\\timesB200s\.

## Appendix EAdditional Results

### E\.1Two Findings on the Diversified Subset

Finding 1: Stronger models yield smaller diversified subsets\.Figure[8](https://arxiv.org/html/2606.00251#A5.F8)shows the number of Math queries that pass the diversification filter as a function of model size\. The subset shrinks sharply with scale: the 0\.6B policy contributes roughly 1,500 diversified samples, the 1\.7B policy roughly 550, and the 4B/8B/14B policies all fall below 200\. This is consistent with the diversification criterion acting as an*uncertainty filter*: as the policy becomes stronger, its 16 rollouts on a given query are more likely to agree \(allSelf\-Solveor allDelegate\), and fewer queries survive the “both labels present” condition\. Larger models are simply more internally consistent about what they can and cannot do, and the per\-step cost of DFW decreases naturally with scale\.

Finding 2: Rollout diversity emerges primarily on queries the model should delegate\.Figure[8](https://arxiv.org/html/2606.00251#A5.F8)contrasts the label distribution of the diversified subset \(solid\) against the full training set \(hatched\)\. The comparison against the full set is what makes this finding meaningful: since the diversified subset is by construction a subset of the full training data, if the model had no preference about*which*queries trigger rollout disagreement, theSelf\-Solve/Delegateproportions in the two distributions should roughly match\. Instead, we observe a clear and systematic shift—across all five model sizes, the diversified subset is skewed towardDelegate\(57%–70%\), compared to a 20%–50%Delegaterate in the full data, and the gap*widens*with scale\. This shift indicates that rollout diversity does not arise uniformly across the data, but emerges preferentially on queries that lie*beyond*the model’s reliable solving capability\. On clearly\-within\- and clearly\-beyond\-capability queries the model’s 16 rollouts tend to agree; the boundary cases—where the model*should*delegate but is occasionally tempted to attempt—are exactly what surface in the diversified subset, and exactly where DFW’s training signal is most needed\.

![Refer to caption](https://arxiv.org/html/2606.00251v1/Figures/dfw_subset_size_by_model.png)Figure 7:Number of samples per model size on Math whose 16 rollouts during DFW yield bothSelf\-SOLVEandDELEGATEresponses\.
![Refer to caption](https://arxiv.org/html/2606.00251v1/Figures/dfw_subset_vs_full_distribution.png)Figure 8:Ground\-truth label distribution \(Self\-SOLVEvs\.DELEGATE\) in the diversified subset during DFW \(solid\) compared to the full training set \(hatched\)\.

### E\.2Case Studies

To illustrate concretely how RLVR shapes CSA, we present case studies Math and Science\.

##### Math Problem

As shown in Figure[9](https://arxiv.org/html/2606.00251#A5.F9), we provide the complete reasoning traces for the math case study presented in the main text\. The Vanilla Qwen3\-4B model confidently chooses toSelf\-SOLVE, but produces an incorrect answer across all five sampled responses\. In contrast, the RLVR\-trained model recognizes that the problem is beyond its reliable capability and chooses toDELEGATE\. This example illustrates how RLVR teaches the model to recognize when a query exceeds its competence\.

![Refer to caption](https://arxiv.org/html/2606.00251v1/x2.png)Figure 9:Full reasoning traces for the math case study: vanilla Qwen3\-4B vs\. RLVR\-trained\.
##### Science Problem

Figure[10](https://arxiv.org/html/2606.00251#A5.F10)presents an additional case study on a science question, where the same pattern emerges: the vanilla model answers confidently but incorrectly, while the RLVR\-trained model recognizes its uncertainty and delegates\.

![Refer to caption](https://arxiv.org/html/2606.00251v1/x3.png)Figure 10:Case study on a science question: vanilla Qwen3\-4B vs\. RLVR\-trained\.

### E\.3Applications of CSA

#### E\.3\.1Self\-Routing

We provide illustrative figures for*self\-routing*in Figure[11](https://arxiv.org/html/2606.00251#A5.F11)\. Formally, given a queryxx, the local model first produces a CSA decisiony^=ℳlocalCSA​\(x\)∈\{Self\-SOLVE,DELEGATE\}\\hat\{y\}=\\mathcal\{M\}\_\{\\text\{local\}\}^\{\\text\{CSA\}\}\(x\)\\in\\\{\\textsc\{Self\-SOLVE\},\\textsc\{DELEGATE\}\\\}, and the routed answer is

a^​\(x\)=\{ℳlocal​\(x\),y^=Self\-SOLVE,ℳcloud​\(x\),y^=DELEGATE\.\\hat\{a\}\(x\)=\\begin\{cases\}\\mathcal\{M\}\_\{\\text\{local\}\}\(x\),&\\hat\{y\}=\\textsc\{Self\-SOLVE\},\\\\ \\mathcal\{M\}\_\{\\text\{cloud\}\}\(x\),&\\hat\{y\}=\\textsc\{DELEGATE\}\.\\end\{cases\}\(9\)This setup exposes a direct cost\-accuracy trade\-off between the two extremes of local\-only inference \(cheap but limited by the local model’s capability\) and cloud\-only inference \(accurate but expensive on every query\)\. An ideal CSA policy delegates exactly the queries the local model cannot solve, approaching cloud\-only accuracy at a fraction of the cost\.

![Refer to caption](https://arxiv.org/html/2606.00251v1/x4.png)Figure 11:Illustration of*self\-routing*\. Upon receiving a user query x, a local model equipped with CSA assesses whether the query falls within its reliable solving capability and decides between two actions:Self\-SOLVE, where the local model answers directly, orDELEGATE, where the query is routed to a cloud model for inference\.
#### E\.3\.2Self\-Guided Data Selection

![Refer to caption](https://arxiv.org/html/2606.00251v1/x5.png)Figure 12:Overview of CSA\-based data selection\. A CSA\-equipped model scores a fresh candidate pool, prioritizing examples it would delegate while filtering out extreme outliers\. the selected subset is then used to fine\-tune the model\.Our main data\-selection experiment tests whether examples preferred by an RLVR\-trained CSA model are better fine\-tuning data than examples selected by standard data\-selection baselines\. We begin with a base modelM0M\_\{0\}and train a CSA modelMRLVRM\_\{\\mathrm\{RLVR\}\}to decide whether a problem should be answered directly or delegated\. After training, the model is used as a scorer: it is applied to a fresh candidate training pool

𝒟pool=\{\(xi,yi\)\}i=1N,\\mathcal\{D\}\_\{\\mathrm\{pool\}\}=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{N\},\(10\)
and assigns each candidate a delegation\-derived score\. Letziselfz\_\{i\}^\{\\mathrm\{self\}\}andzidelz\_\{i\}^\{\\mathrm\{del\}\}denote the logits assigned byRRLVRR\_\{\\mathrm\{RLVR\}\}to theSelf\_SolveandDelegatedecision tokens for examplexix\_\{i\}\. We convert these logits into a normalized delegation probability

pidel=exp⁡\(zidel\)exp⁡\(zidel\)\+exp⁡\(ziself\)\.p\_\{i\}^\{\\mathrm\{del\}\}=\\frac\{\\exp\(z\_\{i\}^\{\\mathrm\{del\}\}\)\}\{\\exp\(z\_\{i\}^\{\\mathrm\{del\}\}\)\+\\exp\(z\_\{i\}^\{\\mathrm\{self\}\}\)\}\.\(11\)
Intuitively,pidelp\_\{i\}^\{\\mathrm\{del\}\}estimates how strongly the trained CSA router believes thatM0M\_\{0\}should not solvexix\_\{i\}directly\.

A naive delegate\-only selector would choose examples with the largestpidelp\_\{i\}^\{\\mathrm\{del\}\}\. However, this can over\-select extreme examples that are far beyond the model’s current capabilities or are noisy, ambiguous, or mismatched to the target distribution\. To avoid this, we use a*delegate\-based loss\-bandpass*selector\. First, we compute the base model’s supervised loss on each candidate:

ℓi=−1\|yi\|​∑t=1\|yi\|log⁡pM0​\(yi,t∣xi,yi,<t\)\.\\ell\_\{i\}=\-\\frac\{1\}\{\|y\_\{i\}\|\}\\sum\_\{t=1\}^\{\|y\_\{i\}\|\}\\log p\_\{M\_\{0\}\}\(y\_\{i,t\}\\mid x\_\{i\},y\_\{i,<t\}\)\.\(12\)
We then retain only examples whose loss lies within a middle band:

ℬα,β=\{i:Qα​\(\{ℓj\}j=1N\)≤ℓi≤Qβ​\(\{ℓj\}j=1N\)\},\\mathcal\{B\}\_\{\\alpha,\\beta\}=\\left\\\{i:Q\_\{\\alpha\}\(\\\{\\ell\_\{j\}\\\}\_\{j=1\}^\{N\}\)\\leq\\ell\_\{i\}\\leq Q\_\{\\beta\}\(\\\{\\ell\_\{j\}\\\}\_\{j=1\}^\{N\}\)\\right\\\},\(13\)
whereQαQ\_\{\\alpha\}andQβQ\_\{\\beta\}are empirical loss quantiles\. This removes examples that are already too easy forM0M\_\{0\}as well as extreme high\-loss outliers\. Among the remaining examples, we rank by delegation probability:

siDLB=pidel⋅𝟏​\{i∈ℬα,β\}\.s\_\{i\}^\{\\mathrm\{DLB\}\}=p\_\{i\}^\{\\mathrm\{del\}\}\\cdot\\mathbf\{1\}\\\{i\\in\\mathcal\{B\}\_\{\\alpha,\\beta\}\\\}\.\(14\)
For a target selection budgetkk, the selected subset is

𝒮DLB​\(k\)=TopKi∈\{1,…,N\}⁡\(siDLB,k\)\.\\mathcal\{S\}\_\{\\mathrm\{DLB\}\}\(k\)=\\operatorname\{TopK\}\_\{i\\in\\\{1,\\ldots,N\\\}\}\\left\(s\_\{i\}^\{\\mathrm\{DLB\}\},k\\right\)\.\(15\)
We then fine\-tune the model on𝒮DLB​\(k\)\\mathcal\{S\}\_\{\\mathrm\{DLB\}\}\(k\)\. Figure[12](https://arxiv.org/html/2606.00251#A5.F12)provides an overview of the method\.

For the science\-pool experiments, we trainM0M\_\{0\}on MMLU\-Pro\[[63](https://arxiv.org/html/2606.00251#bib.bib63)\]to getMRLVRM\_\{\\mathrm\{RLVR\}\}\. The candidate pool is MedMCQA\[[49](https://arxiv.org/html/2606.00251#bib.bib49)\]\. We randomly sampleN=50,000N=50\{,\}000examples from the MedMCQA training split and score them using our method and all baselines\. We conduct experiments on Qwen3\-1\.7B and Qwen3\-4B\[[69](https://arxiv.org/html/2606.00251#bib.bib69)\]\. We use loss\-bandpass quantilesQα=0\.2,Qβ=0\.8Q\_\{\\alpha\}=0\.2,Q\_\{\\beta\}=0\.8\. We compare delegate\-based loss\-bandpass against random selection, high\-loss selection, low\-loss selection, and development\-set similarity selection\. We also include an all\-samples condition, which fine\-tunes on the full 50,000\-example candidate pool\. For development\-set similarity, we reserve a held\-out set

𝒟dev=\{\(xjdev,yjdev\)\}j=1m\\mathcal\{D\}\_\{\\mathrm\{dev\}\}=\\\{\(x\_\{j\}^\{\\mathrm\{dev\}\},y\_\{j\}^\{\\mathrm\{dev\}\}\)\\\}\_\{j=1\}^\{m\}\(16\)
ofm=500m=500randomly selected MedMCQA training examples that are excluded from the candidate pool\.

The baseline selectors are defined as follows\. Random selection samples a subset uniformly without replacement:

𝒮rand​\(k\)∼Unif⁡\(\{𝒮⊂𝒟pool:\|𝒮\|=k\}\)\.\\mathcal\{S\}\_\{\\mathrm\{rand\}\}\(k\)\\sim\\operatorname\{Unif\}\\left\(\\\{\\mathcal\{S\}\\subset\\mathcal\{D\}\_\{\\mathrm\{pool\}\}:\|\\mathcal\{S\}\|=k\\\}\\right\)\.\(17\)
High\-loss selection ranks examples by the base model loss in Eq\.[12](https://arxiv.org/html/2606.00251#A5.E12)and selects the largest\-loss examples:

𝒮high​\(k\)=TopKi⁡\(ℓi,k\)\.\\mathcal\{S\}\_\{\\mathrm\{high\}\}\(k\)=\\operatorname\{TopK\}\_\{i\}\\left\(\\ell\_\{i\},k\\right\)\.\(18\)
This baseline tests whether our method is simply performing hard\-example mining\. Low\-loss selection selects the examples with the smallest base\-model loss:

𝒮low​\(k\)=TopKi⁡\(−ℓi,k\)\.\\mathcal\{S\}\_\{\\mathrm\{low\}\}\(k\)=\\operatorname\{TopK\}\_\{i\}\\left\(\-\\ell\_\{i\},k\\right\)\.\(19\)
This baseline tests whether improvements can instead be obtained by emphasizing examples that are already easy and likely to be clean\.

For development\-set similarity, we embed each candidatexix\_\{i\}and each development examplexjdevx\_\{j\}^\{\\mathrm\{dev\}\}using a fixed sentence encoderf​\(⋅\)f\(\\cdot\)\. We compute the maximum cosine similarity between each candidate and the held\-out development set:

sisim=max1≤j≤m⁡f​\(xi\)⊤​f​\(xjdev\)‖f​\(xi\)‖2​‖f​\(xjdev\)‖2\.s\_\{i\}^\{\\mathrm\{sim\}\}=\\max\_\{1\\leq j\\leq m\}\\frac\{f\(x\_\{i\}\)^\{\\top\}f\(x\_\{j\}^\{\\mathrm\{dev\}\}\)\}\{\\\|f\(x\_\{i\}\)\\\|\_\{2\}\\,\\\|f\(x\_\{j\}^\{\\mathrm\{dev\}\}\)\\\|\_\{2\}\}\.\(20\)
The selected subset is then

𝒮sim​\(k\)=TopKi⁡\(sisim,k\)\.\\mathcal\{S\}\_\{\\mathrm\{sim\}\}\(k\)=\\operatorname\{TopK\}\_\{i\}\\left\(s\_\{i\}^\{\\mathrm\{sim\}\},k\\right\)\.\(21\)
This baseline controls for the possibility that delegate\-based selection works only because it selects examples similar to the evaluation distribution\.

Finally, the all\-samples condition uses

𝒮all=𝒟pool,\\mathcal\{S\}\_\{\\mathrm\{all\}\}=\\mathcal\{D\}\_\{\\mathrm\{pool\}\},\(22\)
and therefore measures the performance obtained when no data selection is performed\. All selected subsets of sizekkare fine\-tuned using the same optimization hyperparameters, number of epochs, and training budget, so differences in downstream performance can be attributed to the selection rule rather than to changes in compute or training procedure\.

Given the scored candidate pool, we run fixed\-budget SFT experiments\. For each selection method, we fine\-tune on one of seven budget fractions of the 50,000\-example pool: 5%, 10%, and 20% corresponding to 2,500, 5,000, and 10,000 selected examples\. Fine\-tuning is performed with full\-parameter training on Nvidia L40S nodes\. The main hyperparameters are a learning rate of5×10−55\\times 10^\{\-5\}, per\-device batch size 4, maximum sequence length 4096, maximum training steps 2000, and bfloat16 training\. Evaluation is performed on the MedMCQA validation split rather than the official test split\. Each run evaluates on 500 randomly sampled validation examples a maximum generation length of 256 tokens\. The primary metric is validation accuracy after fine\-tuning\.

## Appendix FLimitations

While our results demonstrate that CSA is a teachable and transferable trait, several aspects merit further investigation\.

First, our experiments focus on the Qwen3 and OLMo2 model families and on math and science domains; extending the study to additional model families and to more diverse domains such as coding, commonsense reasoning, and multilingual tasks would help further characterize the generality of CSA\.

Second, we frame CSA as a binarySelf\-Solve/Delegatedecision, which isolates the core capability\-assessment problem; richer action spaces \(e\.g\., multi\-tier routing, partial answers, or clarification requests\) are a natural next step\.

Third, the ground\-truth CSA labels are constructed by sampling\-based probing, which is effective but introduces some sensitivity to the sampling budget and aggregation rule; exploring alternative labeling strategies could further improve label quality\.

Fourth, our data\-selection experiments are constrained by the difficulty of constructing clean, contamination\-free candidate pools for modern LLMs\. Since many open\-weight models are trained on large and only partially documented internet\-scale corpora, it is difficult to determine whether a benchmark appeared directly in pretraining or whether the model has seen closely related QA content\. Thus, our data\-selection results should not be interpreted as a fully controlled study of learning from entirely novel data\. At the same time, this setting resembles many realistic deployment scenarios: a model may have seen some related material during pretraining, but still fail on particular questions or subdomains\. In this sense, based on results, MedMCQA provides a useful testbed for whether CSA can identify examples that are valuable for continued fine\-tuning under partial prior exposure\.