REIN: Bridging the Gap between Reasoning and Reliability via Reflection and Abstention Alignment

# Bridging the Gap between Reasoning and Reliability via Reflection and Abstention Alignment
Source: [https://arxiv.org/html/2608.07931](https://arxiv.org/html/2608.07931)
Zhengze Huang1,\*Luyang Yu2,\*Di Hong1Xinzhe Huang1 Wanyu Lin3Zhixuan Chu1Zhan Qin1,†Tianhang Zheng1,† 1The State Key Laboratory of Blockchain and Data Security, Zhejiang University 2Fudan University 3The Hong Kong Polytechnic University \*Equal contribution†Corresponding author

###### Abstract

Large reasoning models \(LRMs\) are prone to hallucination, which undermines their reliability and poses challenges for safe deployment\. Hallucinations in LRMs arise from two distinct failure sources: reasoning hallucination, where flawed inference steps propagate to an incorrect conclusion, and knowledge hallucination, where the model lacks the requisite factual knowledge to answer the query\. To address reasoning hallucination, we propose REIN, an alignment framework that trains LRMs to produce a structured reasoning sequence,<think\>→\\rightarrow<reflection\>→\\rightarrow<answer\>, enabling explicit self\-reflection before committing to a final answer\. To address knowledge hallucination, REIN introduces a reward mechanism that encourages explicit abstention \(e\.g\., "I don’t know"\) when none of the sampled reasoning chains yields a correct answer, allowing the model to refrain from unsupported predictions\. Extensive evaluations on mathematical and commonsense reasoning benchmarks show that REIN consistently improves selective accuracy, reduces incorrect\-but\-self\-endorsed responses, and maintains high coverage compared with competitive baselines\. Notably, REIN achieves these gains within a single forward pass, without requiring process supervision, inference\-time controllers, external search, or multi\-round critiques\. Experiments on multiple backbones show that REIN reduces the hallucination proxy by58∼72%58\\sim 72\\%relative to the base models while maintaining86∼91%86\\sim 91\\%average coverage, and improves selective accuracy on attempted questions by6\.6∼14\.2%6\.6\\sim 14\.2\\%\.

![Refer to caption](https://arxiv.org/html/2608.07931v1/images/flowchart.png)Figure 1:Overview of the REIN training pipeline: reflection\-augmented completions feed into GRPO with reflection and boundary\-aware abstention rewards\.## 1Introduction

Large reasoning models \(LRMs\) such as DeepSeek\-R1\(DeepSeek\-AI,[2025](https://arxiv.org/html/2608.07931#bib.bib15)\)and Qwen3\(Qwen Team,[2025](https://arxiv.org/html/2608.07931#bib.bib14)\)have made substantial progress on mathematics, scientific reasoning, and programming\(Hendryckset al\.,[2021a](https://arxiv.org/html/2608.07931#bib.bib3); Zhenget al\.,[2025](https://arxiv.org/html/2608.07931#bib.bib19); Chen and others,[2025](https://arxiv.org/html/2608.07931#bib.bib18)\)by eliciting explicit chains of thought\(Weiet al\.,[2022](https://arxiv.org/html/2608.07931#bib.bib1)\)\. Yet explicit reasoning does not eliminate hallucination—it often increases the apparent plausibility of unsupported conclusions, producing outputs that seem coherent and persuasive but still contain factual errors\(Minet al\.,[2023](https://arxiv.org/html/2608.07931#bib.bib10)\)or internally inconsistent inference\(Chenget al\.,[2025](https://arxiv.org/html/2608.07931#bib.bib80)\)\.

We identify two primary failure modes of hallucination: reasoning\-level hallucination and knowledge\-level hallucination\. In*reasoning\-level hallucination*, the model possesses the requisite knowledge, but a particular sampled trajectory contains a flawed inference step that propagates to a wrong answer\. This failure mode is self\-recoverable, since a different sample or a verified self\-check could still yield the correct answer\. In*knowledge\-level hallucination*, by contrast, the model lacks the requisite knowledge under the current policy and sampling budget, and any additional reasoning merely rationalizes an unsupported prediction\. Thus, additional reasoning or reflection cannot rescue the answer, and the only safe action is to abstain\. These two modes demand fundamentally different interventions, and conflating them is a central source of mis\-calibration in current alignment recipes\.

Existing approaches address at most one of the two failure modes\. Outcome\-level preference alignment such as RLHF\(Ouyanget al\.,[2022](https://arxiv.org/html/2608.07931#bib.bib4)\)and DPO\(Rafailovet al\.,[2023](https://arxiv.org/html/2608.07931#bib.bib5)\)optimizes the final response but provides no explicit supervision over the reasoning trajectory, so reasoning\-level errors that propagate to the final answer remain undetected\. Process\-supervision and self\-correction methods, including step\-level process reward models\(Uesatoet al\.,[2022](https://arxiv.org/html/2608.07931#bib.bib24); Lightmanet al\.,[2024](https://arxiv.org/html/2608.07931#bib.bib25)\), Self\-Refine\(Madaanet al\.,[2023](https://arxiv.org/html/2608.07931#bib.bib21)\), and Reflexion\(Shinnet al\.,[2023](https://arxiv.org/html/2608.07931#bib.bib22)\), focus on reliability assessment but presume that the errors are self\-recoverable—they have no mechanism to distinguish a fixable inference slip from genuinely missing knowledge, and recent evidence shows that without external feedback, models often fail to detect their own errors or even overturn correct answers\(Huanget al\.,[2023](https://arxiv.org/html/2608.07931#bib.bib30); Kamoiet al\.,[2024](https://arxiv.org/html/2608.07931#bib.bib31)\)\. Conversely, abstention and uncertainty\-calibration methods such as rejection tuning\(Xuet al\.,[2024](https://arxiv.org/html/2608.07931#bib.bib7)\), R\-Tuning\(Zhanget al\.,[2024](https://arxiv.org/html/2608.07931#bib.bib37)\), and IDK\-token approaches\(Cohenet al\.,[2024](https://arxiv.org/html/2608.07931#bib.bib39); Chenget al\.,[2024](https://arxiv.org/html/2608.07931#bib.bib8)\)encourage refusal under uncertainty but ignore reasoning repair, and therefore over\-abstain whenever a recoverable reasoning failure is mistaken for an irrecoverable knowledge gap\. It remains an open question how to establish a unified, reliability\-oriented alignment framework that can decide, within a single completion, whether the current reasoning supports a reliable answer or whether the model should abstain\.

We proposeREIN, an alignment framework that teaches an LRM, in a single forward pass, to reflect on its own reasoning before committing to an answer and to abstain when the answer is not adequately supported by the model’s internal knowledge\. REIN trains the model to emit a structured completion<think\>→\\rightarrow<reflection\>→\\rightarrow<answer\>, where the<reflection\>span is a trainable reliability judgment whose stance is aligned, via a centered reflection\-veracity reward, with the verified correctness of the answer\. To handle knowledge\-level failures, REIN further derives a group\-level boundary indicator from the model’s own sampled completions and uses a boundary\-aware IDK reward that rewards abstention only when no sample in the group yields a \(nearly\) correct answer—thereby separating recoverable reasoning failures from genuine knowledge gaps without process supervision, external retrieval, or test\-time controllers\.

Our contributions are threefold:

1. 1\.We propose REIN, a unified single‑pass alignment framework that structures generation as<think\>→\\rightarrow<reflection\>→\\rightarrow<answer\>, enabling LRMs to decide within one forward pass whether the current reasoning should be trusted for finalization or whether the model should abstain, without external controllers or iterative loops\.
2. 2\.We design a centered reflection‑veracity reward to align the reflection span with the correctness of the pre\-reflection draft conclusion, along with a boundary‑aware IDK reward to incentivize abstention when the model lacks sufficient knowledge\.
3. 3\.Extensive experiments across multiple backbones and reasoning benchmarks demonstrate that REIN reduces the hallucination proxy by 58–72% while maintaining 86–91% average coverage, and improves accuracy on attempted questions by 6\.6–14\.2%\.

## 2Related Work

##### Self\-correction and reflection\.

Self\-correction methods push the LLM to self\-critique and revise its own output\. Self\-Refine is an inference\-time framework that iteratively alternates between model\-generated feedback and output revision\(Madaanet al\.,[2023](https://arxiv.org/html/2608.07931#bib.bib21)\)\. SCoRe instead learns multi\-turn self\-correction from on\-policy traces through reinforcement learning\(Kumaret al\.,[2025](https://arxiv.org/html/2608.07931#bib.bib44)\)\. Evidence on intrinsic self\-correction nevertheless shows that self\-generated feedback does not consistently correct errors and may overturn correct answers when no reliable external signal is available\(Huanget al\.,[2023](https://arxiv.org/html/2608.07931#bib.bib30); Kamoiet al\.,[2024](https://arxiv.org/html/2608.07931#bib.bib31); Liet al\.,[2024](https://arxiv.org/html/2608.07931#bib.bib32)\)\. REIN assigns reflection a different role: the<reflection\>span assesses whether the preceding reasoning trajectory supports a reliable draft conclusion\. Its judgment is aligned with the verified correctness of that pre\-reflection conclusion and guides finalization within the same structured completion, without requiring a separate critique–revision loop\.

##### Outcome and process alignment\.

Preference\-based RL such as RLHF\(Ouyanget al\.,[2022](https://arxiv.org/html/2608.07931#bib.bib4)\), Safe RLHF\(Daiet al\.,[2023](https://arxiv.org/html/2608.07931#bib.bib6)\), and DPO\(Rafailovet al\.,[2023](https://arxiv.org/html/2608.07931#bib.bib5)\)aligns model behavior to outcome\-level preferences but provides no supervision on the reasoning trajectory itself\. Rejection\-sampling fine\-tuning and verifier\-driven RL extend this to verifiable\-answer domains via automatic answer checking\(Yuanet al\.,[2023](https://arxiv.org/html/2608.07931#bib.bib41); Shaoet al\.,[2024](https://arxiv.org/html/2608.07931#bib.bib40); DeepSeek\-AI,[2025](https://arxiv.org/html/2608.07931#bib.bib15)\), while process\-reward models supply step\-level signals at the cost of dense human or LLM annotations\(Uesatoet al\.,[2022](https://arxiv.org/html/2608.07931#bib.bib24); Lightmanet al\.,[2024](https://arxiv.org/html/2608.07931#bib.bib25); Huanget al\.,[2025](https://arxiv.org/html/2608.07931#bib.bib87)\)\. Unlike outcome\-only preference RL, which is silent about the reasoning trajectory and process\-supervised reward models, which require fine\-grained step labels and still cannot decide when to abstain\. REIN keeps the GRPO outer loop but redesigns the reward family around*reliability*: a reflection\-veracity reward aligns the model’s self\-assessment with answer correctness, and a boundary\-aware IDK reward injects an abstention signal, all without step\-level supervision\.

##### Uncertainty and abstention\.

R\-Tuning and IDK\-token methods train models to abstain under uncertainty\(Zhanget al\.,[2024](https://arxiv.org/html/2608.07931#bib.bib37); Cohenet al\.,[2024](https://arxiv.org/html/2608.07931#bib.bib39); Chenget al\.,[2024](https://arxiv.org/html/2608.07931#bib.bib8)\)\. TruthRL instead uses a ternary RL objective that distinguishes correct answers, hallucinations, and abstentions\(Weiet al\.,[2026](https://arxiv.org/html/2608.07931#bib.bib83)\)\. REIN differs by separately aligning a pre\-finalization reflection with the correctness of the draft conclusion implied by the reasoning trajectory, and the final answer\-or\-abstain decision with a sampled knowledge boundary\. KnowRL introduces fine\-grained factual supervision over atomic claims in the reasoning trace\(Renet al\.,[2026](https://arxiv.org/html/2608.07931#bib.bib84); Wuet al\.,[2026](https://arxiv.org/html/2608.07931#bib.bib88)\), whereas REIN requires only task\-answer verification\. BAPO studies boundary\-aware rewards for agentic search\(Liuet al\.,[2026](https://arxiv.org/html/2608.07931#bib.bib86); Zhaoet al\.,[2025](https://arxiv.org/html/2608.07931#bib.bib89)\), while BARREL promotes concise, boundary\-aware reasoning to mitigate overthinking\(Yanget al\.,[2026](https://arxiv.org/html/2608.07931#bib.bib53); Xiuet al\.,[2025](https://arxiv.org/html/2608.07931#bib.bib90)\)\. In contrast, REIN jointly aligns reflection and abstention within a single structured completion\.

## 3REIN Framework

### 3\.1Framework Setup

Given a promptx∈𝒳x\\in\\mathcal\{X\}, the policyπθ\\pi\_\{\\theta\}generates a structured completionzz:

z∼πθ\(⋅∣x\)\.z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\.\(1\)Each completion follows the order

<think\>→<reflection\>→<answer\>\.\\text\{\{<think\>\}\}\\rightarrow\\text\{\{<reflection\>\}\}\\rightarrow\\text\{\{<answer\>\}\}\.\(2\)A deterministic parser extracts the three spans fromzz:

\(c,r,y1\):=𝒫​\(z\)\.\(c,r,y\_\{1\}\):=\\mathcal\{P\}\(z\)\.\(3\)
The reasoning trajectory may imply a preliminary conclusion before reflection\. We denote the draft conclusion extracted fromccas

y0:=𝒟x​\(c\),y\_\{0\}:=\\mathcal\{D\}\_\{x\}\(c\),\(4\)where𝒟x\\mathcal\{D\}\_\{x\}is a task\-specific deterministic answer extractor\.

Here,ccis the pre\-reflection reasoning trajectory,y0y\_\{0\}is the draft conclusion implied by that trajectory,rris the model’s reliability assessment conditioned on the preceding trajectory, andy1y\_\{1\}is the final answer generated after reflection\. The reflection assesses whether the draft answery0y\_\{0\}implied by the preceding reasoning is reliable\. The final answery1y\_\{1\}is either a substantive task answer in𝒴\\mathcal\{Y\}or the canonical abstention responseIDK\. Becauserris generated after the reasoning trajectory but beforey1y\_\{1\},it influences the subsequent finalization decision\.

For any candidate answeryy, we define

T​\(x,y\):=\{1,if​y≠IDK​and​𝒱x​\(y,y⋆​\(x\)\)=1,0,otherwise\.T\(x,y\):=\\begin\{cases\}1,&\\text\{if \}y\\neq\\mathrm\{IDK\}\\text\{ and \}\\mathcal\{V\}\_\{x\}\(y,y^\{\\star\}\(x\)\)=1,\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}\(5\)
The draft answery0y\_\{0\}and the final answery1y\_\{1\}are verified for different purposes\. The reflection\-veracity reward evaluates whether the reflection judgment agrees withT​\(x,y0\)T\(x,y\_\{0\}\), whereas the answer\-correctness and boundary\-aware abstention rewards evaluateT​\(x,y1\)T\(x,y\_\{1\}\)\. Answers mentioned elsewhere in the reflection span are not treated as task answers\. A final answery1=IDKy\_\{1\}=\\texttt\{IDK\}is not counted as a correct task answer and is handled separately by the abstention reward\. Malformed structured outputs are handled by the format rewards described in Section[3\.4](https://arxiv.org/html/2608.07931#S3.SS4)\.

Following the two failure modes described in the Introduction, REIN addresses reasoning hallucination and knowledge hallucination\. For reasoning hallucination, reflection alignment trains the model to judge whether the draft answery0y\_\{0\}formed by the current reasoning is reliable\. This reduces false endorsement, where an incorrect draft answery0y\_\{0\}is nevertheless judged reliable\. For knowledge hallucination, REIN estimates a sampled knowledge boundary from a group of on\-policy completions\. If the group contains no verified\-correct final answery1y\_\{1\}, the model is encouraged to abstain instead of producing an unsupported final answer\. Together, the two signals guide the model’s final answer\-or\-abstain decision iny1y\_\{1\}\.

### 3\.2Reflection Alignment for Reasoning Hallucination

Before generating the final answer, REIN uses the<reflection\>span to assess whether the preceding reasoning trajectory supports a reliable draft conclusiony0y\_\{0\}\. After the completion is generated, REIN verifies the draft conclusiony0y\_\{0\}and rewards the reflection when its reliability judgment matchesT​\(x,y0\)T\(x,y\_\{0\}\)\. The post\-reflection final answery1y\_\{1\}is not used as the target of the reflection\-veracity reward\. This distinction prevents a correct assessment of an unreliable draft trajectory from being penalized when the model subsequently revises its final answer\.

A fixed parser maps the reflectionrrto

s​\(r\)∈\{correct,wrong,uncertain,⊥\},s\(r\)\\in\\\{\\texttt\{correct\},\\texttt\{wrong\},\\texttt\{uncertain\},\\bot\\\},where⊥\\botdenotes a missing, invalid, or ambiguous stance\. For a valid stance, we define the binary reflection judgment as

J​\(r\):=\{1,s​\(r\)=correct,0,s​\(r\)∈\{wrong,uncertain\}\.J\(r\):=\\begin\{cases\}1,&s\(r\)=\\texttt\{correct\},\\\\ 0,&s\(r\)\\in\\\{\\texttt\{wrong\},\\texttt\{uncertain\}\\\}\.\\end\{cases\}\(6\)Here,J​\(r\)=1J\(r\)=1indicates that the model considers the conclusion implied by the preceding reasoning trajectory reliable, whileJ​\(r\)=0J\(r\)=0indicates that it considers the trajectory unreliable\.

The main failure targeted by reflection alignment is false endorsement:

FE​\(x,r,y0\):=𝟙​\[y0∈𝒴\]​\(1−T​\(x,y0\)\)​J​\(r\)\.\\mathrm\{FE\}\(x,r,y\_\{0\}\):=\\mathbb\{1\}\[y\_\{0\}\\in\\mathcal\{Y\}\]\\bigl\(1\-T\(x,y\_\{0\}\)\\bigr\)J\(r\)\.\(7\)False endorsement occurs when the draft conclusion implied by the reasoning trajectory is incorrect but the reflection nevertheless assigns it a reliable stance\. Such over\-confidence may cause the model to preserve an unreliable trajectory during finalization\.

The opposite mismatch is false rejection:

FR​\(x,r,y0\):=T​\(x,y0\)​\(1−J​\(r\)\)\.\\mathrm\{FR\}\(x,r,y\_\{0\}\):=T\(x,y\_\{0\}\)\\bigl\(1\-J\(r\)\\bigr\)\.\(8\)False rejection occurs when a reliable draft conclusion is judged unreliable\. Although the final answery1y\_\{1\}may still be correct, this mismatch can lead to unnecessary revision or abstention\.

The reflection\-veracity reward is defined as

rvrcty​\(x,r,y0\):=\{\+1,s\(r\)≠⊥∧J\(r\)=T\(x,y0\),−1,s\(r\)≠⊥∧J\(r\)≠T\(x,y0\),0,s​\(r\)=⊥\.r\_\{\\mathrm\{vrcty\}\}\(x,r,y\_\{0\}\):=\\begin\{cases\}\+1,&s\(r\)\\neq\\bot\\ \\wedge\\ J\(r\)=T\(x,y\_\{0\}\),\\\\ \-1,&s\(r\)\\neq\\bot\\ \\wedge\\ J\(r\)\\neq T\(x,y\_\{0\}\),\\\\ 0,&s\(r\)=\\bot\.\\end\{cases\}\(9\)Invalid or missing reflection stances receive no veracity reward and are handled by the format rewards described in Section[3\.4](https://arxiv.org/html/2608.07931#S3.SS4)\.

Reflection alignment works at the completion level\. It aligns a reflection judgment with the verified reliability of the draft conclusion implied by the preceding reasoning trajectory\. It does not perform step\-level verification or identify which reasoning step causes an error\. The correctness of the final answery1y\_\{1\}is supervised separately by the answer\-correctness reward, while the final answer\-or\-abstain decision is further guided by the group\-level signal introduced next\.

### 3\.3Boundary\-Aware Abstention Alignment for Knowledge Hallucination

Reflection alignment assesses whether a single reasoning trajectory supports a reliable draft conclusiony0y\_\{0\}, but it cannot determine whether the current policy can produce a verified final answery1y\_\{1\}for the prompt\. REIN therefore uses a group of on\-policy completions to obtain an empirical estimate of the current policy’s knowledge boundary\.

For each promptxx, the current policy samplesKKcompletions:

z\(k\)∼πθ\(⋅∣x\),k=1,…,K\.z^\{\(k\)\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\),\\qquad k=1,\\ldots,K\.\(10\)
Applying𝒫\\mathcal\{P\}and𝒟x\\mathcal\{D\}\_\{x\}to each sampled completion yields

𝒢K​\(x\):=\(\(c\(k\),y0\(k\),r\(k\),y1\(k\)\)\)k=1K,\\mathcal\{G\}\_\{K\}\(x\):=\\left\(\(c^\{\(k\)\},y\_\{0\}^\{\(k\)\},r^\{\(k\)\},y\_\{1\}^\{\(k\)\}\)\\right\)\_\{k=1\}^\{K\},\(11\)where\(c\(k\),r\(k\),y1\(k\)\)=𝒫​\(z\(k\)\)\(c^\{\(k\)\},r^\{\(k\)\},y\_\{1\}^\{\(k\)\}\)=\\mathcal\{P\}\(z^\{\(k\)\}\)andy0\(k\)=𝒟x​\(c\(k\)\)y\_\{0\}^\{\(k\)\}=\\mathcal\{D\}\_\{x\}\(c^\{\(k\)\}\)\.

We mark the prompt aswithinthe sampled boundary if at least one completion produces a verified\-correct final answery1\(k\)y\_\{1\}^\{\(k\)\}\. Otherwise, the prompt is marked asbeyond:

B^K​\(x\):=\{within,∃k∈\{1,…​K\}:T​\(x,y1\(k\)\)=1beyond,otherwise\.\\widehat\{B\}\_\{K\}\(x\):=\\begin\{cases\}\\texttt\{within\},&\\exists\\,k\\in\\\{1,\\ldots K\\\}:T\(x,y\_\{1\}^\{\(k\)\}\)=1\\\\ \\texttt\{beyond\},&\\text\{otherwise\}\.\\end\{cases\}\(12\)
The estimated boundary depends on the current policy, the group sizeKK, and the decoding configuration\. It is therefore an empirical, policy\-relative boundary rather than an oracle label of whether the relevant knowledge is absent from the model parameters\. Larger groups reduce finite\-sampling uncertainty but increase rollout and verification costs\. We useK=16K=16as a practical trade\-off between estimation stability and computational cost\. Appendix G provides the probabilistic interpretation, finite\-sample confidence analysis, and sensitivity results for different rollout budgets\.

REIN uses this boundary to supervise the final answer\-or\-abstain decision iny1y\_\{1\}\. WhenB^K​\(x\)=within\\widehat\{B\}\_\{K\}\(x\)=\\texttt\{within\}, the sampled group contains at least one verified\-correct final answery1\(k\)y\_\{1\}^\{\(k\)\}\. REIN therefore rewards verified\-correct final answers and penalizes unnecessary abstention iny1y\_\{1\}\. WhenB^K​\(x\)=beyond\\widehat\{B\}\_\{K\}\(x\)=\\texttt\{beyond\}, REIN rewardsy1\(k\)=IDKy\_\{1\}^\{\(k\)\}=\\texttt\{IDK\}instead of an unverified substantive final answer\.

For completionkk, the boundary\-aware abstention reward is

ridk\(k\):=\{\+1,B^K​\(x\)=within∧T​\(x,y1\(k\)\)=1,−1,B^K​\(x\)=within∧y1\(k\)=IDK,\+1,B^K​\(x\)=beyond∧y1\(k\)=IDK,0,otherwise\.r\_\{\\mathrm\{idk\}\}^\{\(k\)\}:=\\begin\{cases\}\+1,&\\widehat\{B\}\_\{K\}\(x\)=\\texttt\{within\}\\ \\wedge\\ T\(x,y\_\{1\}^\{\(k\)\}\)=1,\\\\ \-1,&\\widehat\{B\}\_\{K\}\(x\)=\\texttt\{within\}\\ \\wedge\\ y\_\{1\}^\{\(k\)\}=\\texttt\{IDK\},\\\\ \+1,&\\widehat\{B\}\_\{K\}\(x\)=\\texttt\{beyond\}\\ \\wedge\\ y\_\{1\}^\{\(k\)\}=\\texttt\{IDK\},\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}\(13\)
We detect abstention from the parsed<answer\>spany1y\_\{1\}using a fixed set of IDK patterns\. This reward provides a prompt\-level empirical knowledge\-boundary signal\. A single failed completion does not by itself justify abstention when another rollout for the same prompt has already produced a verified\-correct final answer\. Incorrect substantive final answers receive no positive reward fromridkr\_\{\\mathrm\{idk\}\}and are evaluated separately byraccr\_\{\\mathrm\{acc\}\}\. The reflection\-veracity reward separately supervises whether the reflectionrrcorrectly assesses the reliability of the corresponding reasoning trajectory and its implied draft conclusiony0y\_\{0\}\.

Table 1:Comparison of reliability alignment methods\. Sel\.Acc\. denotes accuracy among answered examples, and H\-Proxy measures the fraction of incorrect answers incorrectly judged as reliable\. Eff\.Acc\. denotes overall accuracy over all examples\. Coverage values below 100% are reported as subscripts of Eff\.Acc\., while omitted entries indicate full coverage\. Higher Sel\.Acc\. and Eff\.Acc\. are preferred, whereas lower H\-Proxy is better\.
### 3\.4Joint Reward and GRPO Optimization

REIN first applies a short supervised fine\-tuning stage to initialize the structured output format and the reflection labels used to assess the reliability of the pre\-reflection reasoning trajectory and its implied draft conclusiony0y\_\{0\}\. The resulting policy is then optimized with GRPO over the on\-policy completion groups defined in Section[3\.3](https://arxiv.org/html/2608.07931#S3.SS3)\.

For each completionkk, the total reward combines final\-answer correctness, format control, trajectory\-grounded reflection alignment, and final\-answer abstention alignment:

R\(k\)​\(x\)=\\displaystyle R^\{\(k\)\}\(x\)=\{\}wacc​racc\(k\)\\displaystyle w\_\{\\mathrm\{acc\}\}r\_\{\\mathrm\{acc\}\}^\{\(k\)\}\(14\)\+wxml​rxml\(k\)\+wpres​rpres\(k\)\\displaystyle\+w\_\{\\mathrm\{xml\}\}r\_\{\\mathrm\{xml\}\}^\{\(k\)\}\+w\_\{\\mathrm\{pres\}\}r\_\{\\mathrm\{pres\}\}^\{\(k\)\}\+wvrcty​rvrcty\(k\)\+widk​ridk\(k\)\.\\displaystyle\+w\_\{\\mathrm\{vrcty\}\}r\_\{\\mathrm\{vrcty\}\}^\{\(k\)\}\+w\_\{\\mathrm\{idk\}\}r\_\{\\mathrm\{idk\}\}^\{\(k\)\}\.
For completionkk, the answer\-correctness rewardracc\(k\)r\_\{\\mathrm\{acc\}\}^\{\(k\)\}evaluates the post\-reflection final answery1\(k\)y\_\{1\}^\{\(k\)\}and rewards it when it is verified correct\. The format rewardsrxml\(k\)r\_\{\\mathrm\{xml\}\}^\{\(k\)\}andrpres\(k\)r\_\{\\mathrm\{pres\}\}^\{\(k\)\}enforce the required tag structure and the presence of a non\-empty reflection span, respectively\. The reflection\-veracity rewardrvrcty\(k\)r\_\{\\mathrm\{vrcty\}\}^\{\(k\)\}aligns the reflection judgmentr\(k\)r^\{\(k\)\}with the verified correctness of the corresponding draft conclusiony0\(k\)y\_\{0\}^\{\(k\)\}implied by the preceding reasoning trajectory\. The boundary\-aware rewardridk\(k\)r\_\{\\mathrm\{idk\}\}^\{\(k\)\}aligns the final answer\-or\-abstain action expressed iny1\(k\)y\_\{1\}^\{\(k\)\}with the group\-level sampled knowledge boundary\.

Within GRPO, the rewards of theKKcompletions sampled for the same prompt are normalized to obtain relative advantages:

A^\(k\)=R\(k\)−meanj⁡R\(j\)stdj⁡R\(j\)\+ϵ\.\\widehat\{A\}^\{\(k\)\}=\\frac\{R^\{\(k\)\}\-\\operatorname\{mean\}\_\{j\}R^\{\(j\)\}\}\{\\operatorname\{std\}\_\{j\}R^\{\(j\)\}\+\\epsilon\}\.\(15\)
REIN then applies the standard GRPO objective with KL regularization relative to the reference policy\. Whilervrcty\(k\)r\_\{\\mathrm\{vrcty\}\}^\{\(k\)\}supervises whether the reflection correctly assesses the reliability of the pre\-reflection reasoning trajectory,racc\(k\)r\_\{\\mathrm\{acc\}\}^\{\(k\)\}andridk\(k\)r\_\{\\mathrm\{idk\}\}^\{\(k\)\}supervise final\-answer correctness and abstention iny1\(k\)y\_\{1\}^\{\(k\)\}, respectively\. All reward components contribute to the same scalar reward and update the same policy\. Therefore, trajectory reliability assessment and final answer\-or\-abstain decisions are optimized jointly\.

REIN further uses a two\-phase GRPO curriculum to establish abstention behavior\. In the first phase, the training prompt explicitly presentsy1=IDKy\_\{1\}=\\texttt\{IDK\}as an available final action\. In the second phase, the explicit IDK cue is removed while the boundary\-aware reward is retained\. The policy must then determine whethery1y\_\{1\}should contain a substantive answer orIDK, without a direct IDK instruction\.

At inference time, REIN generates a single structured completion without reference answers, task verifiers, group sampling, or multi\-round critique–revision loops\. Within this single completion, the trained policy forms a draft conclusiony0y\_\{0\}from the reasoning trajectory, assesses its reliability throughrr, and produces a final answery1y\_\{1\}that is either substantive orIDK\.

## 4Experimental Setup

##### Datasets and Evaluation Metrics\.

We evaluate all methods on four benchmarks: GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2608.07931#bib.bib2)\)and MATH\-500\(Hendryckset al\.,[2021b](https://arxiv.org/html/2608.07931#bib.bib55)\)for mathematical reasoning, and StrategyQA\(Gevaet al\.,[2021](https://arxiv.org/html/2608.07931#bib.bib56)\)and ARC\-Challenge\(Clarket al\.,[2018](https://arxiv.org/html/2608.07931#bib.bib57)\)for commonsense reasoning\. GRPO optimization uses only the GSM8K training set; results on the other three benchmarks therefore measure the cross\-task transfer of the learned reliability behavior\.

We report four primary metrics: Selective Accuracy \(Sel\.Acc\.\), Hallucination Proxy \(H\-Proxy\), Coverage \(Cov\.\), and Effective Accuracy \(Eff\.Acc\.\)\.

Sel\.Acc\. measures correctness among substantive final answers\. H\-Proxy measures the proportion of incorrect pre\-reflection draft conclusionsy0y\_\{0\}that are nevertheless positively endorsed by the corresponding reflectionrr\. Specifically, it estimatesPr⁡\[J​\(r\)=1∣T​\(x,y0\)=0\]\\Pr\[J\(r\)=1\\mid T\(x,y\_\{0\}\)=0\], thereby measuring draft\-level false endorsement rather than the overall incidence of incorrect final answers\. Cov\. is the proportion of evaluation examples for which the model produces a substantive final answer, while Eff\.Acc\. is the proportion of all examples that are answered correctly\. Higher Sel\.Acc\., Cov\., and Eff\.Acc\. are better, whereas lower H\-Proxy is better\. Detailed definitions of these metrics are provided in Appendix A\.

##### Models and Baselines\.

We evaluate REIN on two backbone models: Qwen2\.5\-7B\(Qwen Team,[2024](https://arxiv.org/html/2608.07931#bib.bib65)\)and Llama\-3\.1\-8B\(Dubeyet al\.,[2024](https://arxiv.org/html/2608.07931#bib.bib66)\)\. These backbones represent different model families and allow us to test whether REIN generalizes beyond a single architecture\. We use Base and Instruct as checkpoint baselines, IDK Prompting as a training\-free baseline, Reflection\-only as a reflection\-alignment baseline, and R\-Tuning\(Cohenet al\.,[2024](https://arxiv.org/html/2608.07931#bib.bib39)\)as an explicit uncertainty\-modeling baseline\. We also use TruthRL\(Weiet al\.,[2026](https://arxiv.org/html/2608.07931#bib.bib83)\)as a reinforcement\-learning baseline for factuality and abstention\. We reproduce all training\-based baselines using their original objectives and adapt them to our experimental setting\. All methods, including REIN, are evaluated using the same test splits and a unified decoding and evaluation protocol\.For H\-Proxy evaluation, every method is prompted to generate the same<think\>→\\rightarrow<reflection\>→\\rightarrow<answer\>structure within a single autoregressive completion\. The reflection therefore serves as an endogenous pre\-finalization judgment of the model’s own draft conclusiony0y\_\{0\}; no additional post\-hoc judging call is used\.

We additionally evaluate REIN on Mistral\-7B\-v0\.3\(Mistral AI,[2024](https://arxiv.org/html/2608.07931#bib.bib67)\)and DeepSeek\-R1\-Distill\-8B\(DeepSeek\-AI,[2025](https://arxiv.org/html/2608.07931#bib.bib15)\), and observe consistent improvements\. The corresponding results and further implementation details are provided in Appendix C\.

##### Training Details\.

Unless otherwise stated, all backbones use the same training configuration\. We first perform a short SFT stage to initialize the structured output format and reflection behavior, followed by GRPO training on GSM8K\. For each prompt, GRPO samplesK=16K=16completions with a maximum length of 2048 and temperature0\.70\.7\. We use LoRA withr=16r=16andα=32\\alpha=32, and set the KL coefficient to0\.050\.05\. The weights for answer correctness, XML structure, reflection presence, reflection veracity, and IDK alignment are8\.08\.0,3\.03\.0,1\.51\.5,2\.02\.0, and4\.04\.0, respectively\. Training follows the two\-phase IDK curriculum described in Section[3\.4](https://arxiv.org/html/2608.07931#S3.SS4)\. At evaluation, all models use single\-pass greedy decoding without self\-consistency or multi\-round revision\.

## 5Main Results

##### Overall results\.

Table[1](https://arxiv.org/html/2608.07931#S3.T1)reports the main comparison on Qwen2\.5\-7B and Llama\-3\.1\-8B across four benchmarks\. Macro\-averaged over the eight backbone–benchmark pairs, REIN obtains 78\.7% selective accuracy, 12\.0% H\-Proxy, 90\.5% coverage, and 71\.7% effective accuracy\. REIN achieves the best selective accuracy and the lowest H\-Proxy in every reported cell, indicating that the improvement is not restricted to a particular backbone or task family\. Relative to the Base checkpoints, REIN improves selective accuracy by 14\.0 % and effective accuracy by 7\.0 % on average, while reducing H\-Proxy by 28\.5 %\. The accompanying 9\.6\-point reduction in coverage is therefore not a simple loss of utility: the model answers fewer low\-support cases but still produces more correct answers over the complete evaluation set\.

![Refer to caption](https://arxiv.org/html/2608.07931v1/images/reflection_reliability1.png)Figure 2:Reliability judgment calibration across methods and backbones\. We report reflection–answer agreement and Matthews correlation coefficient \(MCC\), showing that REIN improves the alignment between self\-reflection judgments and actual answer correctness\.
##### Comparison with different methods

Compared with IDK Prompting, REIN reduces H\-Proxy by 12\.6 % while improving coverage by 6\.4 % and effective accuracy by 8\.0 %\. This comparison is important because both methods permit abstention, but prompting alone tends to refuse without learning a stable reliability decision rule\. Compared with R\-Tuning, REIN improves selective accuracy by 6\.3 % and effective accuracy by 6\.9 %, with both higher coverage and lower H\-Proxy\. Relative to TruthRL, REIN reduces H\-Proxy by 13\.9 % and improves effective accuracy by 3\.8 %, at the cost of a moderate 2\.4\-point reduction in coverage\. These results place REIN in a more favorable reliability–utility region rather than merely moving along a refusal\-only trade\-off\.

##### Behavior across task difficulty\.

The gains are consistent across mathematical and commonsense reasoning, but their form varies with task difficulty\. On MATH, REIN improves selective accuracy over Base by 19\.8 % for Qwen2\.5\-7B and 21\.0 % for Llama\-3\.1\-8B, while reducing coverage by 16\.5 and 18\.5 %, respectively\. In contrast, the coverage reductions on GSM8K and ARC\-Challenge are substantially smaller\. This pattern is consistent with a difficulty\-sensitive policy: on a benchmark with more prompts near the current policy’s capability frontier, REIN abstains more aggressively, whereas on easier tasks it preserves most of the original answer rate\. Importantly, effective accuracy still increases on MATH for both backbones, showing that the additional abstentions remove errors faster than they remove correct answers\.

### 5\.1Ablation Study

Table 2:Controlled component ablation on GSM8K with Qwen2\.5\-7B\. All variants share the same SFT initialization, GRPO configuration, accuracy reward, and format rewards\.Additional analyses on group size, reward components, and IDK behavior are reported in Appendix E\.

##### Effect of Reflection Alignment\.

To investigate how the amount of reflection affects reliability, we vary the reflection depthd∈\{1,2,3\}d\\in\\\{1,2,3\\\}during training, whered=1d=1uses a single<reflection\>block andd\>1d\>1uses nested reflection\-correction cycles\. Table[3](https://arxiv.org/html/2608.07931#S5.T3)reports results on GSM8K\.

Table 3:Effect of reflection depthddon GSM8K \(7B\)\. Deeper reflection further suppresses hallucination but reduces accuracy and coverage due to overcorrection\.A single reflection layer \(d=1d=1\) yields the best trade\-off between hallucination reduction and reasoning fidelity\. Deeper reflection \(d=2,3d=2,3\) further suppresses hallucinations \(0\.11→\\rightarrow0\.08→\\rightarrow0\.06\) but reduces coverage \(93\.8%→\\rightarrow90\.3%→\\rightarrow88\.4%\) and effective accuracy \(85\.0→\\rightarrow80\.4→\\rightarrow77\.9\), suggesting an*overcorrection*phenomenon where the model becomes overly cautious in self\-judgment\. This result supports our design choice that reflection alignment should remain lightweight\.

We compare Base with Reflection\-only to examine the overall effect of reflection\-structured training\. Reflection\-only uses the same structured output format and reflection rewards as REIN, but removes the boundary\-aware IDK reward\. Averaged over the four benchmarks, Reflection\-only improves selective accuracy over Base for all two backbones and substantially reduces the hallucination proxy\. However, Reflection\-only keeps 100% coverage, showing that reflection\-structured training improves reliability assessment but does not teach the model when to abstain\.

##### Effect of Boundary\-Aware Abstention\.

Table[4](https://arxiv.org/html/2608.07931#S5.T4)reports IDK precision and recall on all datasets, where “truly beyond boundary” is determined by oracle evaluation \(the gold answer appears in no sampled completion across 64 independent samples\)\.

Table 4:IDK quality across benchmarks \(7B\)\. IDK Precision=Pr⁡\[truly beyond∣says IDK\]=\\Pr\[\\text\{truly beyond\}\\mid\\text\{says IDK\}\]; IDK Recall=Pr⁡\[says IDK∣truly beyond\]=\\Pr\[\\text\{says IDK\}\\mid\\text\{truly beyond\}\]; IDK Rate==fraction of questions where the model abstains\.We compare Reflection\-only with REIN to isolate the effect of boundary\-aware abstention\. Both methods use the same reflection format, but REIN additionally uses the group\-level boundary indicator and IDK reward\. Across the evaluated backbones, REIN further improves selective accuracy and reduces the hallucination proxy compared with Reflection\-only\. This comes with a controlled reduction in coverage, because REIN learns to abstain on low\-support prompts\. Effective accuracy remains close to Reflection\-only, indicating that the reliability gain is not obtained by excessive refusal\.

### 5\.2Mechanism Analysis

##### Two\-level reliability\-aware finalization\.

The ablation results suggest that REIN makes reliability judgments at two different levels\. At the completion level, reflection assesses the solution implied by the preceding reasoning\. At the prompt level, the boundary objective estimates whether the current policy can produce a verified solution under a fixed sampling budget\. The distinction matters because a failed rollout does not necessarily mean that the prompt is beyond the model’s capability: another rollout may still succeed\. In such cases, correction is preferable to refusal\. By contrast, when none of the sampled rollouts succeeds, repeated refinement may only make an unsupported answer more convincing\. The two objectives therefore discourage both premature refusal and unjustified continued answering\.

The current evidence supports completion\-level reliability alignment and answer\-level correction, but not step\-level error localization\. REIN receives no supervision indicating which token or reasoning step caused the failure\. An incorrect draft followed by a correct final answer therefore shows that reflection can support a better final decision, but does not establish that the model identified the exact source of the error\. Step\-level diagnostics are reported separately in Appendix A\.

##### Selective boundary behavior\.

The difficulty\-stratified results in Appendix E show that abstention becomes more frequent as the tasks become harder\. The IDK rate increases from 2\.1% on two\-step GSM8K problems to 14\.8% on problems requiring at least five steps, and from 3\.2% on MATH Level 1 to 31\.8% on Level 5\. ARC\-Challenge shows a similar pattern across grade levels\. Although benchmark difficulty is only an approximate indicator of policy\-relative answerability, this monotonic trend argues against a uniform refusal strategy\. The IDK precision of 78\.4–83\.2% and recall of 65\.7–72\.8% in Table[4](https://arxiv.org/html/2608.07931#S5.T4)further indicate that refusals are concentrated on low\-support prompts rather than applied indiscriminately\.

## 6Conclusion

We presented REIN, a framework that improves the reliability of large reasoning models by enabling them to assess whether a generated solution should be trusted or rejected before finalizing an answer\. REIN addresses two complementary failure modes of hallucination\. Reflection alignment helps the model identify unreliable reasoning outcomes, while boundary\-aware abstention encourages the model to avoid unsupported answers when the current capability is insufficient\. Together, these signals provide a unified reliability\-oriented training approach without requiring step\-level supervision or inference\-time intervention\. Experiments across multiple backbones and reasoning benchmarks show that REIN reduces confident but incorrect responses while maintaining strong accuracy and coverage\. These results highlight that reliable reasoning requires not only generating better solutions, but also knowing when a solution should not be trusted\. Future work will investigate more fine\-grained reliability assessment and broader uncertainty modeling\.

## References

- R1\-Code\-Interpreter: LLMs reason with code via supervised and multi\-stage reinforcement learning\.arXiv preprint arXiv:2505\.21668\.Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p1.1)\.
- F\. Cheng, H\. Li, F\. Liu, R\. van Rooij, K\. Zhang, and Z\. Lin \(2025\)Empowering llms with logical reasoning: a comprehensive survey\.External Links:2502\.15652,[Link](https://arxiv.org/abs/2502.15652)Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p1.1)\.
- Q\. Cheng, T\. Sun, X\. Liu, W\. Zhang, Z\. Yin, S\. Li, L\. Li, Z\. He, K\. Chen, and X\. Qiu \(2024\)Can AI assistants know what they don’t know?\.InProceedings of the 41st International Conference on Machine Learning,pp\. 8184–8202\.Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p3.1),[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px3.p1.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try ARC, the AI2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.External Links:[Link](https://arxiv.org/abs/1803.05457)Cited by:[§4](https://arxiv.org/html/2608.07931#S4.SS0.SSS0.Px1.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4](https://arxiv.org/html/2608.07931#S4.SS0.SSS0.Px1.p1.1)\.
- R\. Cohen, E\. Biran, O\. Yoran, M\. Geva, and A\. Globerson \(2024\)I don’t know: explicit modeling of uncertainty with an IDK token\.Advances in Neural Information Processing Systems37\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/14c018d2e72c521605b0567029ef0efb-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p3.1),[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2608.07931#S4.SS0.SSS0.Px2.p1.3)\.
- J\. Dai, X\. Pan, R\. Sun, J\. Ji, X\. Xu, M\. Liu, Y\. Wang, and Y\. Yang \(2023\)Safe RLHF: safe reinforcement learning from human feedback\.arXiv preprint arXiv:2310\.12773\.Cited by:[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px2.p1.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-R1: incentivizing reasoning capability in LLMs via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p1.1),[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2608.07931#S4.SS0.SSS0.Px2.p2.1)\.
- A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.\(2024\)The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.External Links:[Link](https://arxiv.org/abs/2407.21783)Cited by:[§4](https://arxiv.org/html/2608.07931#S4.SS0.SSS0.Px2.p1.3)\.
- M\. Geva, D\. Khashabi, E\. Segal, T\. Khot, D\. Roth, and J\. Berant \(2021\)Did Aristotle Use a Laptop? a question answering benchmark with implicit reasoning strategies\.Transactions of the Association for Computational Linguistics9,pp\. 346–361\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00370),[Link](https://aclanthology.org/2021.tacl-1.21/)Cited by:[§4](https://arxiv.org/html/2608.07931#S4.SS0.SSS0.Px1.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021a\)Measuring mathematical problem solving with the MATH dataset\.InThirty\-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track,Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021b\)Measuring mathematical problem solving with the MATH dataset\.InThirty\-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track,Cited by:[§4](https://arxiv.org/html/2608.07931#S4.SS0.SSS0.Px1.p1.1)\.
- J\. Huang, X\. Chen, S\. Mishra, H\. S\. Zheng, A\. W\. Yu, X\. Song, and D\. Zhou \(2023\)Large language models cannot self\-correct reasoning yet\.arXiv preprint arXiv:2310\.01798\.External Links:[Link](https://arxiv.org/abs/2310.01798)Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p3.1),[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Huang, K\. Xiu, T\. Zheng, C\. Zeng, W\. Ni, Z\. Qin, K\. Ren, and C\. Chen \(2025\)DualBreach: efficient dual\-jailbreaking via target\-driven initialization and multi\-target optimization\.arXiv preprint arXiv:2504\.18564\.Cited by:[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px2.p1.1)\.
- R\. Kamoi, Y\. Zhang, N\. Zhang, J\. Han, and R\. Zhang \(2024\)When can LLMs actually correct their own mistakes? a critical survey of self\-correction of LLMs\.Transactions of the Association for Computational Linguistics12,pp\. 1417–1440\.External Links:[Link](https://aclanthology.org/2024.tacl-1.78/)Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p3.1),[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Kumar, V\. Zhuang, R\. Agarwal, Y\. Su, J\. D\. Co\-Reyes, A\. Singh, K\. Baumli, S\. Iqbal, C\. Bishop, R\. Roelofs, L\. M\. Z\. Ma, T\. L\. Paine, J\. Schrittwieser, Y\. Zhou, C\. Farabet, B\. Shahriari, J\. Wieting, Q\. Vuong, P\. Kumar, N\. Fiedel, N\. Shazeer, K\. Swersky, and S\. Levine \(2025\)Training language models to self\-correct via reinforcement learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=CjwERcAU7w)Cited by:[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px1.p1.1)\.
- K\. Li, O\. Patel, F\. Vi
- \(33\)’egas
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2024\)Let’s verify step by step\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=v8L0pN6EOi)Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p3.1),[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Liu, Y\. Yin, J\. Yan, Y\. Tang, Q\. Zhang, B\. Li, X\. Chen, J\. Wang, X\. Cai, and J\. Su \(2026\)BAPO: boundary\-aware policy optimization for reliable agentic search\.External Links:2601\.11037,[Link](https://arxiv.org/abs/2601.11037)Cited by:[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Gupta, B\. P\. Majumder, K\. Hermann, S\. Welleck, A\. Yazdanbakhsh, and P\. Clark \(2023\)Self\-refine: iterative refinement with self\-feedback\.InAdvances in Neural Information Processing Systems,Vol\.36\.External Links:[Link](https://arxiv.org/abs/2303.17651)Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p3.1),[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\. Yih, P\. W\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. Hajishirzi \(2023\)FActScore: fine\-grained atomic evaluation of factual precision in long form text generation\.arXiv preprint arXiv:2305\.14251\.Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p1.1)\.
- Mistral AI \(2024\)Mistral\-7B\-v0\.3\.Note:Model cardExternal Links:[Link](https://huggingface.co/mistralai/Mistral-7B-v0.3)Cited by:[§4](https://arxiv.org/html/2608.07931#S4.SS0.SSS0.Px2.p2.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. Lowe \(2022\)Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p3.1),[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px2.p1.1)\.
- Qwen Team \(2024\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.External Links:[Link](https://arxiv.org/abs/2412.15115)Cited by:[§4](https://arxiv.org/html/2608.07931#S4.SS0.SSS0.Px2.p1.3)\.
- Qwen Team \(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2305.18290)Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p3.1),[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px2.p1.1)\.
- B\. Ren, S\. Qiao, D\. Zheng, H\. Chen, and N\. Zhang \(2026\)KnowRL: exploring knowledgeable reinforcement learning for factuality\.External Links:2506\.19807,[Link](https://arxiv.org/abs/2506.19807)Cited by:[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.External Links:[Link](https://arxiv.org/abs/2402.03300)Cited by:[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px2.p1.1)\.
- N\. Shinn, F\. Cassano, E\. Berman, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.36\.External Links:[Link](https://arxiv.org/abs/2303.11366)Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p3.1)\.
- J\. Uesato, N\. Kushman, R\. Kumar, F\. Song, N\. Siegel, L\. Wang, A\. Creswell, G\. Irving, and I\. Higgins \(2022\)Solving math word problems with process\- and outcome\-based feedback\.arXiv preprint arXiv:2211\.14275\.External Links:[Link](https://arxiv.org/abs/2211.14275)Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p3.1),[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in Neural Information Processing Systems35,pp\. 24824–24837\.Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p1.1)\.
- Z\. Wei, X\. Yang, K\. Sun, J\. Wang, R\. Shao, J\. Chen, M\. Kachuee, T\. Gollapudi, Y\. Liao, N\. Scheffer, R\. Wanga, A\. Kumar, Y\. Meng, W\. Yih, and X\. L\. Dong \(2026\)TruthRL: incentivizing truthful llms via reinforcement learning\.External Links:2509\.25760,[Link](https://arxiv.org/abs/2509.25760)Cited by:[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2608.07931#S4.SS0.SSS0.Px2.p1.3)\.
- Z\. Wu, W\. Qi, J\. Chen, T\. Zheng, D\. Hong, C\. Lu, L\. He, Z\. Qin, and K\. Ren \(2026\)DataShield: uncovering risky fine\-tuning data across llms through consensus subspace alignment\.arXiv preprint arXiv:2607\.15081\.Cited by:[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px3.p1.1)\.
- K\. Xiu, Y\. Yang, C\. Zeng, T\. Zheng, X\. Huang, D\. Wang, P\. Zhao, Z\. Qin, and K\. Ren \(2025\)Dynamic jailbreaking attack\.External Links:2510\.02422,[Link](https://arxiv.org/abs/2510.02422)Cited by:[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Xu, S\. Liu, Q\. Guo, L\. Yao, Y\. Du, C\. Xu, K\. Xu, D\. Song, C\. Chen, J\. Zhang, and H\. Xiong \(2024\)Rejection improves reliability: training LLMs to refuse unknown questions using reinforcement learning from knowledge feedback\.arXiv preprint arXiv:2403\.18349\.Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p3.1)\.
- J\. Yang, J\. Tu, H\. Liu, X\. Wang, C\. Zheng, Z\. Zhang, S\. Cui, C\. Chen, T\. He, H\. Wang, Y\. Ong, and M\. Huang \(2026\)BARREL: boundary\-aware reasoning for factual and reliable lrms\.External Links:2505\.13529,[Link](https://arxiv.org/abs/2505.13529)Cited by:[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Yuan, H\. Yuan, C\. Li, G\. Dong, C\. Lu, C\. Tan, C\. Zhou, and J\. Zhou \(2023\)Scaling relationship on learning mathematical reasoning with large language models\.arXiv preprint arXiv:2308\.01825\.External Links:[Link](https://arxiv.org/abs/2308.01825)Cited by:[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px2.p1.1)\.
- H\. Zhang, S\. Chen, Y\. Wang, C\. Chen, L\. Wang, Z\. Liu, and M\. Sun \(2024\)R\-Tuning: instructing large language models to say “i don’t know”\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics,pp\. 7113–7130\.External Links:[Link](https://aclanthology.org/2024.naacl-long.394/)Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p3.1),[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px3.p1.1)\.
- H\. Zhao, R\. Wang, T\. Zheng, Q\. Li, B\. Lv, Y\. Wang, and W\. Du \(2025\)From topology to behavioral semantics: enhancing bgp security by understanding bgp’s language with llms\.External Links:2511\.14467,[Link](https://arxiv.org/abs/2511.14467)Cited by:[§2](https://arxiv.org/html/2608.07931#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Zheng, Q\. Cheng, J\. Yao, M\. Wu, H\. He, N\. Ding, Y\. Cheng, S\. Hu, L\. Bai, D\. Zhou, G\. Cui, and P\. Ye \(2025\)Scaling physical reasoning with the PHYSICS dataset\.arXiv preprint arXiv:2506\.00022\.Cited by:[§1](https://arxiv.org/html/2608.07931#S1.p1.1)\.

## Appendix AMetric Definitions

### A\.1Outcome Parsing and Notation

For an evaluation set containingN≥1N\\geq 1instances, each method first generates one response using its native inference procedure\. A deterministic output adapter extracts the final\-answer span and maps it to exactly one of three mutually exclusive outcomes: a substantive answer, an explicitIDK, or an invalid output\. We define

Ai=𝟙​\[yi​is a substantive answer\],Di=𝟙​\[yi​is an explicitIDK\],Ui=𝟙​\[yi​is invalid\],Ai\+Di\+Ui=1\.\\begin\{aligned\} A\_\{i\}&=\\mathbb\{1\}\[y\_\{i\}\\text\{ is a substantive answer\}\],\\\\ D\_\{i\}&=\\mathbb\{1\}\[y\_\{i\}\\text\{ is an explicit \{IDK\}\}\],\\\\ U\_\{i\}&=\\mathbb\{1\}\[y\_\{i\}\\text\{ is invalid\}\],\\end\{aligned\}\\qquad A\_\{i\}\+D\_\{i\}\+U\_\{i\}=1\.\(16\)
The abstention parser examines only the extracted final\-answer span; uncertainty expressed in the reasoning or reflection does not constitute an abstention\. An empty or missing answer, malformed required delimiters, multiple conflicting answers, a mixture of an abstention and a substantive answer, or an answer that cannot be parsed for the target task is considered invalid\. Method\-specific output adapters may be used to accommodate different native output formats, but they are fixed before evaluation and cannot access reference answers or verifier feedback\.

Let

Ti=𝟙​\[Ai=1∧yi​passes the task\-specific verifier\]\.T\_\{i\}=\\mathbb\{1\}\\left\[A\_\{i\}=1\\ \\wedge\\ y\_\{i\}\\text\{ passes the task\-specific verifier\}\\right\]\.\(17\)Thus,Ti≤AiT\_\{i\}\\leq A\_\{i\}, andTi=0T\_\{i\}=0wheneverAi=0A\_\{i\}=0\. GSM8K uses normalized numerical equivalence, MATH\-500 uses mathematical\-equivalence checking, StrategyQA uses normalized exact matching overyes/no, and ARC\-Challenge uses normalized exact matching of the selected option label\.

We define

Na=∑i=1NAi,Nc=∑i=1NTi,Nw=∑i=1N\(Ai−Ti\),N\_\{a\}=\\sum\_\{i=1\}^\{N\}A\_\{i\},\\qquad N\_\{c\}=\\sum\_\{i=1\}^\{N\}T\_\{i\},\\qquad N\_\{w\}=\\sum\_\{i=1\}^\{N\}\(A\_\{i\}\-T\_\{i\}\),\(18\)whereNaN\_\{a\},NcN\_\{c\}, andNwN\_\{w\}denote the numbers of substantive, correct substantive, and incorrect substantive answers, respectively\. Consequently,Na=Nc\+NwN\_\{a\}=N\_\{c\}\+N\_\{w\}\.

### A\.2Native Reflection Indicator

For H\-Proxy evaluation, every method generates a structured<think\>→\\rightarrow<reflection\>→\\rightarrow<answer\>completion in a single autoregressive pass\. We use*native*to mean that the reflection is emitted before the final answer within this same completion, regardless of whether the method was explicitly trained with a reflection objective\. No additional post\-hoc judging call is used to compute the primary H\-Proxy\. Each native completion is parsed into a pre\-reflection draftyi\(0\)y\_\{i\}^\{\(0\)\}, a reflectionrir\_\{i\}, and a final answeryi\(1\)y\_\{i\}^\{\(1\)\}\. The draftyi\(0\)y\_\{i\}^\{\(0\)\}is extracted only from the content preceding the reflection\.

Let

Pi\(0\)=𝟙​\[yi\(0\)​is a parseable substantive draft\]P\_\{i\}^\{\(0\)\}=\\mathbb\{1\}\\left\[y\_\{i\}^\{\(0\)\}\\text\{ is a parseable substantive draft\}\\right\]\(19\)denote whether a valid draft answer can be extracted\. Draft extraction failures are excluded from draft\-correctness metrics and are reported separately through the draft extraction rate\.

Draft correctness is defined as

Ti\(0\)=𝟙​\[Pi\(0\)=1∧yi\(0\)​passes the task\-specific verifier\]\.T\_\{i\}^\{\(0\)\}=\\mathbb\{1\}\\left\[P\_\{i\}^\{\(0\)\}=1\\ \\wedge\\ y\_\{i\}^\{\(0\)\}\\text\{ passes the task\-specific verifier\}\\right\]\.\(20\)
The native reflection stance is parsed as

si∈\{correct,wrong,uncertain,⊥\},s\_\{i\}\\in\\\{\\texttt\{correct\},\\texttt\{wrong\},\\texttt\{uncertain\},\\bot\\\},\(21\)where⊥\\botdenotes an unparsable reflection\. We define the binary native reliability indicator as

Jinat=𝟙​\[si=correct\]\.J\_\{i\}^\{\\mathrm\{nat\}\}=\\mathbb\{1\}\\left\[s\_\{i\}=\\texttt\{correct\}\\right\]\.\(22\)Thus,wrong,uncertain, and unparsable reflections are not treated as endorsements\. Uncertain and unparsable cases remain separately visible in the native\-reflection diagnostics\.

Let

Nw\(0\)=∑i=1NPi\(0\)​\(1−Ti\(0\)\)N\_\{w\}^\{\(0\)\}=\\sum\_\{i=1\}^\{N\}P\_\{i\}^\{\(0\)\}\(1\-T\_\{i\}^\{\(0\)\}\)\(23\)denote the number of incorrect parseable drafts, and let

Nenat=∑i=1NPi\(0\)​\(1−Ti\(0\)\)​JinatN\_\{e\}^\{\\mathrm\{nat\}\}=\\sum\_\{i=1\}^\{N\}P\_\{i\}^\{\(0\)\}\(1\-T\_\{i\}^\{\(0\)\}\)J\_\{i\}^\{\\mathrm\{nat\}\}\(24\)denote the number of incorrect drafts that are endorsed ascorrectby their native reflections\.

### A\.3Primary Evaluation Metrics

Since REIN allows the model to abstain, standard accuracy alone is not sufficient for evaluation\. A model can obtain higher accuracy on attempted questions simply by refusing many difficult questions\. Therefore, we evaluate reliability under a selective answering setting using four metrics: Selective Accuracy, Hallucination Proxy \(H\-Proxy\), Coverage, and Effective Accuracy\.

Sel\.Acc\.\\displaystyle\\mathrm\{Sel\.Acc\.\}=NcNa,\\displaystyle=\\frac\{N\_\{c\}\}\{N\_\{a\}\},H​\-​Proxy\\displaystyle\\mathrm\{H\\text\{\-\}Proxy\}=NenatNw\(0\),\\displaystyle=\\frac\{N\_\{e\}^\{\\mathrm\{nat\}\}\}\{N\_\{w\}^\{\(0\)\}\},Cov\.\\displaystyle\\mathrm\{Cov\.\}=NaN,\\displaystyle=\\frac\{N\_\{a\}\}\{N\},Eff\.Acc\.\\displaystyle\\mathrm\{Eff\.Acc\.\}=Sel\.Acc\.×Cov\.=NcN\.\\displaystyle=\\mathrm\{Sel\.Acc\.\}\\times\\mathrm\{Cov\.\}=\\frac\{N\_\{c\}\}\{N\}\.
Selective Accuracymeasures the correctness of answers that the model actually attempts\. It reflects whether the model gives more accurate answers when it chooses to answer\.

Hallucination Proxymeasures the fraction of incorrect pre\-reflection drafts that are nevertheless endorsed ascorrectby the native reflection generated within the same completion\. It captures the failure mode in which the model incorrectly endorses an unreliable draft before producing its final answer\.

Coveragemeasures the fraction of questions for which the model provides a substantive answer instead of abstaining or producing an invalid output\.

Effective Accuracymeasures the overall fraction of questions answered correctly after accounting for abstention\. It combines answer correctness and coverage, and therefore reflects the actual utility of the model under selective answering\.

Higher Selective Accuracy, Coverage, and Effective Accuracy are better, while lower Hallucination Proxy is better\. Together, these metrics show whether REIN improves reliability without relying on excessive abstention\.

### A\.4Edge Cases and Reporting

IfNa=0N\_\{a\}=0, Coverage and Effective Accuracy are defined as zero, whereas Selective Accuracy is undefined because its denominator is zero\. Because H\-Proxy is computed from pre\-reflection drafts, its definition does not depend onNaN\_\{a\}\.

IfNw\(0\)=0N\_\{w\}^\{\(0\)\}=0, H\-Proxy is undefined and is reported as “–”\. Other conditional reliability metrics are likewise reported as “–” whenever their corresponding denominators are zero\.

All metrics are computed directly from sample\-level events and are reported as percentages\. Algebraic identities are used only for interpretation; final values are not reconstructed by multiplying rounded table entries\. Metrics are computed separately for each checkpoint–benchmark pair, and every cross\-benchmark aggregate explicitly states whether macro\- or micro\-averaging is used\.

## Appendix BSupplementary Experimental Results

Table 5:Supplementary experimental results on Mistral\-7B\-v0\.3 and DeepSeek\-R1\-Distill\-8B\.### B\.1Comparison with Existing Methods

We compare REIN with existing methods for mitigating reasoning hallucinations in figure[3](https://arxiv.org/html/2608.07931#A2.F3), including In\-Context Learning \(ICL\), knowledge distillation, and BARREL\. To ensure a fair comparison, we adopt three same key evaluation metrics with BARREL: Accuracy \(Acc\.\), Truthfulness \(Truth\.\), and Reliability \(Rel\.\), which are defined as follows:Acc\.=NcN\\text\{Acc\.\}=\\frac\{N\_\{c\}\}\{N\},Truth\.=Nc\+NrN\\text\{Truth\.\}=\\frac\{N\_\{c\}\+N\_\{r\}\}\{N\}, andRel\.=ans\.⋅Truth\.\+\(1−ans\.\)⋅Acc\.\\text\{Rel\.\}=\\text\{ans\.\}\\cdot\\text\{Truth\.\}\+\(1\-\\text\{ans\.\}\)\\cdot\\text\{Acc\.\}, whereans\.=1−NrN\\text\{ans\.\}=1\-\\frac\{N\_\{r\}\}\{N\}\. As illustrated in Figure[3](https://arxiv.org/html/2608.07931#A2.F3), REIN achieves the highest Acc\., Truth\., and Rel\. among all compared baselines\. Furthermore, we also compare our method against token\-level calibration approaches such as R\-Tuning and IDK\-token training\. We observe that these methods significantly interfere with the model’s original output distribution, thereby degrading its inherent reasoning capabilities\.

![Refer to caption](https://arxiv.org/html/2608.07931v1/images/comparison_plot.png)Figure 3:Comparison of REIN with existing methods\. REIN achieves the highest Accuracy, Truthfulness, and Reliability\. The model is Deepseek\-Distill\-8B and the benchmark is TriviaQA\.
### B\.2Extended Baseline Comparison under a Unified Protocol

Table 6:Comprehensive experimental results on Qwen2\.5\-7B and LLaMA\-3\.1\-8B\.Table[6](https://arxiv.org/html/2608.07931#A2.T6)places REIN alongside nine published methods spanning training\-free self\-correction, refusal\-aware fine\-tuning, and RL\-based reliability alignment, together with the Base checkpoint and the IDK\-prompting control of Table 1 of the main paper\. The results separate two quantities that the literature usually reports together\. Reasoning\-oriented optimization reliably buys accuracy: vanilla GRPO lifts average effective accuracy from 64\.9 to 70\.7 on Qwen2\.5\-7B, andS2​RS^\{2\}R, which explicitly rewards self\-verification and self\-correction, reaches the highest effective accuracy of any method in the table at 74\.1\. Neither buys reliability: their H\-Proxy stays at 34\.8 and 30\.8, barely below the 40\.0 of the untrained backbone, so roughly one wrong answer in three is still delivered with an explicit assertion that it is sound\. Test\-time sampling is not an alternative route to it either—sixteen\-sample self\-consistency adds 6\.8 points of effective accuracy and simultaneously raises H\-Proxy to 42\.0, because majority voting removes exactly the errors the model was least sure of and leaves behind the self\-consistent ones that reflection is most willing to endorse\. Abstention supervision moves the second quantity but at a cost on the first: R\-Tuning reaches the lowest H\-Proxy of any external method \(17\.7\) yet abstains hard enough that its effective accuracy of 62\.3 falls below the untrained baseline, and BARREL, whose IDK recipe was calibrated on long\-CoT models that guess at the last moment, transfers only partially to short\-CoT instruction\-tuned backbones and settles at a conservative operating point of 87\.3 coverage and 66\.3 effective accuracy\.

The most informative comparison is with TruthRL, the strongest RL\-based abstention baseline, and here the two quantities come apart cleanly\. On Qwen2\.5\-7B the two methods sit at almost the same coverage \(90\.7 against our 90\.4\), and at that shared operating point REIN adds 4\.7 points of selective accuracy and 4\.1 points of effective accuracy while cutting H\-Proxy from 24\.6 to 11\.3\. On LLaMA\-3\.1\-8B TruthRL retains 4\.5 more coverage points \(95\.0 against 90\.5\) and still converts them into 3\.3 fewer correct answers over the full evaluation set \(68\.6 against 71\.9\)\. The advantage is not carried by one benchmark: REIN attains higher effective accuracy in all eight backbone–benchmark cells, and the largest single margin, 6\.5 points, appears on StrategyQA with Qwen2\.5\-7B, where TruthRL answers 92\.6% of the questions and endorses 26\.9% of the wrong answers it delivers\.

That contrast also localizes the mechanism\. TruthRL’s ternary reward leaves H\-Proxy above our reflection\-only ablation on both backbones \(24\.6 against 18\.7 on Qwen2\.5\-7B, 27\.3 against 20\.2 on LLaMA\-3\.1\-8B\), which is what one expects of any signal that prices whether to answer without constraining how the model assesses the answer it does give; only the full objective, in which the reflection\-veracity term aligns the stance of an explicit reflection span with verified correctness while the boundary\-aware term redirects the answer distribution itself, brings the rate below 20\.0 on every benchmark and every backbone, and to 12\.0 or below outside MATH\. What REIN does not claim is uniform superiority on utility alone:S2​RS^\{2\}Ranswers every question and still reaches 74\.1 average effective accuracy against our 71\.5, so a deployment that never pays for a wrong answer is better served by refusing nothing\. The case for REIN is that at a comparable answer rate it is right more often and, when it is wrong, says so\.

### B\.3Supplementary MCC Results

##### Supplementary reflection\-aware metric

We also present the Matthews Correlation Coefficient \(MCC\) as a secondary measure of reflection\-aware reliability because it measures how much agreement there is between whether a model’s output is correct or the output is reliably classified according to its reflection judgement \(reliable vs\.

MCC=TP⋅TN−FP⋅FN\(TP\+FP\)​\(TP\+FN\)​\(TN\+FP\)​\(TN\+FN\)\\mathrm\{MCC\}=\\frac\{\\mathrm\{TP\}\\cdot\\mathrm\{TN\}\-\\mathrm\{FP\}\\cdot\\mathrm\{FN\}\}\{\\sqrt\{\(\\mathrm\{TP\}\+\\mathrm\{FP\}\)\(\\mathrm\{TP\}\+\\mathrm\{FN\}\)\(\\mathrm\{TN\}\+\\mathrm\{FP\}\)\(\\mathrm\{TN\}\+\\mathrm\{FN\}\)\}\}unreliable\)\. Here solution correctness will be used as the ground\-truth label and the reflection judgment will be used as a binary prediction\. In terms of definitions: TP, TN, FP, and FN are defined as true positives, true negatives, false positives, and false negatives respectively\. Therefore, the MCC can be calculated as In our scenario, a positive label means that a solution is correct and the associated reflection judgement labels the solution as reliable, whereas a negative label indicates that the solution is incorrect and the associated reflection judgement labels the solution as unreliable\. The MCC does not depend on whether the two classes of examples are equally distributed \(i\.e\., has class imbalance\) and therefore will give penalties to both false\-positive \(incorrect answers that are classified by the reflection judgement as being reliable\) and false\-negative \(correct answers that are classified by the reflection judgement as being unreliable\) classifications\. Where indicated, we map correct to reliable and incorrect/uncertain to unreliable\.

### B\.4Evolution of Reward Components

![Refer to caption](https://arxiv.org/html/2608.07931v1/images/qwen_reward_components.png)Figure 4:Reward Composition Dynamics During REIN Training \(Qwen2\.5\-7B\)Figure[4](https://arxiv.org/html/2608.07931#A2.F4)illustrates the evolution of the overall reward and its five components during REIN training\. The training exhibits two distinct phases\. In the first phase, the model rapidly acquires the required response format and reasoning behavior, as reflected by the fast convergence of the XML structure and reflection presence rewards\. Meanwhile, the accuracy reward steadily improves, indicating that the enforced reflection mechanism does not compromise task\-solving ability\.

After the phase transition, the reflection veracity and IDK alignment rewards continue to increase, suggesting that the model gradually learns to calibrate its self\-evaluation and align its abstention behavior with its actual capability boundary\. The consistent improvement of total reward demonstrates that REIN jointly optimizes answer correctness, reliable self\-reflection, and selective abstention rather than optimizing these objectives independently\.

\(a\) One model, one fixed answer set, seven abstention scoresSelective scoreAUROC↑\\uparrowAURC↓\\downarrowSel\.@90↑\\uparrowSel\.@80↑\\uparrow\#fwdGSM8K \(raw=88\.1, risk@100=11\.9\)Sequence log\-prob\.712\.06389\.691\.21Length\-norm\. log\-prob\.741\.05890\.192\.01P\(True\)\.779\.05190\.893\.12Semantic entropy @16\.836\.04391\.694\.516Self\-consistency agr\. @16\.849\.04091\.994\.916Reflection stance \(ours\)\.857\.03892\.295\.31\+ self\-consistency @16\.884\.03193\.196\.417MATH \(raw=55\.0, risk@100=45\.0\)Sequence log\-prob\.688\.28856\.458\.21Length\-norm\. log\-prob\.703\.28157\.059\.11P\(True\)\.724\.27257\.660\.22Semantic entropy @16\.781\.25958\.461\.816Self\-consistency agr\. @16\.792\.25558\.762\.416Reflection stance \(ours\)\.786\.25758\.562\.01\+ self\-consistency @16\.818\.24659\.363\.517

Table 7:Selective prediction with different abstention scores\. The native reflection stance provides a competitive single\-pass selective signal, outperforming conventional likelihood\-based confidence measures and matching or exceeding sixteen\-sample self\-consistency on GSM8K\. Its combination with self\-consistency yields the strongest selective performance, indicating that reflection\-based diagnosis and sample agreement provide complementary reliability information\.Table 8:Selective prediction comparison under a fixed coverage of 90%\.

## Appendix CAblation and Diagnostic Analysis

Table 9:Ablation study of REIN’s reliability alignment components on GSM8K \(Qwen2\.5\-7B\)\. Each row removes one objective component\. Removing Reflection Veracity increases hallucination errors, while removing IDK alignment affects answer\-abstain calibration\.##### Key findings

\(1\)Accuracy reward is essential: removing it causes a large drop in selective accuracy \(−12\.1\-12\.1\), confirming that correctness signal is the primary learning driver\. \(2\)Reflection Veracity controls hallucination: removing it has minimal effect on selective accuracy \(−0\.4\-0\.4\) but nearly doubles the hallucination rate \(0\.11→\\rightarrow0\.20\), showing this reward is specifically responsible for aligning the reflection’s self\-assessment with actual correctness\. \(3\)IDK alignment enables abstention: without it, the model never abstains \(Cov\.=100%\) and achieves identical results to “REIN w/o IDK” in the main table, confirming this reward is the sole driver of IDK behavior\. \(4\)XML Structure supports reflection quality: removing it reduces effective accuracy by 2\.8 points and increases hallucination \(0\.11→\\rightarrow0\.14\), indicating that consistent formatting is a prerequisite for effective self\-checking\.

### C\.1Sensitivity to Group SizeKK

The knowledge boundary indicatorB​\(x\)B\(x\)depends on the group sizeKK: largerKKprovides a more reliable estimate of whether a question is truly beyond the model’s capability\. Table[10](https://arxiv.org/html/2608.07931#A3.T10)shows the effect of varyingKKon GSM8K and MATH\.

LargerKKyields more accurate boundary detection on both benchmarks \(\+11\.3\+11\.3on GSM8K,\+13\.2\+13\.2on MATH fromKK=4 to 16\), which translates to better selective accuracy and lower hallucination rate\. Boundary detection is inherently harder on MATH \(85\.3% vs\. 89\.7% atKK=16\) due to its broader difficulty distribution, where more questions lie near the model’s capability frontier\. We useK=16K=16as the default, which achieves a good balance between detection accuracy and computational efficiency\.

Table 10:Effect of group sizeKKon boundary estimation and downstream metrics\. LargerKKimproves boundary detection but increases sampling cost\.
### C\.2Difficulty\-Stratified Abstention

DatasetDifficulty BinIDK Rate \(%\)GSM8K \(by steps\)2\-step2\.13–4 step5\.45\+ step14\.8MATH \(by level\)Level 13\.2Level 28\.5Level 315\.1Level 422\.4Level 531\.8ARC\-C \(by grade\)Elementary2\.8Middle school6\.5High school13\.4

Table 11:IDK rate stratified by difficulty across benchmarks \(7B\)\. The model’s abstention rate increases monotonically with question difficulty, confirming calibrated knowledge\-boundary awareness\.

## Appendix DReflection Causality and Draft\-to\-Final Transition

Section 4 of the main paper attributes REIN’s reliability improvement to the<reflection\>span\. This section investigates whether reflection causally affects the downstream finalization process\. Rather than treating reflection as an auxiliary textual annotation, we perform inference\-time interventions on the reflection span and trace how different reflection states influence the transition from the initial drafty0y\_\{0\}to the final delivered answery1y\_\{1\}\.

### D\.1Draft\-to\-Final Transition Analysis

To further analyze how reflection changes the final output, we trace each instance from the initial draft answery0y\_\{0\}to the final delivered answery1y\_\{1\}\.

Table 12:Draft\-to\-final transition matrix of REIN on Qwen2\.5\-7B pooled over the four benchmarks\. Rows indicate the correctness status of the initial drafty0y\_\{0\}, while columns indicate the final delivered outcomey1y\_\{1\}\. The matrix characterizes how reflection\-guided finalization transforms initial reasoning states into final decisions\.Among 747 initially incorrect drafts, 82 become correct after reflection, yielding a repair rate of

82747=11\.0%\.\\frac\{82\}\{747\}=11\.0\\%\.
In contrast, only 22 out of 2454 initially correct drafts become incorrect, corresponding to a corruption rate of

222454=0\.9%\.\\frac\{22\}\{2454\}=0\.9\\%\.
Therefore, reflection improves finalization quality by correcting incorrect reasoning outcomes substantially more often than it damages correct ones\.

The transition analysis also shows that abstention is targeted rather than uniform\. Among the 262 non\-substantive final outputs, most originate from incorrect or unreliable initial drafts\. This indicates that the abstention mechanism operates jointly with reflection: unreliable reasoning trajectories are more likely to be removed rather than forced into unsupported final answers\.

## Appendix EFinite\-Sample Interpretation and Rollout\-Budget Sensitivity

##### Notation\.

We use the same notation as in Section 3\. For a promptxx,y⋆​\(x\)y^\{\\star\}\(x\)denotes the gold answer,T​\(x,y\)=𝟙​\[y=y⋆​\(x\)\]T\(x,y\)=\\mathbb\{1\}\[y=y^\{\\star\}\(x\)\]denotes answer correctness, andp​\(x\)=𝔼​\[T​\(x,y\)∣x\]p\(x\)=\\mathbb\{E\}\[T\(x,y\)\\mid x\]denotes the per\-prompt correctness probability\. The parsed reflection judgment is denoted byJ​\(r\)∈\{0,1\}J\(r\)\\in\\\{0,1\\\}, where11means reliable and0means unreliable\.

### E\.1Finite\-Sample Interpretation and Rollout\-Budget Sensitivity

Let

pθ​\(x\)=Prz∼πθ\(⋅∣x\)⁡\[T​\(x,y1\)=1\]p\_\{\\theta\}\(x\)=\\Pr\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\left\[T\(x,y\_\{1\}\)=1\\right\]\(25\)denote the probability that the current policy produces a verified\-correct final answer for promptxx\. Under conditionally independent sampling, the probability that none of theKKcompletions succeeds is

Pr⁡\(B^K​\(x\)=beyond∣x\)=\(1−pθ​\(x\)\)K\.\\Pr\\left\(\\widehat\{B\}\_\{K\}\(x\)=\\texttt\{beyond\}\\mid x\\right\)=\\left\(1\-p\_\{\\theta\}\(x\)\\right\)^\{K\}\.\(26\)
Therefore, a no\-success group does not imply thatpθ​\(x\)=0p\_\{\\theta\}\(x\)=0\. Instead, it provides increasingly strong evidence of a low policy\-level success probability as the rollout budget increases\.

When zero successes are observed amongKKsamples, a one\-sided\(1−α\)\(1\-\\alpha\)upper confidence bound on the success probability is

pθ​\(x\)≤1−α1/K\.p\_\{\\theta\}\(x\)\\leq 1\-\\alpha^\{1/K\}\.\(27\)At the 95% confidence level, the corresponding upper bounds are 31\.23%, 17\.07%, 8\.94%, and 4\.57% forK=8,16,32,K=8,16,32,and6464, respectively\. Larger rollout budgets therefore reduce finite\-sampling uncertainty, but increase rollout generation and verification costs approximately linearly\. We useK=16K=16during GRPO training as a practical compromise between boundary\-estimation stability and computational cost\.

We further evaluate the sensitivity of the empirical boundary usingK∈1,2,4,8,16,32,64K\\in\{1,2,4,8,16,32,64\}\. For each budget, we report the proportion of prompts for which no verified solution is observed\. We also compare the training budgetK=16K=16with the higher\-budgetK=64K=64reference by reporting the proportion ofK=16K=16beyond\-boundary prompts that remain beyond atK=64K=64, together with the proportion that recover to within\-boundary status\. TheK=64K=64result is used only as a higher\-budget empirical reference and should not be interpreted as an oracle label of intrinsic knowledge absence\.

![Refer to caption](https://arxiv.org/html/2608.07931v1/images/reflection_reliability2.png)Figure 5:Reduction of false endorsement through reflection alignment\. False endorsement measures the fraction of incorrect answers that are incorrectly judged as reliable\. REIN substantially reduces false endorsements while maintaining a low rate of correct\-answer misjudgment\.
### E\.2Confidence Bound of the Group\-level Boundary Indicator

The group\-level boundary indicatorB​\(x\)B\(x\)estimates whether a prompt lies within the model’s effective capability under the current policy and sampling budget\. AlthoughB​\(x\)B\(x\)does not represent the true epistemic boundary of the model, we show that observing no correct samples provides a probabilistic upper bound on the probability of producing a correct answer\.

###### Lemma 1\(Confidence bound ofB​\(x\)B\(x\)\)\.

Assume thatKKcompletions are independently sampled from the policy

\{\(c\(k\),r\(k\),y\(k\)\)\}k=1K∼πθ\(⋅\|x\),\\\{\(c^\{\(k\)\},r^\{\(k\)\},y^\{\(k\)\}\)\\\}\_\{k=1\}^\{K\}\\sim\\pi\_\{\\theta\}\(\\cdot\|x\),where each completion has success probability

p​\(x\)=Pr⁡\[y=y⋆​\(x\)∣x,πθ\]\.p\(x\)=\\Pr\[y=y^\{\\star\}\(x\)\\mid x,\\pi\_\{\\theta\}\]\.If the group\-level boundary indicator satisfies

B​\(x\)=beyond,B\(x\)=\\mathrm\{beyond\},meaning that none of theKKsampled completions produces the correct answer, then, with confidence level1−α1\-\\alpha,

p​\(x\)≤1−α1/K\.p\(x\)\\leq 1\-\\alpha^\{1/K\}\.

###### Proof\.

For a single completion sampled fromπθ\(⋅\|x\)\\pi\_\{\\theta\}\(\\cdot\|x\), the probability of obtaining an incorrect answer is

Since theKKsamples are independent, the probability that all sampled completions fail to produce the correct answer is

Pr⁡\[B​\(x\)=beyond\]=\(1−p​\(x\)\)K\.\\Pr\[B\(x\)=\\mathrm\{beyond\}\]=\(1\-p\(x\)\)^\{K\}\.
ObservingB​\(x\)=beyondB\(x\)=\\mathrm\{beyond\}means that zero successful samples are obtained inKKindependent trials\. For a confidence level1−α1\-\\alpha, the failure probability satisfies

\(1−p​\(x\)\)K≥α\.\(1\-p\(x\)\)^\{K\}\\geq\\alpha\.
Taking theKK\-th root on both sides gives

1−p​\(x\)≥α1/K\.1\-p\(x\)\\geq\\alpha^\{1/K\}\.
Therefore,

p​\(x\)≤1−α1/K\.p\(x\)\\leq 1\-\\alpha^\{1/K\}\.
Hence, when no correct answer is observed amongKKindependent samples, the probability that the model can correctly answer the prompt is bounded above by1−α1/K1\-\\alpha^\{1/K\}with confidence level1−α1\-\\alpha\. ∎

##### Example\.

ForK=16K=16samples and a90%90\\%confidence level \(α=0\.10\\alpha=0\.10\), the upper bound becomes

p​\(x\)≤1−0\.11/16≈0\.134\.p\(x\)\\leq 1\-0\.1^\{1/16\}\\approx 0\.134\.
Therefore, if none of the 16 sampled completions produces the correct answer, we can infer with90%90\\%confidence that the probability of the current policy solving the prompt is at most13\.4%13\.4\\%\.

## Appendix FImplementation Details and Computational Resources

![Refer to caption](https://arxiv.org/html/2608.07931v1/images/training_curves.png)Figure 6:Training curves of KL divergence, reward, and loss\. KL remains low and stable, while reward steadily increases and loss decreases throughout training\.### F\.1Implemented Variants and Scope

The repository contains several implemented variants beyond the minimal baseline described above\. These include 7B and 8B training configs, veracity\-augmented reward variants, and additional prompt templates that increase the amount of reflective structure\. We treat these as implementation variants rather than as fully reported analyses unless accompanied by committed experimental outputs\.

#### F\.1\.1Reward Variants

The primary distinction between the released GRPO recipes is whether the training objective uses three rewards \(accuracy\+xml\_structure\+reflection\_presence\) or a four\-reward variant that additionally includesreflection\_veracity\. This makes it possible to compare a simpler format\-and\-correctness baseline against a variant that explicitly aligns reflection judgment with drafty0y\_\{0\}correctness\.

#### F\.1\.2Training and Evaluation Details

The training config and hyperparameter are shown in table[13](https://arxiv.org/html/2608.07931#A6.T13)\.

Table 13:We have a shared configuration across all the backbones unless we say otherwise\. Other implementation details will adhere to the released configurations unless otherwise indicated\.As per the instructions provided in the main evaluation script, we use the single\-pass generation decoding protocol withdo\_sample=False\. The main evaluation script supports the modes of reflection, instruction, and no\-CoT\. It produces the final answers usingboxed\{\}for MATH/AIME\-style evaluation and delimited by\#\#\#\#for GSM8K\. There is no use of test\-time majority voting, self\-consistency, or multi\-round critique loops\.

#### F\.1\.3Diagnostic Scope

In addition to task\-level metrics, the repository includes a diagnostic script for evaluating whether reflections identify the erroneous step and error type on a manually annotated subset of incorrect generations\. We view this as a useful analysis tool for future experiments, but not as a replacement for the main benchmark metrics of accuracy, MCC, and hallucination rate\.

![Refer to caption](https://arxiv.org/html/2608.07931v1/x1.png)Figure 7:Reflection judgments align with answer outcomes across backbones and benchmarks\. Each small point denotes one observed model–dataset cell, while large markers report method\-level macro averages\. False endorsement measures the fraction of wrong attempted answers judged reliable; false rejection measures the fraction of correct attempted answers judged unreliable\. Lower values are better\. Comparisons are descriptive because the available numbers of model–dataset cells differ across methods\.

### F\.2Hardware and Duration

We conduct our experiments on a computing system with NVIDIA RTX PRO™ 6000 Blackwell GPUs, each equipped with 96GB of VRAM\. We trained the models on two GPUs and the average duration of one full training of a model with final learning rates and exhaustive evaluation around 30h\.

## Appendix GRobustness of the Measurement

Every number in this paper passes through a prompt, a decoding pass and a parser before it becomes a metric\. Each of those is a degree of freedom that could be tuned, after the fact, in a direction that favours the proposed method\. This appendix fixes all three and measures how much the headline results move when they are varied: across abstention surface forms, across the output conflicts the parser must adjudicate, across paraphrases of the evaluation prompt, and across four parser implementations\.

### G\.1Abstention Surface Forms

The abstention parser recognizes a fixed lexicon rather than a single string, because a model that abstains in words the parser does not know would be scored as producing an invalid output and would lose coverage it has in fact earned\.

Table[14](https://arxiv.org/html/2608.07931#A7.T14)shows the question is close to moot for a trained model: 97\.0% of REIN’s abstentions use the canonical string the reward was written against, and 99\.6% fall in the top three forms\. Restricting the parser to the canonical string alone reclassifies the seven non\-canonical abstentions as invalid outputs\. It cannot move coverage, because Eq\. \(1\) makes abstentions and invalid outputs alike non\-substantive, and it cannot move selective accuracy or H\-Proxy, because neither outcome enters their numerators or denominators; the lexicon governs only the abstention–invalid split reported in Appendix G\. What the lexicon does affect is the prompted baselines, whose abstentions are far less uniform, and for which a canonical\-only parser would misclassify genuine refusals as malformed output\. Going the other way and instructing REIN to abstain in each of the eight forms in turn leaves macro coverage in\[89\.9,90\.9\]\[89\.9,90\.9\]and H\-Proxy in\[10\.9,11\.8\]\[10\.9,11\.8\], so no reported conclusion depends on which phrase the abstention is required to take\. The four forms the trained model never produces are in the lexicon for the baselines alone\.

Table 14:Abstention surface forms, REIN on Qwen2\.5\-7B pooled over the four benchmarks\. Detect\. is the parser’s recall on held\-out completions manually labelled as abstentions; Emitted is the share of REIN’s 236 explicit abstentions taking that form under the canonical prompt, so the column is 229/4/2/1/0/0/0/0 expressed as percentages\. Both columns are percentages\.
### G\.2Conflicting Outputs

A completion can contradict itself, and how those cases are resolved is a choice that must be declared rather than buried in an implementation\.

Self\-contradiction is rare, 1\.65% of completions, and the three adjudication rules of Table[15](https://arxiv.org/html/2608.07931#A7.T15)separate the headline metrics by at most 0\.4 points\. The conservative default is not the one that flatters REIN: the permissive rule would report a lower H\-Proxy \(10\.9\) and higher coverage \(91\.0\)\. We keep the conservative rule because it is the only one of the three that cannot be accused of resolving ambiguity in the proposed method’s favour, and we apply it identically to every baseline\.

Class C5 deserves a note because it is the only conflict the two judges of Appendix A handle differently\. Eight completions carry an endorsing stance and an explicit abstention: the model asserts its reasoning is sound and then declines to commit to an answer\. The native reading records no substantive answer, so these instances leave the H\-Proxy denominator entirely; the post\-hoc probe is never invoked, since it is elicited only for substantive answers\. They are counted as abstentions throughout\.

\(a\) Taxonomy and incidence

\(b\) Sensitivity to the adjudication rule

Table 15:Self\-contradictory outputs from the REIN model on Qwen2\.5\-7B, pooled over the four benchmarks \(N=3220N=3220\)\. The conservative rule treats C1–C3 and C6 as invalid and C4 as a non\-endorsement, which is the convention used everywhere else in this paper; the permissive rule resolves each conflict in the model’s favour by taking the first well\-formed answer and reading C4 as an endorsement; the third rule removes the 53 affected instances from all denominators\.

## Appendix HQualitative Examples

A representative case of correcting reasoning errors within the knowledge boundarySystem:You are a careful reasoner\. For every problem, reply with EXACTLY this structure:<think\>\.\.\.</think\> <reflection\>\.\.\.</reflection\> <answer\>\.\.\.</answer\>\.User:A store has 120 apples\. They sell13\\frac\{1\}\{3\}of the apples in the morning and14\\frac\{1\}\{4\}of the remaining apples in the afternoon\. How many apples are left?Assistant:<think\>Morning:120×13=40120\\times\\frac\{1\}\{3\}=40sold, leaving 80 apples\. Afternoon:120×14=30120\\times\\frac\{1\}\{4\}=30sold, leaving80−30=5080\-30=50apples\.</think\><reflection\>Wait\.The afternoon sale should be14\\frac\{1\}\{4\}of the remaining 80 apples, not the original 120\. Recalculating:80×14=2080\\times\\frac\{1\}\{4\}=20sold in the afternoon\. Remaining apples:80−20=6080\-20=60\. The initial calculation contains an error in step 2\.</reflection\><answer\> 60</answer\>

A representative case of the REIN framework triggering selective abstentionSystem:You are a careful reasoner\. For every problem, reply with EXACTLY this structure:<think\>\.\.\.</think\> <reflection\>\.\.\.</reflection\> <answer\>\.\.\.</answer\>\.User:Find the unique 3\-digit integerNNsuch thatNNis twice the sum of its digits cubed\.Assistant: <think\> Such Armstrong\-like number is uncommon\. I would need to search digit\-by\-digit, but this is computationally heavy without a calculator\. </think\><reflection\>Unsure\.The search space requires careful enumeration that I cannot do reliably by hand\.</reflection\><answer\> I don’t know </answer\>

An empirical failure mode of the self\-correction mechanism within the REIN frameworkSystem:You are a careful reasoner\. For every problem, reply with EXACTLY this structure:<think\>\.\.\.</think\> <reflection\>\.\.\.</reflection\> <answer\>\.\.\.</answer\>\.User:Solve:x2−4=0x^\{2\}\-4=0\.Assistant: <think\> x=±2x=\\pm 2\. Recorded: 2\. </think\><reflection\> Wrong\.There are two roots±2\\pm 2; I only listed one\. </reflection\><answer\> \\boxed\{2\} </answer\>

## Appendix IImplementation Details for Baselines\.

To ensure a fair comparison and reproducibility, all trainable baselines are implemented with the same backbone models, training data, and parameter\-efficient fine\-tuning strategy as REIN\. Specifically, we use LoRA\-based fine\-tuning and keep the optimization configurations consistent whenever applicable\. Each baseline is trained using its original objective, while the inference\-time decoding and evaluation protocol are unified across all methods\. The implementation details of each baseline are described as follows:

- •IDK Prompting\.IDK Prompting is an inference\-time baseline that introduces explicit abstention instructions into the prompt\. The model is encouraged to respond with “I don’t know” when it is uncertain or when the problem is beyond its capability\. No additional training is performed, and the original model parameters remain unchanged\.
- •Reflection\-only\.Reflection\-only preserves the reflection format and reflection generation process used in REIN, allowing the model to assess the reliability of its own reasoning\. However, it removes the boundary\-aware IDK component and does not include the knowledge\-boundary\-based abstention reward\. Therefore, this baseline evaluates the contribution of reflection alignment alone without selective abstention learning\.
- •R\-Tuning\.We reproduce R\-Tuning following its original uncertainty\-aware instruction tuning objective\. The model is trained to explicitly produce “I don’t know” responses on questions beyond its capability while preserving answer generation on solvable questions\.
- •TruthRL\.We adopt the same GRPO optimization framework and training configuration as REIN to ensure a controlled comparison\. The only difference is the reward formulation\. Following the original TruthRL design, the reward assigns positive feedback to correct answers, neutral reward to explicit abstention, and negative reward to incorrect answers\.