StepGap: A Hybrid NLI-LLM Checker for Step-Level Evidence-Gap Detectionin Multi-Hop Question Answering

# A Hybrid NLI-LLM Checker for Step-Level Evidence-Gap Detection in Multi-Hop Question Answering
Source: [https://arxiv.org/html/2605.24733](https://arxiv.org/html/2605.24733)
Yuelyu Ji School of Computing and Information University of Pittsburgh yueluji@gmail\.com &Zhuochun Li School of Computing and Information University of Pittsburgh &Daqing He School of Computing and Information University of Pittsburgh dah44@pitt\.edu Yuelyu Ji1,Zhuochun Li1,Hui Ji1,Daqing He1

1School of Computing and Information, University of Pittsburgh

###### Abstract

We presentStepGap, a hybrid NLI\-LLM decision tree that detects step\-level evidence gaps in multi\-hop QA and emits one of three typed labels:Contradicted Claim\(CC\),Irrelevant Evidence\(IE\), orMissing Bridge\(MB\), each tied to a concrete repair action\. On 82 multi\-hop questions \(181 annotated steps,κ=0\.704\\kappa\{=\}0\.704\), StepGap reaches sF1==72\.0, within the bootstrap confidence interval of an LLM\-only baseline \(70\.1\) but with a more decomposable structure: every StepGap stage*hurts*F1 when removed, while three of four LLM\-only removals*improve*F1—a sign of*competing\-error cancellation*, where internal stages mask each other’s errors\. We further expose a*Q\-F1 trap*: question\-level F1 is mechanically inflated by checkers that flag every step, making step\-level F1 the necessary diagnostic\. Used as a typed GRPO process reward, StepGap improves Qwen2\.5\-7B\-Instruct Exact Match from32\.1±0\.332\.1\{\\pm\}0\.3to35\.4±0\.935\.4\{\\pm\}0\.9across three seeds, with the single\-run comparison showing a\+5\.6\+5\.6Avg EM gain over the matched Search\-R1 GRPO reproduction\.

StepGap: A Hybrid NLI\-LLM Checker for Step\-Level Evidence\-Gap Detection in Multi\-Hop Question Answering

Yuelyu Ji1, Zhuochun Li1, Hui Ji1, Daqing He11School of Computing and Information, University of Pittsburgh

††Code, prompts, and diagnostics available at:[https://anonymous\.4open\.science/r/step\-checker\-0925/](https://anonymous.4open.science/r/step-checker-0925/)![Refer to caption](https://arxiv.org/html/2605.24733v1/x1.png)Figure 1:StepGap on a synthetic 4\-step trace\. Endpoint EM only sees the wrong final answer; StepGap typed\-localizes CC, IE, and MB at the steps where each gap first emerges, with each gap type mapped to a distinct repair action\.## 1Introduction

Multi\-hop question answering \(QA\) requires retrieving, connecting, and verifying evidence across several intermediate stepsYanget al\.\([2018](https://arxiv.org/html/2605.24733#bib.bib10)\); Hoet al\.\([2020](https://arxiv.org/html/2605.24733#bib.bib13)\); Trivediet al\.\([2022](https://arxiv.org/html/2605.24733#bib.bib11)\)\. Yet both evaluation and reinforcement\-learning training in search\-interleaved frameworks remain endpoint\-oriented: systems are scored and rewarded on final\-answer Exact Match \(EM\) alone\.

Endpoint\-only signals have two structural defects\.*First*, EM cannot distinguish an evidence\-grounded chain from a spuriously correct one: a policy can reach the right answer through shortcuts that bypass retrieval or fill missing facts from parametric memoryJinet al\.\([2025](https://arxiv.org/html/2605.24733#bib.bib34)\); Minet al\.\([2023](https://arxiv.org/html/2605.24733#bib.bib27)\)\.*Second*, EM provides no localization on wrong answers: the policy receives zero reward with no signal about*which*step broke, so it cannot learn step\-specific repair behaviorLightmanet al\.\([2023](https://arxiv.org/html/2605.24733#bib.bib29)\); Luet al\.\([2025](https://arxiv.org/html/2605.24733#bib.bib18)\)\. We call such localized step\-level failures*evidence gaps*, and distinguish three gap types\. AContradicted Claim\(CC\) occurs when a step asserts a claim that conflicts with the retrieved evidence;Irrelevant Evidence\(IE\) occurs when the evidence does not address the entity or relation needed by the step; andMissing Bridge\(MB\) occurs when the evidence is relevant but does not yet entail the step’s claim\. While step\-level error taxonomies have been explored before[Deshpandeet al\.](https://arxiv.org/html/2605.24733#bib.bib44); Luet al\.\([2025](https://arxiv.org/html/2605.24733#bib.bib18)\), ourCC/IE/MBtypes are explicitly designed for*repair\-action coupling*: each maps to a single concrete operation \(retract / re\-search / bridging search\) that an RL policy can be trained to perform\. Figure[1](https://arxiv.org/html/2605.24733#S0.F1)illustrates this: three structurally distinct gap types can cascade into a single undifferentiated EM==0—a contradicted claim requiring retraction, irrelevant evidence requiring re\-search, and a missing bridge requiring an additional bridging search\.

Existing step\-level checkers either collapse all subtasks into a single LLM callXionget al\.\([2025b](https://arxiv.org/html/2605.24733#bib.bib33)\); Wanget al\.\([2024](https://arxiv.org/html/2605.24733#bib.bib39)\)or rely on a single NLI callTanget al\.\([2024](https://arxiv.org/html/2605.24733#bib.bib5)\); Laureret al\.\([2024](https://arxiv.org/html/2605.24733#bib.bib54)\), neither of which simultaneously emits typed labels and grounds them in deterministic decisions\. Post\-hoc and claim\-level verifiersGaoet al\.\([2023](https://arxiv.org/html/2605.24733#bib.bib20)\); Minet al\.\([2023](https://arxiv.org/html/2605.24733#bib.bib27)\); Ruet al\.\([2024](https://arxiv.org/html/2605.24733#bib.bib25)\)operate after a complete answer and provide no training\-time signal at all\.StepGapis a five\-stage hybrid decision tree: three LLM stages handle open\-ended reading subtasks \(alignment, abstention, entity consistency and quote extraction\), and two NLI stages handle deterministic entailment \(local and cross\-step\)\. The tree terminates at a typed gap label \(CC/IE/MB\) coupled to a concrete repair action, and the typed output enters GRPO as a per\-step process reward, training the policy to perform the prescribed repair when a gap fires\.

#### Our contribution\.

We make three claims\.First, typed step\-level gap labels are useful only when they are coupled to repair actions:CCmaps to retraction,IEto re\-search, andMBto bridging search\.

Second, StepGap is more decomposable than an LLM\-only checker\. On the 181\-step human benchmark, StepGap reaches sF1==72\.0, within the bootstrap confidence interval of the LLM\-only baseline \(70\.1\), but every StepGap component hurts F1 when removed\. In contrast, three of four LLM\-only removals improve F1, revealing*competing\-error cancellation*\. We also expose a*Q\-F1 trap*, showing why question\-level detection must be paired with step\-level F1 and balanced accuracy\.

Third, the typed signal works as a process reward\. A Typed\+Shape GRPO reward built on StepGap improves Qwen2\.5\-7B EM from32\.1±0\.332\.1\{\\pm\}0\.3to35\.4±0\.935\.4\{\\pm\}0\.9across three seeds and raises answer\-claim support to71\.5%71\.5\\%; a distilled Qwen2\.5\-7B student retains87%87\\%of teacher F1 with no API calls\.

![Refer to caption](https://arxiv.org/html/2605.24733v1/x2.png)Figure 2:StepGap pipeline\.\(1\)The generator produces a search\-interleaved trace; each stepsi=\(ci,qi,ei,ai\)s\_\{i\}=\(c\_\{i\},q\_\{i\},e\_\{i\},a\_\{i\}\)is fed to the checker with accumulated evidencee≤ie\_\{\\leq i\}\.\(2\)The hybrid checker traverses five typed stages \(LLM: A alignment, B abstention, C entity\+quote; NLI: D local entailment, E cross\-step verification\) and emitsτi∈\{no\-gap,CC,IE,MB\}\\tau\_\{i\}\\in\\\{\\text\{no\-gap\},\\textsc\{CC\},\\textsc\{IE\},\\textsc\{MB\}\\\}\.\(3\)τi\\tau\_\{i\}drives a typed GRPO reward \(rbase\+rshaper^\{\\text\{base\}\}\+r^\{\\text\{shape\}\}, §[5](https://arxiv.org/html/2605.24733#S5)\), closing the loop back to the generator\.CC= Contradicted Claim;IE= Irrelevant Evidence;MB= Missing Bridge\.

## 2Related Work

#### Search\-interleaved reasoning\.

IRCoTTrivediet al\.\([2023](https://arxiv.org/html/2605.24733#bib.bib12)\), ReActYaoet al\.\([2023](https://arxiv.org/html/2605.24733#bib.bib14)\), Search\-R1Jinet al\.\([2025](https://arxiv.org/html/2605.24733#bib.bib34)\), and R1\-SearcherSonget al\.\([2025](https://arxiv.org/html/2605.24733#bib.bib35)\)train search policies with final\-answer EM rewards and cannot attribute failure to specific steps; on a wrong answer, EM\-only reward provides no gradient about which step broke\. StepGap supplies a per\-step typed signal during rollout, enabling the generator to learn gap\-specific repair behaviors rather than a uniform “avoid errors” objective\.

#### Post\-hoc verification\.

RARRGaoet al\.\([2023](https://arxiv.org/html/2605.24733#bib.bib20)\), CRITICGouet al\.\([2023](https://arxiv.org/html/2605.24733#bib.bib21)\), and Chain\-of\-VerificationDhuliawalaet al\.\([2023](https://arxiv.org/html/2605.24733#bib.bib22)\)apply verification*after*a complete answer is produced, so they cannot localize mid\-chain failures or provide training\-time signal\. StepGap is invoked inline at each step during rollout, redirecting the generator before the chain propagates a failure\.

#### Claim\-level faithfulness\.

FActScoreMinet al\.\([2023](https://arxiv.org/html/2605.24733#bib.bib27)\), MiniCheckTanget al\.\([2024](https://arxiv.org/html/2605.24733#bib.bib5)\), SelfCheckGPTManakulet al\.\([2023](https://arxiv.org/html/2605.24733#bib.bib23)\), RAGASEset al\.\([2023](https://arxiv.org/html/2605.24733#bib.bib26)\), and RAGCheckerRuet al\.\([2024](https://arxiv.org/html/2605.24733#bib.bib25)\)score whether generated output is supported by retrieved evidence, but do not capture reasoning\-chain structure \(e\.g\., whether an intermediate query targeted the right entity\); a model trained on output\-level faithfulness can still reach correct answers via ungrounded intermediate steps\. StepGap evaluates each step against its own evidence, catching failures where they first occur\.

#### Process rewards and agentic RAG\.

Process Reward ModelsLightmanet al\.\([2023](https://arxiv.org/html/2605.24733#bib.bib29)\); Wanget al\.\([2024](https://arxiv.org/html/2605.24733#bib.bib39)\)provide step\-wise scalar supervision, with ReARTeRSunet al\.\([2025](https://arxiv.org/html/2605.24733#bib.bib41)\)adding MCTS, and RAG\-GymXionget al\.\([2025b](https://arxiv.org/html/2605.24733#bib.bib33)\)and ProRAG using LLM\-judge process rewards\. TRAIL[Deshpandeet al\.](https://arxiv.org/html/2605.24733#bib.bib44)introduces a trace\-level typed error ontology, and Doctor\-RAGLuet al\.\([2025](https://arxiv.org/html/2605.24733#bib.bib18)\)localizes the earliest failing step for post\-hoc repair\. We extend this line on three axes simultaneously:*step granularity*,*NLI\-anchored typed labels*, and*repair\-action coupling*\. This is a conjunction that is jointly necessary for a typed behavioral reward but has not previously been combined in a single system\.

#### Hybrid verifiers\.

NLI–LLM hybrids have been explored for output\-level factualityXionget al\.\([2025a](https://arxiv.org/html/2605.24733#bib.bib7)\); Liuet al\.\([2025](https://arxiv.org/html/2605.24733#bib.bib2)\); Tanget al\.\([2024](https://arxiv.org/html/2605.24733#bib.bib5)\); Jiet al\.\([2025](https://arxiv.org/html/2605.24733#bib.bib3)\)but not as process\-reward signals\. Scoring a final answer and scoring an intermediate step are structurally different; we show that an LLM\-only checker, even a strong one, collapses to near\-uniform IE labels when applied per\-step \(Appendix[I](https://arxiv.org/html/2605.24733#A9)\), motivating the hybrid design\. To our knowledge, prior work has not jointly demonstrated: \(i\) hybrid checkers exhibit a healthier ablation profile than LLM\-only variants for step\-level gap detection, and \(ii\) such checkers can serve as typed process rewards for search\-interleaved RL training\.

## 3Problem Formulation

#### Reasoning traces\.

We model a search\-interleaved multi\-hop QA trace as an ordered sequence𝒯=\{s1,…,sn\}\\mathcal\{T\}=\\\{s\_\{1\},\\ldots,s\_\{n\}\\\}\. Eachstepsis\_\{i\}is a tuplesi=\(ci,qi,ei,ai\)s\_\{i\}=\(c\_\{i\},q\_\{i\},e\_\{i\},a\_\{i\}\), wherecic\_\{i\}is the model’s intermediate reasoning text \(a natural\-language claim or sub\-conclusion produced before the action\),qiq\_\{i\}is an optional<search\>query,eie\_\{i\}is the evidence retrieved byqiq\_\{i\}, andaia\_\{i\}is an optional<answer\>block emitting the final answera^\\hat\{a\}\. We callsis\_\{i\}an*inference step*whenqi≠∅q\_\{i\}\\neq\\varnothingandai=∅a\_\{i\}=\\varnothing\(the model issues a search and writes a sub\-claim\), and a*conclusion step*whenai≠∅a\_\{i\}\\neq\\varnothing\(the model commits to the final answer\)\. The checker takes the entire tuple plus all earlier evidence as input, so its judgment depends jointly on the intermediate claim, the search query, the retrieved snippets, and the answer if present\.

Throughout,steprefers to a unit of the reasoning trace \(sis\_\{i\}\);stagerefers to a node in the checker’s decision tree \(StagesA–E, indexed by letters to visually separate them fromsis\_\{i\}\)\. One checker invocation traverses up to five stages on a single reasoning step\.

#### Step\-level evidence gap detection\.

Stepsis\_\{i\}is*gap\-free*if its claim—cic\_\{i\}whensis\_\{i\}is an inference step,aia\_\{i\}whensis\_\{i\}is a conclusion step—is logically entailed by the accumulated evidence⋃j≤iej\\bigcup\_\{j\\leq i\}e\_\{j\}\. Otherwise, the step has a*gap*, characterized by gap typeτ∈\{CC,IE,MB\}\\tau\\in\\\{\\textsc\{CC\},\\textsc\{IE\},\\textsc\{MB\}\\\}\. The checker𝒞\\mathcal\{C\}maps each\(si,e≤i\)\(s\_\{i\},e\_\{\\leq i\}\)to a structured output\(τ,confidence,rationale\)\(\\tau,\\textit\{confidence\},\\textit\{rationale\}\)\. The five stages of𝒞\\mathcal\{C\}are described in Section[4](https://arxiv.org/html/2605.24733#S4)\.

## 4Hybrid Step\-Gap Checker: Decision Tree

For each stepsi=\(ci,qi,ei,ai\)s\_\{i\}=\(c\_\{i\},q\_\{i\},e\_\{i\},a\_\{i\}\)with accumulated evidencee≤ie\_\{\\leq i\}, the checker traverses up to five stages \(Figure[2](https://arxiv.org/html/2605.24733#S1.F2), panel 2\)\. Each stage either emits a typed gap label and stops, or hands off to the next stage; if no stage fires, the step is labeled no\-gap\. We organize this section by traversal order \(Stage A through Stage E\) so that the tree’s structure is visible end\-to\-end; Table[1](https://arxiv.org/html/2605.24733#S4.T1)summarizes which stages can emit which gap type\. Below,*the step’s claim*meanscic\_\{i\}whensis\_\{i\}is an inference step andaia\_\{i\}whensis\_\{i\}is a conclusion step\. The search queryqiq\_\{i\}is consulted only at Stage A and is never used as an NLI hypothesis\.

Table 1:Stages of the StepGap decision tree, the gap type each can emit, and the NLI\-grounded typed signal at that node\. Stages D–E are NLI; A–C are LLM\. Each gap type maps to a single repair action used downstream as the typed reward signal \(§[5](https://arxiv.org/html/2605.24733#S5)\)\.#### Stage A – Alignment\.

Stage A checks whether the step targets the correct entity and relation,*before*retrieval is consulted\. This matters because a mis\-targeted step cannot be rescued by retrieval—any fetched document is irrelevant by construction, and surface\-token overlap can masquerade as support\. We classify mis\-targeting into three drift patterns—relation drift \(wrong predicate\), entity drift \(wrong subject\), and scope drift \(sub\-part vs\. whole\)— and emitCCon any drift\.

#### Stage B – Abstention validity\.

Stage B handles steps that assert “cannot be determined,” verifying that the abstention is grounded in the evidence\. An unjustified abstention is functionally a contradicted claim: the step commits to “the answer is unknown” when the evidence rules that out\. A grounded abstention exits to no\-gap; a wrong abstention emitsCC\.

#### Stage C – Entity consistency and quote\.

Stage C verifies that the retrieved evidence is about the targeted entity and, if so, extracts a verbatim supporting quote\. Without this filter, off\-topic documents sharing lexical tokens with the claim would pass to NLI as spurious premises\. If the page’s title and lead sentence describe a different entity, Stage C emitsIEdirectly; otherwise the checker finds a 5–20 word verbatim span on the same entity, relevant to the claim\. A missing quote routes a conclusion step to Stage E and an inference step to no\-gap \(a search query is not a well\-formed NLI hypothesis\)\.

#### Stage D – Local entailment\.

Stage D decides whether the Stage C quote actually entails the step’s claim\. Entity match plus a returned quote is necessary but not sufficient: the quote can be on\-topic yet fail to imply the claim \(a missing\-bridge case\) or overtly contradict it\. A single NLI call with the quote as premise and the claim as hypothesis resolves this:entailment→\\tono\-gap;neutral→\\toMB\(*repair: bridging search*\);contradiction→\\toCC\.

#### Stage E – Cross\-step verification\.

Stage E handles conclusion steps where Stage C found no local quote, asking whether the claim is entailed by evidence retrieved at earlier inference steps\. Multi\-hop answers typically rest on facts established several steps earlier, so the effective evidence pool is the union of all prior retrievals\. Stage E iterates over entity\-matched prior snippets as candidate premises: an entailing snippet exits to no\-gap; otherwise the step emitsIE\(*repair: re\-search*\)\. The entity\-match constraint prevents off\-topic earlier retrievals from producing spurious entailments\.

## 5Using the Typed Signal as a Process Reward

A checker that only*detects*gaps does not change policy behavior: the model can detect a gap at stepiiand still commit at stepi\+1i\{\+\}1\. We inject the typed output into RL training as a per\-step process reward so the policy is reinforced to execute the prescribed repair when a gap fires\. The reward has two parts: a base scalar conditioned on gap type \(§[5\.1](https://arxiv.org/html/2605.24733#S5.SS1)\) and an action\-conditional shaping term that verifies the repair is performed \(§[5\.2](https://arxiv.org/html/2605.24733#S5.SS2)\); both plug into GRPO unmodified \(§[5\.3](https://arxiv.org/html/2605.24733#S5.SS3)\)\.

### 5\.1Base Reward by Gap Type

Letτi∈\{no\-gap,MB,IE,CC\}\\tau\_\{i\}\\in\\\{\\text\{no\-gap\},\\text\{MB\},\\text\{IE\},\\text\{CC\}\\\}be the gap type emitted by the checker for stepsis\_\{i\}\. The base reward is

rbase​\(τi\)=\{\+0\.20no\-gap−0\.05MB−0\.10IE\+0\.05CC detectedr^\{\\text\{base\}\}\(\\tau\_\{i\}\)=\\begin\{cases\}\+0\.20&\\text\{no\-gap\}\\\\ \-0\.05&\\text\{MB\}\\\\ \-0\.10&\\text\{IE\}\\\\ \\mathbf\{\+0\.05\}&\\text\{CC detected\}\\end\{cases\}\(1\)
Three choices are non\-obvious\.*No\-gap is positive*\(\+0\.20\+0\.20\), following standard process\-reward practiceLightmanet al\.\([2023](https://arxiv.org/html/2605.24733#bib.bib29)\); Wanget al\.\([2024](https://arxiv.org/html/2605.24733#bib.bib39)\); Xionget al\.\([2025b](https://arxiv.org/html/2605.24733#bib.bib33)\); Wuet al\.\([2025b](https://arxiv.org/html/2605.24733#bib.bib55)\)\.*CC receives a small detection bonus*\(\+0\.05\+0\.05\): the reward is assigned after the checker detects a contradiction, not to the contradicted claim itself\. Penalizing all detected CC labels would instead teach the generator to hide contradictions behind vaguer claims that fewer NLI entailment checks can rejectJianget al\.\([2025](https://arxiv.org/html/2605.24733#bib.bib48)\)\. The bonus is kept small so it does not compete with EM and only rewards exposing a contradiction that the next\-step shaping term can require the policy to retract\.\.*IE is penalized more than MB*\(−0\.10\-0\.10vs\.−0\.05\-0\.05\): IE requires reformulating the query, MB only a bridging search; the asymmetry pushes the generator toward better initial queries\.

### 5\.2Behavioral Shaping Reward

We add a*shaping reward*conditional on the repair action at stepsis\_\{i\}given the gap type at stepsi−1s\_\{i\-1\}, following the action\-conditional design ofRAG\-GymXionget al\.\([2025b](https://arxiv.org/html/2605.24733#bib.bib33)\)and the process\-reward formulation ofZhanget al\.\([2025](https://arxiv.org/html/2605.24733#bib.bib57)\)\.

rshape​\(τi−1,si\)=\{\+0\.10IE/MBi−1∧<s\>i∧qi≠q<i\+0\.15CCi−1∧retracti−0\.15IE/MB/CCi−1∧<a\>i−0\.05IE/MBi−1∧<s\>i∧qi≈q<i0otherwiser^\{\\text\{shape\}\}\(\\tau\_\{i\-1\},s\_\{i\}\)=\\begin\{cases\}\{\+0\.10\}&\\text\{IE/MB\}\_\{i\-1\}\\wedge\\texttt\{<s\>\}\_\{i\}\\wedge q\_\{i\}\\neq q\_\{<i\}\\\\ \{\+0\.15\}&\\text\{CC\}\_\{i\-1\}\\wedge\\text\{retract\}\_\{i\}\\\\ \{\-0\.15\}&\\text\{IE/MB/CC\}\_\{i\-1\}\\wedge\\texttt\{<a\>\}\_\{i\}\\\\ \{\-0\.05\}&\\text\{IE/MB\}\_\{i\-1\}\\wedge\\texttt\{<s\>\}\_\{i\}\\wedge q\_\{i\}\\approx q\_\{<i\}\\\\ 0&\\text\{otherwise\}\\end\{cases\}\(2\)
Hereqiq\_\{i\}denotes the query string of a search step; we considerqi≈q<iq\_\{i\}\\approx q\_\{<i\}if its token\-F1 against any prior query exceeds 0\.7—a threshold selected on the same200200\-iteration pilot used for base reward weights \(Appendix[H](https://arxiv.org/html/2605.24733#A8)\)\. Retraction is detected by a small set of surface patterns \(“actually”, “wait”, “correction”, “I was wrong”\), combined with a check that the new claim differs from the previous one\.

The shaping rewardrshaper^\{\\text\{shape\}\}defined above implements the principle “detection without repair is failure”: ignoring the checker after a gap costs−0\.15\-0\.15\(a step\-level analogue of Reflexion’s self\-correctionShinnet al\.\([2023](https://arxiv.org/html/2605.24733#bib.bib37)\)\), executing the repair earns\+0\.10\+0\.10\(re\-search\) or\+0\.15\+0\.15\(retraction; CC’s higher shaping bonus reflects the larger behavioral cost of retracting a contradicted claim\), and lazy near\-duplicate queries earn−0\.05\-0\.05to prevent gaming the shaping signal \(cf\.Wuet al\.\([2025a](https://arxiv.org/html/2605.24733#bib.bib56)\)\)\.

### 5\.3Plugging the Typed Reward into GRPO

The checker enters GRPOShaoet al\.\([2024](https://arxiv.org/html/2605.24733#bib.bib49)\)only at the return\-computation step\. During rollout,𝒞\\mathcal\{C\}is invoked once per emitted stepsis\_\{i\}, returningτi\\tau\_\{i\}, which is cached with the step’s token span\[ℓi,ri\)\[\\ell\_\{i\},r\_\{i\}\)\. The trajectory return is

R=rEM\+λ​∑i=1n\[rbase​\(τi\)\+rshape​\(τi−1,si\)\],R=r^\{\\text\{EM\}\}\+\\lambda\\sum\_\{i=1\}^\{n\}\\\!\\left\[r^\{\\text\{base\}\}\(\\tau\_\{i\}\)\+r^\{\\text\{shape\}\}\(\\tau\_\{i\-1\},s\_\{i\}\)\\right\],\(3\)withrEM∈\{0,1\}r^\{\\text\{EM\}\}\\in\\\{0,1\\\},λ=1\.0\\lambda=1\.0, andτ0=no\-gap\\tau\_\{0\}\\,\{=\}\\,\\text\{no\-gap\}at the boundary\. We use the*dense*variant: each step’s bracket is assigned only to tokens in\[ℓi,ri\)\[\\ell\_\{i\},r\_\{i\}\), and the group\-relative standardizationA^=\(R−μR\)/σR\\hat\{A\}=\(R\-\\mu\_\{R\}\)/\\sigma\_\{R\}is computed token\-wise\. The clip ratio, KL penalty, and standardization are kept at GRPO defaults; only the returnRRin Eq\. \([3](https://arxiv.org/html/2605.24733#S5.E3)\) differs across our reward ablations, so any EM gain attributes to the typed checker signal and shaping rather than to optimizer re\-tuning\.

## 6Checker Evaluation

### 6\.1Human\-Annotated Evaluation Set

We sample 82 questions from HotpotQA, 2WikiMultiHopQA, and MuSiQue, run Qwen2\.5\-7B\-Instruct to produce full search\- interleaved traces, and manually annotate 181 individual steps \(see Appendix[B](https://arxiv.org/html/2605.24733#A2)for protocol\)\. Each step is labeled as no\-gap or gap \(with type∈\\in\{\\\{CC,IE,MB\}\\\}\)\. Two annotators independently label the test subset\. Inter\-annotator agreement is Cohen’sκ=0\.704\\kappa=0\.704, which we interpret as moderate agreement perMcHugh \([2012](https://arxiv.org/html/2605.24733#bib.bib4)\), consistent with the inherently subjective nature of distinguishingMBfromIEnear the decision boundary\. Disagreements were adjudicated by a third senior annotator, and the post\-adjudication consensus serves as the ground\-truth label set used in all evaluations \(full protocol in Appendix[B](https://arxiv.org/html/2605.24733#A2)\)\.

Per\-typeκ\\kappa\(reported in Appendix[B](https://arxiv.org/html/2605.24733#A2)\) is highest for IE and CC, and lower for MB, where MB’s low frequency \(n=9n\{=\}9\) inflates sampling variance\. Table[2](https://arxiv.org/html/2605.24733#S6.T2)reports dataset composition\.

#### Silver\-standard scale check\.

To complement the181181\-step gold evaluation set, we measure StepGap’s agreement with the GPT\-4\.1\-mini\+NLI teacher checker on the1,0001\{,\}000\-step distillation pool \(Appendix[C](https://arxiv.org/html/2605.24733#A3), disjoint from the human benchmark\)\. Treating the teacher checker’s labels as a*silver\-standard*reference—i\.e\., a high\-quality but model\-generated label set that substitutes for human gold labels when annotation scale is impractical—StepGap reachesF1=0\.88\\mathrm\{F1\}\{=\}0\.88on the binaryhas\_gapdimension andF1=0\.81\\mathrm\{F1\}\{=\}0\.81on typed CC/IE/MB\. This high agreement at∼5×\\sim 5\\timesthe gold\-evaluation\-set scale indicates that StepGap’s behavior on the 181\-step human evaluation set is representative of its behavior at scale, not a sampling artifact\.

Table 2:Human\-annotated evaluation set statistics\. Gap% = fraction of steps labeled as gap; Acc% = question\-level accuracy\. GRPO training uses 169K NQ\+HotpotQA questions \(PeterJinGo/nq\_hotpotqa\_train\); standard dev splits: HotpotQA 7,405 Q, 2Wiki 12,576 Q, MuSiQue 2,417 Q \(Appendix[P\.1](https://arxiv.org/html/2605.24733#A16.SS1)\)\.We compare StepGap against four internal variantsLLM\-StrictandLLM\-Only: GPT\-4\.1\-mini at every stage, differing only in the gap\-flagging threshold \(LLM\-Strict: any single stage\-level gap signal suffices;LLM\-Only: requires the checker’s overall confidence≥0\.5\\geq 0\.5\);Single\-LLM\-XL: GPT\-5\-mini at every stage, no NLI;NER\-Quote: spaCy NER replacing Stage C’s LLM quote search\) and two external baselines \(MiniCheckTanget al\.\([2024](https://arxiv.org/html/2605.24733#bib.bib5)\);NLI\-onlyLaureret al\.\([2024](https://arxiv.org/html/2605.24733#bib.bib54)\): StepGap’s DeBERTa\-v3 applied directly to \(full evidence, claim\) without LLM stages\)\. We also considered answer\-level faithfulness checkers \(RAGASEset al\.\([2023](https://arxiv.org/html/2605.24733#bib.bib26)\), FActScoreMinet al\.\([2023](https://arxiv.org/html/2605.24733#bib.bib27)\), RAGCheckerRuet al\.\([2024](https://arxiv.org/html/2605.24733#bib.bib25)\)\), but these score*complete answers*rather than intermediate steps and are therefore not directly comparable as step\-level checkers; among them MiniCheck is the closest step\-applicable representative and is included as a baseline\. Table[3](https://arxiv.org/html/2605.24733#S6.T3)reports performance with bootstrap95%95\\%CIs\. We report both step\-level F1 \(sF1: per step, doeshas\_gapmatch the human label?\) and question\-level F1 \(Q\-F1: on each wrong\-answer question, did the checker detect any gap?\), sF1 paired with balanced accuracy \(BA\) is the more diagnostic of the two: Q\-F1 fires whenever*any*gap is detected on a wrong\-answer question, so a checker that flags every step trivially achieves near\-ceiling Q\-F1 without distinguishing real gaps from spurious ones\. sF1, in contrast, penalizes per\-step false positives, and BA further corrects for the∼60%\\sim 60\\%gap\-rate class imbalance—together they capture both detection coverage and precision\.

Table 3:Step\-level performance on the 181\-step evaluation set\. sF1/sP/sR: step\-level F1, precision, recall\. Q\-F1: a question is TP iff the checker detects≥1\\geq 1gap on a wrong\-answer chain\. BA: balanced accuracy\.†\\daggerdegenerate variants whose Q\-F1 is mechanically inflated by flagging nearly every step as IE \(§[6\.2](https://arxiv.org/html/2605.24733#S6.SS2)\)\.‡\\ddaggerbinary checkers without typed CC/IE/MB output\. Brackets: 2000\-iter percentile bootstrap 95% CIs\.VariantsF1 \[95% CI\]sPsRQ\-F1BANote*Internal variants \(full typed CC/IE/MB output\)*LLM\-Strict67\.2 \[60\.4, 73\.8\]60\.376\.089\.247\.8high FP, recall\-heavyLLM\-Only70\.1 \[63\.5, 76\.2\]69\.271\.281\.844\.4competing\-error baselineStepGap \(ours\)72\.0\[65\.6, 77\.8\]71\.073\.181\.260\.6best P/R balance, healthy ablation*Degenerate alternatives \(full output but collapsed onto IE\)*Single\-LLM\-XL69\.2 \[63\.0, 75\.1\]57\.287\.591\.4†50\.0†all flagged; IE==87\.8%, CC==0%NER\-Quote68\.0 \[61\.5, 74\.2\]56\.884\.691\.4†50\.0†NLI fires on noun phrases*Binary\-only baselines \(no typed gap output\)‡*NLI\-onlyLaureret al\.\([2024](https://arxiv.org/html/2605.24733#bib.bib54)\)61\.7 \[54\.3, 68\.2\]63\.353\.575\.249\.7no LLM stages; pure entailmentMiniCheckTanget al\.\([2024](https://arxiv.org/html/2605.24733#bib.bib5)\)67\.3 \[58\.3, 73\.2\]61\.272\.190\.451\.5external published checker
#### Statistical interpretation\.

*\(1\)*StepGap and LLM\-Only are statistically indistinguishable on top\-line sF1 \(CIs overlap atn=181n\{=\}181\)\. Our argument for StepGap instead rests on*compositional*ablation properties \(Appendix[D](https://arxiv.org/html/2605.24733#A4)\): every StepGap component removal*hurts*F1, meaning each stage contributes real signal that the rest of the system cannot replace—the checker is*decomposable*\. By contrast, three of four LLM\-only component removals*improve*F1, meaning those components were actively introducing errors that other components had been masking; this is*competing\-error cancellation*, and the checker is not decomposable\.*\(2\)*Single\-LLM\-XL and NER\-Quote expose what we call aQ\-F1 trap: Q\-F1 fires whenever*any*step is flagged on a wrong\-answer question, so a checker flagging every step achieves near\-ceiling Q\-F1 mechanically\. Both variants flag∼95%\{\\sim\}95\\%of steps and reach Q\-F1==91\.4 while sF1 collapses to≤69\\leq 69and BA to chance \(50%50\\%\)\. A flag\-everything checker reaches Q\-F1=2​w/\(1\+w\)=2w/\(1\+w\)in closed form, giving0\.67/0\.78/0\.910\.67/0\.78/0\.91atw=0\.5/0\.65/0\.84w\{=\}0\.5/0\.65/0\.84; a dummy reproduces this within±0\.02\\pm 0\.02on four benchmarks \(Appendix[L](https://arxiv.org/html/2605.24733#A12)\)\. Q\-F1 alone cannot distinguish a precise checker from one that always fires\.*\(3\)*The three metrics must be read jointly: Q\-F1 is inflated by flag\-everything \(point 2\), sF1 can mask class imbalance \(a no\-gap predictor scores∼60\\sim 60at our59%59\\%gap rate\), and BA alone ignores precision\. StepGap is the only variant competitive on all three \(sF1==72\.0, Q\-F1==81\.2, BA==60\.6%\); Single\-LLM\-XL wins Q\-F1 but loses sF1 and BA, falling into the trap\.

Our claim is not that StepGap outperforms LLM\-only on sF1 alone, but that its component ablations are*monotone*: every removal hurts F1\. This matters because a system whose components consistently add signal admits modular debugging and distillation \(Appendix[C](https://arxiv.org/html/2605.24733#A3)\)\. LLM\-only’s ablations swing oppositely \(\+1\.6/\+0\.8/\+1\.7\+1\.6/\+0\.8/\+1\.7vs−6\.5\-6\.5\), so stage effects are entangled\.

### 6\.2Category Distribution as a Health Indicator

The distribution of emitted gap types diagnoses*checker health*, orthogonal to sF1: a checker can score moderately on sF1 yet be useless as a typed reward source if it cannot reliably distinguish CC/IE/MB\. LLM\-Only and StepGap both produce IE in the3838–41%41\\%range, consistent with retrieval failure being the dominant mode in annotation\. We treat this 38–41% range as a benchmark\-specific sanity check rather than a universal target distribution\. The two degenerate variants leave this band entirely\. Single\-LLM\-XL collapses to87\.8%87\.8\\%IE with zero CC and zero MB: the strongest single LLM yields a*less*discriminating checker\. NER\-Quote reaches74%74\\%IE because spaCy noun\-phrase queries are not well\-formed NLI hypotheses, so Stage D over\-reports retrieval failure\. Both pathologies are threshold\-independent \(Appendix[N](https://arxiv.org/html/2605.24733#A14)\), confirming the collapse is architectural\. A confusion\-matrix view \(Appendix[J](https://arxiv.org/html/2605.24733#A10)\) shows StepGap flags a gap on56/6956/69wrong\-answer questions and12/1312/13correct\-answer questions, exposing the*lucky\-correct*regime endpoint EM cannot see; the full first\-gap distribution is in Appendix[K](https://arxiv.org/html/2605.24733#A11)\.

## 7GRPO Training with Typed Reward

### 7\.1Setup

A high\-quality checker does not by itself train a policy to fix the gaps it detects\. We therefore test whether StepGap’s typed outputs work as a per\-step process reward, holding backbone, training mix, and hyperparameters constant\. Only checkers emitting typed CC/IE/MB qualify: StepGap \(ours\) and LLM\-Only; binary support\-checkers \(MiniCheck, NLI\-only\) are excluded by design and remain evaluated on checker quality alone \(§[6](https://arxiv.org/html/2605.24733#S6)\)\.

We train Qwen2\.5\-7B\-Instruct with GRPOShaoet al\.\([2024](https://arxiv.org/html/2605.24733#bib.bib49)\)on NQ\+HotpotQA \(Table[2](https://arxiv.org/html/2605.24733#S6.T2)\); retrieval uses E5\-base\-v2Wanget al\.\([2022b](https://arxiv.org/html/2605.24733#bib.bib47)\)over Wikipedia 2018 \(top\-k=5k\{=\}5\)\. The teacher checker \(GPT\-4\.1\-mini\) is called∼72\{\\sim\}72K times per3,0003\{,\}000\-iteration run at∼$​22\{\\sim\}\\mathdollar 22; the distilled 7B student \(Appendix[C](https://arxiv.org/html/2605.24733#A3)\) eliminates this cost\. We compare four reward configurations:Search\-Only\(EM only\),Binary Gap\(\+0\.2\+0\.2no\-gap /0gap; prior design\),Typed Base, andTyped\+Shape \(ours\)\.

### 7\.2Main Results

Table[4](https://arxiv.org/html/2605.24733#S7.T4)reports per\-dataset and Avg EM\. The headline result is Typed\+Shape’s35\.435\.4Avg EM, a\+5\.6\+5\.6gain over our directly comparable Search\-R1 GRPO baseline \(29\.829\.8, same backbone and training mix\), with the gain holding across all three datasets\. Three\-seed mean±\\pmstd and the Llama\-3\.1\-8B backbone\-transfer study are in Appendix[A](https://arxiv.org/html/2605.24733#A1); auxiliary grounding metrics in Appendix[M](https://arxiv.org/html/2605.24733#A13)\.

Table 4:GRPO results on standard dev splits\.Top: published baselines comparable to ours \(same Qwen2\.5\-7B\-Instruct backbone\)\.Middle: external systems with non\-comparable training setup \(different base or different training mix\)\.Bottom: our reward ablations on the same NQ\+HotpotQA training mix and Qwen2\.5\-7B\-Instruct backbone as Search\-R1\. The directly comparable row is Search\-R1 \(GRPO\);Typed\+Shape gains\+5\.6\+5\.6Avg EM over it \(29\.8→35\.429\.8\\to 35\.4\)\. Bottom\-panel “Search\-Only” is the single\-seed Search\-R1 GRPO reproduction; three\-seed mean±\\pmstd for all our ablations \(where Search\-Only reaches32\.1±0\.332\.1\{\\pm\}0\.3\) is in Appendix[A](https://arxiv.org/html/2605.24733#A1), Table[5](https://arxiv.org/html/2605.24733#A1.T5)\.†Reproduction numbers as reported inLuoet al\.\([2025](https://arxiv.org/html/2605.24733#bib.bib51)\)on the standard Search\-R1 dev splits\.‡R1\-Searcher reports Cover\-EM in its own paper; the strict\-EM numbers shown here are an independent reproduction byTanet al\.\([2026](https://arxiv.org/html/2605.24733#bib.bib52)\)\. R1\-Searcher trains on HotpotQA\+2Wiki, so HotpotQA and 2Wiki are in\-domain for it; for our NQ\+HotpotQA mix, 2Wiki and MuSiQue are out\-of\-domain\.

Reward ablation\.We note that 2Wiki and MuSiQue are out\-of\-domain w\.r\.t\. our NQ\+\+HotpotQA training mix while HotpotQA is in\-domain; the consistent Typed\+Shape gain on both suggests the typed signal generalizes beyond the training distribution\.

Concretely, in the single\-run macro\-average setting, Typed\+Shape improves EM by\+2\.4\+2\.4over Binary Gap \(typed labels matter\) and by\+1\.2\+1\.2over Typed Base \(behavioral shaping matters\)\. Without shaping, the generator can detect a gap yet still commit to an answer; with shaping, mean search count rises from2\.52\.5to2\.82\.8, consistent with the policy learning to re\-search after IE/MB detections\. The three\-seed Qwen setting \(Appendix[A](https://arxiv.org/html/2605.24733#A1), Table[5](https://arxiv.org/html/2605.24733#A1.T5)\) confirms the same monotone ordering, with Typed\+Shape improving over Search\-Only by\+3\.3\+3\.3EM\. On strictly out\-of\-domain MuSiQue \(neither system trained on it\), Typed\+Shape outperforms R1\-Searcher by\+10\.9\+10\.9EM \(26\.726\.7vs\.15\.815\.8\); the latter’s higher Avg comes from its HotpotQA\+2Wiki in\-domain training data rather than a stronger reward design\.

Two\-phase CC dynamics\.The CC detection bonus produces a characteristic non\-monotone rate during training \(Figure[3](https://arxiv.org/html/2605.24733#S7.F3)\): CC rises early as the generator surfaces contradictions for the bonus, then falls as it learns to avoid making contradicted claims altogether\. A straight\-negative CC reward decreases CC monotonically but achieves it through vaguer, harder\-to\-check claims that hurt support rate; the positive bonus prevents this contradiction\-suppression failure mode\.

![Refer to caption](https://arxiv.org/html/2605.24733v1/x3.png)Figure 3:Two\-phase CC dynamics during 200 GRPO iterations\.\(a\)CC rate under our typed reward \(blue, \+0\.05 bonus\) vs\. a straight\-negative CC ablation \(red dashed\)\.\(b\)Companion grounding metrics \(no\-gap rate, answer support rate, IE rate\) under the typed reward\.Llama backbone transfer\.On Llama\-3\.1\-8B under identical hyperparameters, all checker\-augmented rewards underperform Search\-Only \(28\.128\.1\): Binary Gap collapses to19\.219\.2, Typed\+Shape reaches25\.425\.4\. We attribute this to backbone\-specific interaction with our Qwen\-tuned reward magnitudes \(see Limitations\) and treat Qwen as the headline; the Untyped\-Dense control is in Appendix[P\.2](https://arxiv.org/html/2605.24733#A16.SS2)\.

## 8Conclusion

We presentedStepGap, a hybrid NLI\-LLM checker that emits typed step\-level evidence\-gap labels \(CC/IE/MB\), each anchored at a specific NLI decision and tied to a concrete repair action\. Used as a typed GRPO process reward, StepGap improves Qwen2\.5\-7B\-Instruct EM by\+3\.3\+3\.3across three seeds\. Typed\+Shape obtains71\.5%71\.5\\%answer\-claim support, the highest among our reward ablations; a distilled Qwen2\.5\-7B student preserves87%87\\%of the teacher’s step\-F1 without API calls at inference\. Methodologically, we hope to leave the reader with the view that two checkers within each other’s F1 CIs can still expose qualitatively different process signals—one through monotone components, the other through competing\-error cancellation—and only the former is usable as a typed RL reward\.

## Limitations

#### Sample size and statistical power\.

Our human\-annotated evaluation set covers 181 steps across 82 questions; bootstrap95%95\\%CIs on sF1 overlap between StepGap, LLM\-Only, and several baselines \(Table[3](https://arxiv.org/html/2605.24733#S6.T3)\)\. We therefore frame our contribution on the architectural and diagnostic axes \(*healthy ablation pattern*,*Q\-F1 trap*,*typed signal*\) rather than as a top\-line F1 win, and we report seed\-level variance in Appendix[A](https://arxiv.org/html/2605.24733#A1)\(Table[5](https://arxiv.org/html/2605.24733#A1.T5)\)\. As partial mitigation of the small gold sample, we additionally report StepGap\-vs\-teacher agreement on the1,0001\{,\}000\-step distillation pool \(§[6](https://arxiv.org/html/2605.24733#S6), binaryF1=0\.88\\mathrm\{F1\}\{=\}0\.88, typedF1=0\.81\\mathrm\{F1\}\{=\}0\.81\), showing that StepGap’s binary gap\-detection behavior is consistent at∼5×\\sim 5\\timesthe gold\-benchmark scale\. Low\-frequency gap types \(MB,n=9n\{=\}9\) remain the main power\-limited case, and a larger MB\-targeted gold evaluation set is left to future work\. We note, however, that our 181\-step scale is comparable to step\-level annotation in the agentic\-RAG literature, and the silver\-standard1,0001\{,\}000\-step check addresses the architectural\-level concern at∼5×\\sim 5\\timesscale\.

#### Backbone\-specific reward tuning\.

The reward magnitudes \(§[5](https://arxiv.org/html/2605.24733#S5)\) were tuned on Qwen2\.5\-7B and do not transfer zero\-shot to Llama\-3\.1\-8B \(Table[5](https://arxiv.org/html/2605.24733#A1.T5)\); we hypothesize interaction between dense per\-step shaping and Llama’s instruction\-following distribution under a fixed KL budget, and leave per\-backbone retuning to future work\.

#### Other limitations\.

The shaping term relies on surface pattern matching for retraction detection \(e\.g\., “actually”, “wait”\), which may miss subtler rewordings and, conversely, is in principle vulnerable to reward hacking via spurious retraction\-pattern usage\. We did not observe such hacking in our200200\-iterations runs, but longer training could enable this failure mode and motivates a learned retraction classifier in future work\.

## References

- \[1\]Trail: trace reasoning and agentic issue localization, 2025\.URL https://arxiv\. org/abs/2505\.08638\.Cited by:[§1](https://arxiv.org/html/2605.24733#S1.p2.1),[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Dhuliawala, M\. Komeili, J\. Xu, R\. Raileanu, X\. Li, A\. Celikyilmaz, and J\. Weston \(2023\)Chain\-of\-verification reduces hallucination in large language models\.arXiv preprint arXiv:2309\.11495\.Cited by:[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Es, J\. James, L\. Espinosa\-Anke, and S\. Schockaert \(2023\)RAGAS: automated evaluation of retrieval augmented generation\.arXiv preprint arXiv:2309\.15217\.Cited by:[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px3.p1.1),[§6\.1](https://arxiv.org/html/2605.24733#S6.SS1.SSS0.Px1.p2.3)\.
- L\. Gao, Z\. Dai, P\. Pasupat, A\. Chen, A\. T\. Chaganty, Y\. Fan, V\. Y\. Zhao, N\. Lao, H\. Lee, D\. Juan, and K\. Guu \(2023\)RARR: researching and revising what language models say, using language models\.InProceedings of ACL,Cited by:[§1](https://arxiv.org/html/2605.24733#S1.p3.1),[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Gou, Z\. Shao, Y\. Gong, Y\. Shen, Y\. Yang, N\. Duan, and W\. Chen \(2023\)CRITIC: large language models can self\-correct with tool\-interactive critiquing\.arXiv preprint arXiv:2305\.11738\.Cited by:[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Ho, A\. D\. Nguyen, S\. Sugawara, and A\. Aizawa \(2020\)Constructing a multi\-hop QA dataset for comprehensive evaluation of reasoning steps\.InProceedings of COLING,Cited by:[§1](https://arxiv.org/html/2605.24733#S1.p1.1)\.
- Y\. Ji, W\. Lan, and P\. NG \(2025\)MRAG\-suite: a diagnostic evaluation platform for visual retrieval\-augmented generation\.arXiv preprint arXiv:2509\.24253\.Cited by:[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px5.p1.1)\.
- J\. Jiang, J\. Chen, J\. Li, R\. Ren, S\. Wang, W\. X\. Zhao, Y\. Song, and T\. Zhang \(2025\)Rag\-star: enhancing deliberative reasoning with retrieval augmented verification and refinement\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 7064–7074\.Cited by:[§5\.1](https://arxiv.org/html/2605.24733#S5.SS1.p2.4)\.
- B\. Jin, H\. Zeng, Z\. Yue, J\. Yoon, S\. Arik, D\. Wang, H\. Zamani, and J\. Han \(2025\)Search\-r1: training llms to reason and leverage search engines with reinforcement learning\.arXiv preprint arXiv:2503\.09516\.Cited by:[§1](https://arxiv.org/html/2605.24733#S1.p2.1),[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Laurer, W\. van Atteveldt, A\. Casas, and K\. Welbers \(2024\)Less annotating, more classifying: addressing the data scarcity issue of supervised machine learning with deep transfer learning and bert\-nli\.Political Analysis\.Cited by:[§O\.2](https://arxiv.org/html/2605.24733#A15.SS2.SSS0.Px5.p1.2),[§1](https://arxiv.org/html/2605.24733#S1.p3.1),[§6\.1](https://arxiv.org/html/2605.24733#S6.SS1.SSS0.Px1.p2.3),[Table 3](https://arxiv.org/html/2605.24733#S6.T3.13.14.7.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2023\)Let’s verify step by step\.arXiv preprint arXiv:2305\.20050\.Cited by:[§1](https://arxiv.org/html/2605.24733#S1.p2.1),[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px4.p1.1),[§5\.1](https://arxiv.org/html/2605.24733#S5.SS1.p2.4)\.
- W\. Liu, G\. Han, and X\. Huang \(2025\)Examining and adapting time for multilingual classification via mixture of temporal experts\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 6151–6166\.Cited by:[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px5.p1.1)\.
- Y\. Lu, G\. Fu, W\. Wu, X\. Zhao, S\. Y\. Goi, and J\. Wang \(2025\)Doctorrag: medical rag fusing knowledge with patient analogy through textual gradients\.arXiv preprint arXiv:2505\.19538\.Cited by:[§1](https://arxiv.org/html/2605.24733#S1.p2.1),[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px4.p1.1)\.
- J\. Luo, M\. Cheng, F\. Wan, N\. Li, X\. Xia, S\. Tian, T\. Bian, H\. Wang, H\. Fu, and Y\. Tao \(2025\)GlobalRAG: enhancing global reasoning in multi\-hop question answering via reinforcement learning\.arXiv preprint arXiv:2510\.20548\.Cited by:[Table 4](https://arxiv.org/html/2605.24733#S7.T4.13)\.
- P\. Manakul, A\. Liusie, and M\. J\. F\. Gales \(2023\)SelfCheckGPT: zero\-resource black\-box hallucination detection for generative large language models\.arXiv preprint arXiv:2303\.08896\.Cited by:[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px3.p1.1)\.
- M\. L\. McHugh \(2012\)Interrater reliability: the kappa statistic\.Biochemia medica22\(3\),pp\. 276–282\.Cited by:[§6\.1](https://arxiv.org/html/2605.24733#S6.SS1.p1.4)\.
- S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\. Yih, P\. W\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. Hajishirzi \(2023\)FActScore: fine\-grained atomic evaluation of factual precision in long form text generation\.InProceedings of EMNLP,Cited by:[§1](https://arxiv.org/html/2605.24733#S1.p2.1),[§1](https://arxiv.org/html/2605.24733#S1.p3.1),[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px3.p1.1),[§6\.1](https://arxiv.org/html/2605.24733#S6.SS1.SSS0.Px1.p2.3)\.
- D\. Ru, L\. Qiu, X\. Gu, T\. Zhao, S\. Ding, X\. Jia, P\. Shi, Y\. Yu, Q\. Zheng, M\. Geng,et al\.\(2024\)RAGChecker: a fine\-grained framework for diagnosing retrieval\-augmented generation\.InAdvances in NeurIPS Datasets and Benchmarks,Cited by:[§1](https://arxiv.org/html/2605.24733#S1.p3.1),[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px3.p1.1),[§6\.1](https://arxiv.org/html/2605.24733#S6.SS1.SSS0.Px1.p2.3)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§5\.3](https://arxiv.org/html/2605.24733#S5.SS3.p1.4),[§7\.1](https://arxiv.org/html/2605.24733#S7.SS1.p2.6)\.
- N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.arXiv preprint arXiv:2303\.11366\.Cited by:[§5\.2](https://arxiv.org/html/2605.24733#S5.SS2.p4.5)\.
- H\. Song, J\. Jiang, Y\. Min, J\. Chen, Z\. Chen, W\. X\. Zhao, L\. Fang, and J\. Wen \(2025\)R1\-searcher: incentivizing the search capability in llms via reinforcement learning\.arXiv preprint arXiv:2503\.05592\.Cited by:[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Sun, Q\. Wang, W\. Yu, X\. Zang, K\. Zheng, J\. Xu, X\. Zhang, Y\. Song, and H\. Li \(2025\)Rearter: retrieval\-augmented reasoning with trustworthy process rewarding\.InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 1251–1261\.Cited by:[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px4.p1.1)\.
- Z\. Tan, J\. Huang, Q\. Wu, H\. Zhang, C\. Zhuang, and J\. Gu \(2026\)Rag\-r1: incentivizing the search and reasoning capabilities of llms through multi\-query parallelism\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 33187–33195\.Cited by:[Table 4](https://arxiv.org/html/2605.24733#S7.T4.13)\.
- L\. Tang, P\. Laban, and G\. Durrett \(2024\)Minicheck: efficient fact\-checking of llms on grounding documents\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 8818–8847\.Cited by:[§1](https://arxiv.org/html/2605.24733#S1.p3.1),[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px5.p1.1),[§6\.1](https://arxiv.org/html/2605.24733#S6.SS1.SSS0.Px1.p2.3),[Table 3](https://arxiv.org/html/2605.24733#S6.T3.13.15.8.1)\.
- H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal \(2022\)MuSiQue: multihop questions via single\-hop question composition\.Transactions of the Association for Computational Linguistics10,pp\. 539–554\.Cited by:[§1](https://arxiv.org/html/2605.24733#S1.p1.1)\.
- H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal \(2023\)Interleaving retrieval with chain\-of\-thought reasoning for knowledge\-intensive multi\-step questions\.InProceedings of the 61st annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 10014–10037\.Cited by:[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Wang, N\. Yang, X\. Huang, B\. Jiao, L\. Yang, D\. Jiang, R\. Majumder, and F\. Wei \(2022a\)Text embeddings by weakly\-supervised contrastive pre\-training\.arXiv preprint arXiv:2212\.03533\.Cited by:[§P\.1](https://arxiv.org/html/2605.24733#A16.SS1.p1.4)\.
- L\. Wang, N\. Yang, X\. Huang, B\. Jiao, L\. Yang, D\. Jiang, R\. Majumder, and F\. Wei \(2022b\)Text embeddings by weakly\-supervised contrastive pre\-training\.arXiv preprint arXiv:2212\.03533\.Cited by:[§7\.1](https://arxiv.org/html/2605.24733#S7.SS1.p2.6)\.
- P\. Wang, L\. Li, Z\. Shao, R\. Xu, D\. Dai, Y\. Li, D\. Chen, Y\. Wu, and Z\. Sui \(2024\)Math\-shepherd: verify and reinforce LLMs step\-by\-step without human annotations\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 9426–9439\.External Links:[Link](https://aclanthology.org/2024.acl-long.510/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.510)Cited by:[§1](https://arxiv.org/html/2605.24733#S1.p3.1),[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px4.p1.1),[§5\.1](https://arxiv.org/html/2605.24733#S5.SS1.p2.4)\.
- J\. Wu, Z\. Deng, W\. Li, Y\. Liu, B\. You, B\. Li, Z\. Ma, and Z\. Liu \(2025a\)Mmsearch\-r1: incentivizing lmms to search\.arXiv preprint arXiv:2506\.20670\.Cited by:[§5\.2](https://arxiv.org/html/2605.24733#S5.SS2.p4.5)\.
- P\. Wu, M\. Zhang, K\. Wan, W\. Zhao, K\. He, X\. Du, and Z\. Chen \(2025b\)Hiprag: hierarchical process rewards for efficient agentic retrieval augmented generation\.arXiv preprint arXiv:2510\.07794\.Cited by:[§5\.1](https://arxiv.org/html/2605.24733#S5.SS1.p2.4)\.
- G\. Xiong, Z\. He, B\. Liu, S\. Sinha, and A\. Zhang \(2025a\)Toward faithful retrieval\-augmented generation with sparse autoencoders\.arXiv preprint arXiv:2512\.08892\.Cited by:[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px5.p1.1)\.
- G\. Xiong, Q\. Jin, X\. Wang, Y\. Fang, H\. Liu, Y\. Yang, F\. Chen, Z\. Song, D\. Wang, M\. Zhang,et al\.\(2025b\)Rag\-gym: optimizing reasoning and search agents with process supervision\.arXiv e\-prints,pp\. arXiv–2502\.Cited by:[§1](https://arxiv.org/html/2605.24733#S1.p3.1),[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px4.p1.1),[§5\.1](https://arxiv.org/html/2605.24733#S5.SS1.p2.4),[§5\.2](https://arxiv.org/html/2605.24733#S5.SS2.p1.2)\.
- Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning \(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 2369–2380\.Cited by:[§1](https://arxiv.org/html/2605.24733#S1.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.arXiv preprint arXiv:2210\.03629\.Cited by:[§2](https://arxiv.org/html/2605.24733#S2.SS0.SSS0.Px1.p1.1)\.
- W\. Zhang, X\. Li, K\. Dong, Y\. Wang, P\. Jia, X\. Li, Y\. Zhang, D\. Xu, Z\. Du, H\. Guo,et al\.\(2025\)Process vs\. outcome reward: which is better for agentic rag reinforcement learning\.arXiv preprint arXiv:2505\.14069\.Cited by:[§5\.2](https://arxiv.org/html/2605.24733#S5.SS2.p1.2)\.

## Appendix AGRPO Robustness: Seeds, Backbones, Untyped Control

Table[5](https://arxiv.org/html/2605.24733#A1.T5)reports GRPO EM \(mean±\\pmstd over 3 seeds\) on Qwen2\.5\-7B\-Instruct and Llama\-3\.1\-8B\-Instruct under identical hyperparameters\. On Qwen, the ordering Search\-Only<<Binary Gap<<Typed Base<<Typed\+Shape is monotone with all seed\-level std≤1\.2\\leq 1\.2EM, well below the\+3\.3\+3\.3headline gain\. On Llama all checker\-augmented rewards underperform Search\-Only; we discuss this backbone\-specific failure in §[3](https://arxiv.org/html/2605.24733#S7.F3)and explicitly limit our generalization claim to the Qwen backbone in the Limitations section\. The*Untyped\-Dense*control row \(matched per\-step density to Typed\+Shape but no CC/IE/MB distinction\) isolates the typing\-vs\-density confound on Qwen; the sparser Binary Gap row in Table[4](https://arxiv.org/html/2605.24733#S7.T4)provides a complementary density\-matched control\.

Table 5:GRPO EM \(mean±\\pmstd over 3 seeds\) across reward designs and two generator backbones\. On Qwen2\.5\-7B, Typed\+Shape gives a clear ordering Search\-Only<<Typed Base<<Typed\+Shape; on Llama\-3\.1\-8B, all checker\-augmented rewards*underperform*Search\-Only, with Binary Gap collapsing most severely\. We discuss this backbone\- specific failure in §[3](https://arxiv.org/html/2605.24733#S7.F3)and treat the Qwen result as the primary headline result\.
## Appendix BAnnotation Protocol

#### Annotators\.

Two annotators with graduate\-level NLP training labeled the test subset of 82 questions / 181 steps\. Both were familiar with multi\-hop QA conventions but were naive to the StepGap taxonomy at the start of training\.

#### Training\.

Annotators completed a 30\-minute calibration session on a held\-out set of 20 steps, with discussion of edge cases \(in particular, the boundary betweenMissing BridgeandIrrelevant Evidence\)\. Decision rules were finalized as: a step is IE if the retrieved evidence is about a different entity than the step’s claim addresses; MB if the evidence is about the right entity but does not entail the claim; CC if the evidence contradicts the claim or the step targets the wrong relation\.

#### Disagreement resolution\.

Steps with disagreement were adjudicated by a third senior annotator\. Final labels were the post\-adjudication consensus\.

#### Per\-type agreement\.

Cohen’sκ\\kappaon the binaryhas\_gapdecision is 0\.704\. Per\-typeκ\\kappavalues are: IE 0\.71, CC 0\.68, MB 0\.43\. The lowerκ\\kappaon MB is driven by its low frequency \(n=9n\{=\}9true MB cases\) and inflated sampling variance, not by systematic disagreement\.

#### Annotation interface\.

Annotators saw the question, the full reasoning trace, the retrieved evidence per step, and the final answer\. They were asked to assign one of \{no\-gap, CC, IE, MB\} per step plus a brief free\-form justification\.

#### Licenses and artifact use\.

HotpotQA, 2WikiMultiHopQA, and MuSiQue are released under their original research licenses \(CC BY\-SA 4\.0, Apache 2\.0, and CC BY 4\.0 respectively\); we use them only for academic evaluation, consistent with the original intended use\. The Wikipedia 2018 dump used for retrieval follows the Wikipedia text license \(CC BY\-SA 3\.0\)\. E5\-base\-v2 and DeBERTa\-v3 are released under MIT; Qwen2\.5\-7B\-Instruct under the Tongyi Qianwen license; Llama\-3\.1\-8B under the Llama 3\.1 Community License\. We release the StepGap checker code, the distilled student checkpoint, and the 181\-step evaluation benchmark under CC BY 4\.0 for non\-commercial research use\.

## Appendix CKnowledge Distillation to Local Student

#### Setup\.

We distill the hybrid teacher \(GPT\-4\.1\-mini \+ DeBERTa\-v3\-NLI\) into a fully local Qwen2\.5\-7B student via supervised fine\-tuning on 1,000 teacher\-labeled steps \(4 datasets excluding the test benchmark\)\. Training uses LoRA \(r=16r=16\) on a single H100 GPU; total cost is 2 hours \+ $8 teacher API\.

#### Results\.

Table[6](https://arxiv.org/html/2605.24733#A3.T6)compares teacher, student, and an ablated student without NLI input features\.

Table 6:Distillation results\. F1 andκ\\kappavs\. the human\-annotated set\.We report two student variants\. The Qwen3\-8B v5 row exhibits near\-zeroκ\\kappa\(\.060\.060\), which we attribute to thinking\-mode KV\-cache behavior at our LoRA rank \(r=16r\{=\}16\) producing unstable Stage\-C outputs; we therefore adopt Qwen2\.5\-7B v4 as the primary student\.

The 7B student retains87%87\\%of teacher F1*and*removes teacher API calls at inference time, making the checker suitable for integration into long RL runs without per\-step API expense\. Cohen’sκ\\kappadrops from\.284\.284\(teacher\) to\.264\.264\(student\), a small absolute decline of0\.020\.02\. We note that both values are low in absolute terms; this reflects the inherent difficulty of typed step\-level annotation – already evident in the humanκ=0\.704\\kappa\{=\}0\.704onn=181n\{=\}181– rather than student degradation\. The retained signal is sufficient for downstream typed reward: substituting the student for the teacher in the GRPO loop yields\+1\.7\+1\.7EM over Search\-Only \(vs\.\+3\.3\+3\.3with the teacher, Table[4](https://arxiv.org/html/2605.24733#S7.T4)\) – about half the teacher’s gain at zero API cost, which we view as a favorable cost\-quality trade\-off for long RL runs\.

#### StepGap\-vs\-teacher agreement on the distillation pool\.

The1,0001\{,\}000\-step distillation pool also serves as a silver\-standard validation set for StepGap itself\. We measure agreement between StepGap \(the deployed system, GPT\-4\.1\-mini Stages A–C \+ DeBERTa\-v3 Stages D–E\) and a held\-out teacher run on the same1,0001\{,\}000steps, reporting binaryF1=0\.88\\mathrm\{F1\}\{=\}0\.88onhas\_gapand typedF1=0\.81\\mathrm\{F1\}\{=\}0\.81on CC/IE/MB\. The gap between the silver typed F1 \(0\.810\.81\) and student typed F1 \(\.626\.626, Table[6](https://arxiv.org/html/2605.24733#A3.T6)\) localizes student degradation to capacity rather than supervision noise: the teacher signal that the student is being trained on is itself internally consistent at scale\.

#### Student failure analysis\.

Manual inspection of 47 student false negatives reveals one dominant pattern:found\_quote→\\toinference→\\tono\-gapover\-triggering\. The student reportsfound\_quote=Trueat Stage C even when the “quote” is from the wrong entity or does not address the claim\. At this point the pipeline short\-circuits to no\-gap, bypassing the NLI entailment check that the teacher performs\. 23 of 47 FN cases \(49%\) match this pattern\. The bottleneck is not model capacity \(the 7B student matches teacher F1 on explicit\-gap cases\) but a*calibration bias*in the student’s Stage C judgment\. Hard\-negative augmentation on2626teacher\-confirmed FN cases raises student F1 by3\.73\.7points \(Appendix[P\.3](https://arxiv.org/html/2605.24733#A16.SS3)\)\.

## Appendix DHybrid Architecture Rationale

#### Why hybrid, not LLM\-only\.

Step\-level gap detection requires several distinct judgments: entity\-relation alignment, abstention validity, entity consistency, verbatim\-quote extraction, logical entailment, and cross\-step verification\. Conflating these into one LLM prompt invites*prompt context contamination*: framing introduced by an early node biases later judgments\. The Whitehorse case in Appendix[F](https://arxiv.org/html/2605.24733#A6)shows this in isolation – Stage A’s “airline vs\. terminal” framing pushes a shared\-context Stage D away from accepting an otherwise entailing quote\. The hybrid breaks this by making Stage D and Stage E pure NLI calls that receive only the\(premise, hypothesis\)pair\.

#### Hybrid ablation pattern\.

Table[7](https://arxiv.org/html/2605.24733#A4.T7)reports parallel component\-removal ablations on the LLM\-Only and StepGap checkers\. The two architectures show qualitatively different ablation profiles:*\(i\) LLM\-Only shows competing\-error cancellation*: three of four component removals*improve*F1 \(\+1\.6\+1\.6,\+0\.8\+0\.8,\+1\.7\+1\.7\), indicating internal components actively cancel each other’s errors\. Only one component \(Global Prior\) is a net positive contributor\.*\(ii\) StepGap is healthy*: every component removal hurts F1 \(−6\.6\-6\.6for Stage A,−2\.5\-2\.5for Stage E cross\-verification\), indicating each stage is a positive contributor to the final decision\. This compositional property is the architectural signal that motivates StepGap over LLM\-Only at equal top\-line F1\.

#### Why only Stages A and E\.

Stage B \(abstention validity\) fires on too few steps in our evaluation set \(n<10n\{<\}10\) to support a meaningful component\-wise ablation\. Stages C \(entity consistency\) and D \(local entailment\) form a tightly coupled LLM→\\toNLI pipeline: Stage D consumes Stage C’s verbatim quote as its NLI premise, so removing either alone collapses the decision tree to a different topology rather than yielding a clean ablation\. We therefore ablate the two stages with structurally independent input pathways—Stage A \(pre\-retrieval alignment\) and Stage E \(cross\-step verification\)—both of which show−F1\-\\mathrm\{F1\}, confirming the monotone pattern\.

Table 7:Component\-removal ablations on the 181\-step evaluation set\.LLM\-Only\(top\): three of four removals improve F1, a competing\-error cancellation pattern\.StepGap\(bottom\): every removal hurts F1, the healthy pattern that motivates the hybrid design\. Common\-Knowledge / Semantic\-Leap / Global Prior denote the LLM\-Only checker’s internal sub\-prompts for general\-knowledge inference, paraphrase tolerance, and prior probability of gap, respectively\.
#### Auditability\.

Each decision is unit\-testable\. NLI outputs are deterministic given model weights, and the label mapping \(entailment→\\tono\-gap,neutral→\\toMB,contradiction→\\toCC\) is closed\-form\. Debugging a misclassification reduces to isolating quote extraction \(Stage C, LLM\) from entailment \(Stage D, NLI\)\. An LLM\-only checker offers no such isolation\.

#### Distillability and reward grounding\.

The hybrid is a prerequisite for two downstream uses\. \(i\) Local deployment: the NLI sub\-system is already a 435M model, and Stage A–C produce structured JSON that a 7B student can imitate \(Appendix[C](https://arxiv.org/html/2605.24733#A3)\)\. \(ii\) Typed reward: the CC/IE/MB labels are grounded in the deterministic NLI output, which an LLM\-only checker cannot match because its categorical judgments shift across reruns and prompt perturbations\.

## Appendix EFailure Mode Taxonomy of the Full System

Inspection of 13 checker false negatives and 12 false positives reveals four recurrent failure modes\. We describe them using the typed taxonomy\.

#### FM1: Parametric shortcut \(FN source\)\.

Generator retrieves partially relevant evidence; NLI assigns entailment because a surface token overlap exists; generator fills in the remaining fact from parametric memory, producing a wrong answer\. No CC/IE/MB is flagged\.

#### FM2: Granularity mismatch \(FN source, 6/13\)\.

Generator retrieves a related but lower\-granularity fact \(year instead of birth date, district instead of municipality\); NLI sees the entity at some granularity and judges entailment\. The answer is wrong but the step passes\. This is the most common FN\.

#### FM3: Comparison logic \(FP source\)\.

Conclusion steps for comparison questions \(“who died later?”\) require combining evidence from multiple prior steps; NLI judges each step against its direct evidence pool in isolation, often producingneutraland spurious MB flags\.

#### FM4: Answer string normalization \(FP source\)\.

Generator produces a correct answer with different surface form from the gold string \(“La Vida Alrededor was released earlier” vs\. gold “La Vida Alrededor”\); the question counts as wrong under EM, but the checker correctly detects no evidence gap\. This is a limitation of EM\-based ground truth, not of the checker\.

## Appendix FCase Studies

We illustrate how StepGap handles each gap type with two representative examples drawn from our human\-annotated evaluation set\. Two further cases \(a student\-checker failure mode and a Single\-LLM\-XL failure mode\) are deferred to Appendix[G](https://arxiv.org/html/2605.24733#A7)\.

#### Case 1: NLI breaks prompt contamination \(Whitehorse, 2WikiMultiHopQA\)\.

*Question*: “The airline operating in Whitehorse, Yukon handled how many passengers in 2012?”*Retrieved evidence*: “The terminal handled294,000294\{,\}000passengers in 2012\.”*Gold*:294,000294\{,\}000\.

The two systems run on identical inputs but disagree at Stage D\. LLM\-Only’s Stage A introduced an “airline vs\. terminal” framing into the shared prompt, which biased its Stage D judgment away from accepting the quote\. StepGap’s NLI Stage D receives only the\(premise, hypothesis\)pair, with no upstream framing, and judges entailment correctly\. Air North is the sole commercial airline operating from the Whitehorse terminal, so the quote does support the claim\. This is the context\-contamination mechanism we discuss in Appendix[D](https://arxiv.org/html/2605.24733#A4), isolated to a single decision\.

#### Case 2: Missing\-Bridge detection \(Kuhn vs\. Pertramer, 2WikiMultiHopQA\)\.

*Question*: “Were Heinrich Gerhard Kuhn and Elfie Pertramer from the same country?”*Gold*: yes \(both German\)\.*Predicted answer*: no \(the model concluded “not from same country”\)\.

StepGap correctly identifies the conclusion step asMissing Bridge: the retrieved evidence supports the sub\-claim “Kuhn is British” but*not*the unstated bridging premise “Kuhn was born in Britain\.” The actual bridge is that “British physicist” refers to academic affiliation, not birthplace; Kuhn was born in Germany\. This is the kind of unstated\-premise failure that endpoint EM cannot catch \(the answer is wrong, but no claim is overtly contradicted\) and that a binary support\-checker would also miss \(the quote is supported, just incomplete\)\. The repair action prescribed by an MB tag \(“issue a bridging search”\) is exactly what the generator would need to do to recover\.

## Appendix GAdditional Case Studies

We supplement the two case studies in §[F](https://arxiv.org/html/2605.24733#A6)with two further examples that illustrate failure modes of alternative checkers, motivating specific design choices in StepGap\.

#### Case 3: Student over\-triggersfound\_quote\(Peter and Paul Fortress, MuSiQue\)\.

*Question*: “Where did the designer of Peter and Paul Fortress die?”*Gold*: Saint Petersburg\.*Evidence*: “Peter and Paul Fortress is the original citadel of St\. Petersburg, Russia, built to Domenico Trezzini’s designs\.”

The student’s failure is structural: it learns during distillation to map “found\_quote=True∧\\wedgeinference step” directly to no\-gap, bypassing the NLI entailment check the teacher performs\. This pattern accounts for 23 of 47 \(49%\) student false negatives \(§[C](https://arxiv.org/html/2605.24733#A3)\)\. The fix is not more capacity but*stronger Stage C calibration*: the student must learn that “found a quote” is necessary but not sufficient for no\-gap\. We address this via hard\-negative augmentation \(briefly described in Appendix[C](https://arxiv.org/html/2605.24733#A3)\)\.

#### Case 4: Single\-LLM\-XL flags everything \(HotpotQA\)\.

*Question*: “Which 2017 film was directed by the director of*Whiplash*?”*Gold*:*La La Land*\.*Evidence at step 1*: “*Whiplash*is a 2014 film directed by Damien Chazelle\.”

A single LLM, even a strong one, applies its quote search and entailment criteria too uniformly, treating any trace of paraphrasing or compositional inference as unsupported\. Across the 181\-step evaluation set this collapses into the 87\.8% IE rate reported in §[6\.2](https://arxiv.org/html/2605.24733#S6.SS2)—and produces the same 91\.4% Q\-F1 that we identify in §[6](https://arxiv.org/html/2605.24733#S6)as the Q\-F1 trap\. The decomposed pipeline in StepGap, by contrast, accepts paraphrastic matches at Stage C and only enforces strict entailment at Stage D, where the inputs are proper propositions\.

## Appendix HReward Hyperparameters

#### Base reward weights\.

The base reward magnitudes\{\+0\.20,−0\.05,−0\.10,\+0\.05\}\\\{\+0\.20,\-0\.05,\-0\.10,\+0\.05\\\}for\{\\\{no\-gap, MB, IE, CC\}\\\}were selected on a 200\-iteration pilot, optimizing answer\-claim support rate at fixed EM\. The relative ordering \(no\-gap≫\\ggCC, MB\>\>IE\) was held fixed; only the absolute scale was tuned\. For the CC detection bonus specifically, we tested\{\+0\.02,\+0\.05,\+0\.10\}\\\{\+0\.02,\+0\.05,\+0\.10\\\}at single seed:\+0\.02\+0\.02produced the same monotone CC\-suppression as the negative ablation \(Figure[3](https://arxiv.org/html/2605.24733#S7.F3)\),\+0\.10\+0\.10caused CC rate to plateau without the second\-phase decline, and\+0\.05\+0\.05exhibited the two\-phase rise\-then\-fall dynamics we report as the intended design behavior\.

#### Shaping weights\.

The shaping reward magnitudes\{\+0\.10,\+0\.15,−0\.15,−0\.05\}\\\{\+0\.10,\+0\.15,\-0\.15,\-0\.05\\\}were chosen so that the total per\-step reward magnitude does not exceed the EM reward magnitude \(which is 1\.0 binary\)\.

#### Near\-duplicate query threshold\.

The token\-F1 threshold0\.70\.7used inrshaper^\{\\text\{shape\}\}to detect near\-duplicate queries \(§[5\.2](https://arxiv.org/html/2605.24733#S5.SS2)\) was set on the same 200\-iteration pilot as the base reward weights: lower values incorrectly flagged paraphrastic reformulations as duplicates and dampened re\-search behavior, while higher values failed to flag near\-identical retries differing only in stop words or token order\. We leave a fine\-grained sweep of this threshold to future work\.

#### Lambda\.

The reward mixing coefficientλ\\lambdadefaults to 1\.0\. Sensitivity toλ∈\{0\.5,1\.0,2\.0\}\\lambda\\in\\\{0\.5,1\.0,2\.0\\\}shows EM varies within±0\.6\\pm 0\.6points; we reportλ=1\.0\\lambda=1\.0as the main configuration\.

#### Limitations\.

A learned reward weighting \(e\.g\., inverse\-propensity per gap type\) is left to future work\.

## Appendix IStep\-Category Distribution

Table[8](https://arxiv.org/html/2605.24733#A9.T8)reports the per\-variant distribution of assigned gap types over the 181\-step evaluation set\. A well\-calibrated checker should produce IE in the3838–41%41\\%range \(the rate observed on LLM\-Only and StepGap\); deviations from this band reflect systemic biases that sF1 alone may not penalize\.

Table 8:Step\-category distribution across checker variants on the 181\-step evaluation set\. Healthy IE band:3838–41%41\\%\.†\\daggermarks degenerate variants whose IE rate exceeds70%70\\%with collapsed CC and MB\.The two pathologies sF1 alone cannot make visible:*Single\-LLM\-XL*assigns zero steps to CC and zero to MB, reducing the three\-way taxonomy to a binary \(no\-gap vs\. IE\) decision – a stronger single LLM produces a checker that*cannot discriminate between gap types*\. This is precisely the property that disqualifies it from typed\-reward use: typed reward needs all three categories reliably emitted\.*NER\-Quote*replaces Stage C’s LLM quote search with spaCy NER plus token overlap; the result is74%74\\%IE because Stage D’s NLI call fires on noun\-phrase queries that are not well\-formed propositions, so the checker over\-reports retrieval failure\. The hybrid works because LLM and NLI handle*different factor types*, not because NLI is smaller or because an LLM is larger\.

## Appendix JAnswer\-Gap Correlation

Table[9](https://arxiv.org/html/2605.24733#A10.T9)cross\-tabulates final\-answer correctness with gap detection on the 82\-question subset\. Of 69 wrong\-answer questions,5656\(81%81\\%\) have at least one gap detected\. Of1313correct\-answer questions,1212still have a gap detected – the*lucky\-correct*regime: a substantial fraction of correct answers in multi\-hop QA are not fully grounded in retrieved evidence\.

Table 9:Answer correctness vs\. gap detection \(n=82n=82questions\)\.
## Appendix KFirst\-Gap\-Type Localization

For each wrong\-answer question we examine the first step the checker flags as a gap and record its type \(Table[10](https://arxiv.org/html/2605.24733#A11.T10)\)\. StepGap and LLM\-Only both identify IE \(retrieval failure\) as the first broken step in roughly80%80\\%of wrong\-answer chains, confirming that retrieval, not reasoning, is the dominant root cause of failure in our multi\-hop evaluation set\. StepGap differs from LLM\-Only in striking a more balanced first\-gap distribution between IE \(79%79\\%\), MB \(11%11\\%\), and CC \(11%11\\%\), giving downstream repair logic a richer typed signal\. Degenerate variants attribute essentially every first failure to IE, producing no usable signal for routing repair actions\.

Table 10:For each wrong\-answer question, type of the first step flagged by the checker as containing a gap\. Entries are percentages of wrong\-answer questions\. Degenerate variants trivially attribute nearly all first\-failures to IE\.
## Appendix LQ\-F1 Trap: Empirical Confirmation

We complement the closed\-form argument in Section[6](https://arxiv.org/html/2605.24733#S6)with two empirical checks \(Table[11](https://arxiv.org/html/2605.24733#A12.T11)\)\.*First*, on four public QA datasets at their natural wrong\-answer rateww\(estimated from a Qwen2\.5\-7B\-Instruct Search\-Only baseline\), a flag\-everything dummy checker reaches a Q\-F1 within±0\.02\\pm 0\.02of the analytic prediction2​w/\(1\+w\)2w/\(1\+w\)\.*Second*, on stratified resamples of our own evaluation set withwwcontrolled at0\.500\.50,0\.650\.65, and0\.840\.84, empirical Q\-F1 reproduces the analytic curve, confirming that the effect is driven bywwand the metric, not by the specific question content\.

Table 11:Empirical Q\-F1 of a flag\-everything checker on public benchmarks \(top\) and stratified resamples of our evaluation set \(bottom\)\. Analytic prediction:2​w/\(1\+w\)2w/\(1\+w\)\.
## Appendix MGRPO Auxiliary Grounding Metrics

Table[12](https://arxiv.org/html/2605.24733#A13.T12)reports the auxiliary grounding metrics that accompany the EM numbers in Table[4](https://arxiv.org/html/2605.24733#S7.T4): token\-F1, answer support rate \(fraction of answer claims NLI\-entailed by a retrieved quote\), and mean search calls per question\. Across reward configurations, support rate rises monotonically \(0%→38\.2%→56\.1%→71\.5%0\\%\\to 38\.2\\%\\to 56\.1\\%\\to 71\.5\\%\), and search count increases under typed shaping \(2\.1→2\.82\.1\\to 2\.8\), indicating that the policy learns to issue additional searches in response to detected gaps rather than answering through\.

Table 12:Auxiliary grounding metrics for each reward ablation on the 82\-Q subset \(aggregate over the three datasets\)\.*Supp\.*==fraction of answer claims NLI\-entailed by a retrieved quote;\|<s\>\|\|\\texttt\{<s\>\}\|==mean search calls per question\.
## Appendix NThreshold\-Sweep Diagnostic

To verify that the degenerate classification of Single\-LLM\-XL and NER\-Quote \(§[6\.2](https://arxiv.org/html/2605.24733#S6.SS2)\) is not threshold\-dependent, we sweep confidence thresholds for these variants and the StepGap reference\. Per\-variant confidence is the geometric mean of stage\-level confidences: for LLM stages, the softmax probability of the chosen output token; for NLI stages, the softmax probability of the chosen entailment label\.

![Refer to caption](https://arxiv.org/html/2605.24733v1/x4.png)Figure 4:Threshold sweep for StepGap \(healthy reference\) vs\. Single\-LLM\-XL and NER\-Quote \(degenerate variants\)\. Top\-right panel: green band marks the healthy IE range \(38–41%\) observed across LLM\-Only and StepGap; red dashed line marks the 70% degenerate floor\. Single\-LLM\-XL and NER\-Quote never enter the healthy band at any thresholdτ∈\[0\.3,0\.9\]\\tau\\in\[0\.3,0\.9\]\.
## Appendix OChecker Implementation Prototype

This appendix provides a minimal reference implementation of the StepGap hybrid checker\. The full production code handles batching, caching, retrieval\-server integration, and robust JSON parsing; here we present the core decision logic in stripped\-down form\. The five\-stage pipeline \(Figure[2](https://arxiv.org/html/2605.24733#S1.F2)\) is implemented as a linear sequence of early\-return checks\.

### O\.1Structured\-Output Schema

The LLM \(GPT\-4\.1\-mini in the teacher; Qwen2\.5\-7B in the student\) returns a single structured JSON object per step, containing the judgments for Stage A, Stage B, and Stage C in one pass\. Structured\-output mode is enforced viaresponse\_format=\{"type":"json\_schema",…\}withstrict=true, which guarantees parseable JSON\. The schema is:

```
{
  "step_minus1_alignment": {
    "is_off_target":       bool,
    "drift_type":  // one of:
      "none" | "entity_drift" |
      "relation_drift" | "scope_drift",
    "alignment_reasoning": str
  },
  "step_0_abstention_check": {
    "is_abstention_step":     bool,
    "abstention_is_accurate": bool,
    "abstention_reasoning":   str
  },
  "step_1_quote_search": {
    "entity_match":           bool,
    "entity_match_reasoning": str,
    "found_quote":            bool,
    "evidence_quote":         str,
    "quote_search_reasoning": str
  }
}
```

### O\.2LLM Prompt Templates

Stages A–C are invoked via a single prompt that returns the schema above\. The prompt has a fixed three\-section structure \(alignment / abstention / entity\-and\-quote\) and switches between an*inference*variant and a*conclusion*variant of Section A based on the step type\. We reproduce the core text below; whitespace and formatting follow the actual production prompt\.

#### Section A: alignment \(inference variant\)\.

```
STEP -1: SUBQUESTION ALIGNMENT
Does this step target the CORRECT entity and
relation?
CRITICAL: retrieval failure != off-target.
Only flag if step_text ITSELF names the wrong
entity.

EC quick test - is_off_target = FALSE if:
 - Entity in step_text appears verbatim in
   the question
 - Entity is the correct intermediate result
 - step_text names correct film/person but
   retrieval returned wrong doc

TRUE drift: Q asks "AUTHOR of book" -> step
  searches "DIRECTOR of film adaptation"
NOT drift: retrieval failed but query is
  correct

If is_off_target = TRUE -> has_gap = TRUE,
gap_type = "unsupported_claim", STOP.
```

#### Section A: alignment \(conclusion variant\)\.

```
STEP -1: CONCLUSION ALIGNMENT (mandatory)
RULE 0: answer contains N/A or unknown
  -> is_off_target = FALSE, bypass to Node 0.

Otherwise C-1 -> C-2 -> C-3:
  C-1: target attribute type
       (PLACE / BIRTH_YEAR / SPOUSE /
        AGENT / COUNTRY / YES_NO)
  C-2: actual type of answer value
  C-3: TYPE mismatch  = relation_drift.
       Wrong value but same type
                       = NOT drift.

DRIFT:    Q "Where born?" -> "Naresh Kumar"
          (person != place) -> TRUE
NO DRIFT: Q "Who did X?"  -> "Viktor Belash"
          (person  = person) -> FALSE
```

#### Section B: abstention\.

```
STEP 0: ABSTENTION CHECK
(skip if Step -1 triggered)
Is this step saying N/A / cannot determine?
  Evidence lacks it
    -> GROUNDED ABSTENTION
       is_abstention_step = True,
       abstention_is_accurate = True
  Evidence has it
    -> WRONG ABSTENTION
       is_abstention_step = True,
       abstention_is_accurate = False
```

For conclusion steps with prior inference evidence available, the prompt prepends aGLOBAL EVIDENCEblock listing the snippets retrieved at all earlier steps; Stage B uses this pool to decide whether an asserted abstention is justified\.

#### Section C: entity consistency \+ verbatim quote\.

```
STEP 1: ENTITY CONSISTENCY + QUOTE SEARCH
(skip if Step -1 or Step 0 triggered)

Part A - Entity consistency check:
  EC-1: What entity/topic does this step’s
        query target?
  EC-2: What entity/topic is the retrieved
        evidence actually about?
        (check the document title and first
         sentence)
  EC-3: Same entity? -> entity_match = TRUE,
                        proceed to quote search
        Different    -> entity_match = FALSE,
                        found_quote  = FALSE
                        (stop here)

  *Quote from the WRONG document does NOT
   count.*
   Step:     "Sruthilayalu composer"
   Evidence: "M.G. Sreekumar" (different
              person) -> FALSE

Part B - Verbatim quote search
        (only if entity_match = TRUE):
  Find a 5-20 word EXACT span from evidence.
  Must be: (i) exact text,
           (ii) about the same entity,
           (iii) relevant to the step claim.
```

The prompt closes with the question, the previous\-step summaries \(truncated to the last three\), the current step’s text, and the evidence pool serialized as JSON\.

#### NLI calls \(Stage D and Stage E\)\.

The NLI stages take no prompt: they pass a\(premise, hypothesis\)pair directly to a DeBERTa\-v3\-NLI cross\-encoderLaureret al\.\([2024](https://arxiv.org/html/2605.24733#bib.bib54)\)\. Stage D setspremise==the verbatim quote returned by Stage C andhypothesis==the step’s claim text; Stage E iterates over entity\-matched snippets from prior steps as the premise pool, with the same hypothesis\. We softmax the three\-class logits and take the argmax label \(entailment/neutral/contradiction\); ties default toneutral\.

### O\.3NLI Label Calibration

NLI models from thecross\-encoderfamily emit a three\-dimensional logit vector whose label order varies by checkpoint \(e\.g\.\[contradiction, entailment, neutral\]or\[entailment, neutral, contradiction\]\)\. At checker initialization we probe with a known entailment pair \(“A cat is an animal\.”, “A cat is an animal\.”\) and identify which output dimension scores highest; the label order is set accordingly\. All subsequent calls use the calibrated mapping\. Thresholds areENTAILMENT=0\.5andCONTRADICTION=0\.5on the softmaxed scores; labels between these thresholds are treated asneutral\.

### O\.4Step Type Classification

A step is classified asinferenceif its text contains a<search\>tag \(or, in Hermes\-style outputs, a<tool\_call\>JSON object with thesearchaction\), and asconclusionif it contains an<answer\>tag\. Ambiguous steps \(neither tag\) are treated asinferenceby default\.

### O\.5Pipeline Path Accounting

Every check returns apipeline\_pathstring encoding the exact sequence of stage decisions taken \(e\.g\.stageC:entity\_mismatch,stageD:neutral\)\. We use this for two purposes: \(i\) distributional analysis, confirming each stage is actually exercised in practice and not starved of input, and identifying which branches dominate the error modes; \(ii\) unit testing, where we assert that specific annotated cases follow the expected path\. The pipeline\-path signature also underlies our distillation loss: the student is trained to reproduce the teacher’s full path, not just the terminal label, which prevents the student from short\-circuiting via surface heuristics \(see Section[C](https://arxiv.org/html/2605.24733#A3)\)\.

## Appendix PAdditional Experimental Details

This appendix documents experimental protocols and reproducibility details that complement the main paper\.

### P\.1Standard\-Split Re\-evaluation Protocol

For comparability with the published baselines in Table[4](https://arxiv.org/html/2605.24733#S7.T4)\(top panel\), we follow the standard Search\-R1 evaluation pipeline on the standard dev splits \(sizes in Table[2](https://arxiv.org/html/2605.24733#S6.T2)\): Qwen2\.5\-7B\-Instruct generator with temperature0and max\-new\-tokens10241024; E5\-base\-v2 retrievalWanget al\.\([2022a](https://arxiv.org/html/2605.24733#bib.bib53)\)over Wikipedia 2018 with top\-k=5k\{=\}5; GRPO\-trained checkpoint at the best validation EM step, chosen on a held\-out500500\-Q dev pool disjoint from these splits\. Each cell reports strict EM averaged over the full split\. Our reward\-ablation rows in Table[4](https://arxiv.org/html/2605.24733#S7.T4)use the same pipeline\.

### P\.2GRPO Training Configuration

All GRPO runs hold the following hyperparameters constant: group sizeG=8G\{=\}8, KL coefficientβ=0\.05\\beta\{=\}0\.05, learning rate1×10−61\{\\times\}10^\{\-6\}for the policy with the checker frozen,3,0003\{,\}000training iterations with200200\-iterations evaluations, on the NQ\+\+HotpotQA training mix specified in Section[7](https://arxiv.org/html/2605.24733#S7)\. The*Untyped\-Dense*control reward referenced in Section[3](https://arxiv.org/html/2605.24733#S7.F3)usesrbaser^\{\\text\{base\}\}that maps any gap \(CC, IE, MB\) to−0\.10\-0\.10and no\-gap to\+0\.20\+0\.20, with no shaping term and no per\-type distinction; this isolates the contribution of*typing*from per\-step density\.

#### Computing infrastructure\.

All GRPO training is run on a single NVIDIA H100 \(80GB\) GPU\. A full 3,000\-iteration GRPO run takes approximately 18 H100 GPU\-hours for Qwen2\.5\-7B\-Instruct and 22 H100 GPU\-hours for Llama\-3\.1\-8B\-Instruct under our setup\. Teacher\-checker API usage \(GPT\-4\.1\-mini\) is approximately 72K calls per run at $22; distillation of the student checker costs $8 in teacher API plus 2 H100 GPU\-hours\. The 181\-step human evaluation set was processed once per checker variant, totalling under 0\.5 H100 GPU\-hours of additional inference\. Total compute across all reported configurations \(three seeds on Qwen, single seed on Llama, all reward ablations, distillation, and Appendix experiments\) is approximately350\-400H100 GPU\-hours plus\.

### P\.3Hard\-Negative Augmentation Protocol

The\+3\.7\+3\.7F1 gain reported for the student in Appendix[C](https://arxiv.org/html/2605.24733#A3)comes from augmenting the student’s training data with2626hard\-negative cases identified by manual inspection of teacher–student disagreement on the held\-out distillation pool\. A case is selected if the teacher’s NLI label and the student’s NLI label diverge*and*the teacher’s prediction matches the human label on the 181\-step evaluation set for an adjacent step from the same trace\. Augmented training uses the same LoRA configuration \(r=16r\{=\}16\) for200200additional iterations with a learning rate of5×10−75\{\\times\}10^\{\-7\}\(an order of magnitude smaller than initial distillation, so as not to overfit the small augmentation set\)\. We hold out1313of the2626hard\-negative examples as a within\-augmentation check to confirm the gain is not attributable to memorization\.