R2VC: Modular Fact-Checking with Retrieval, Verification, and Confidence Calibration

# R2VC: Modular Fact-Checking with Retrieval, Verification, and Confidence Calibration
Source: [https://arxiv.org/html/2609.11955](https://arxiv.org/html/2609.11955)
Dhruv Dixit Department of Electrical and Computer Engineering Stevens Institute of Technology Hoboken, New Jersey, United States &Paritosh Pandey11footnotemark:1 Department of Computer Science University of North Carolina Chapel Hill, North Carolina, United States

###### Abstract

Large language models are increasingly used for automated fact checking, but end\-to\-end prompting often entangles evidence retrieval, reasoning, and uncertainty estimation, making failures difficult to diagnose and confidence difficult to trust\. We present R2VC, a modular retrieve, reason, verify, calibrate architecture for evidence\-grounded fact checking with citations and abstention\. R2VC combines hybrid sparse\+dense retrieval over Wikipedia, a supervised fine\-tuned and DPO\-aligned generator that produces diverse structured verdict candidates, an external NLI cross\-encoder for evidence\-based candidate selection, and a lightweight sequence\-level calibrator for confidence estimation and selective abstention\. On FEVER, an 8B backbone with R2VC achieves 13\.74% higher accuracy than baseline\. Ablation studies show that verifier\-based candidate selection and confidence calibration are the largest contributors to performance\. Removing candidate selection drops FEVER accuracy to 76\.24%, while removing calibration nearly doubles the Brier score to 0\.161\. A manual analysis of 250 errors further shows that retrieval failures, especially wrong\-entity evidence, remain the dominant bottleneck\. Together, these results show that modular fact\-checking pipelines can substantially improve both predictive accuracy and confidence reliability in open\-domain verification\.

## 1Introduction

Large Language Models \(LLMs\) are increasingly used as end\-to\-end fact checkers, yet they remain brittle when evidence is incomplete, noisy, or adversarial\. A single forward pass muddles retrieval, reasoning, and calibration, making it difficult to guarantee both verdict correctness and citation faithfulness\. Our goal is to turn an LLM into an evidence\-grounded fact\-checking system that predicts a veracity label, returns minimally sufficient citations, and exposes a calibrated probability of correctness to enable selective prediction and abstention using Wikipedia based benchmarks\. Recent retrieval\-augmented generation \(RAG\) systems have shown that adding an explicit retrieval pipeline can substantially improve performance on knowledge\-intensive tasks, including fact checking, by grounding generation in external evidence\(Glasset al\.,[2022](https://arxiv.org/html/2609.11955#bib.bib19)\)\. Entailment \(NLI\) models are increasingly used as verifiers in fact\-checking and attribution\-style pipelines, but their performance can degrade under domain shift when moving from standard NLI benchmarks to naturally occurring claims paired with long, noisy evidence documents\(Kamoiet al\.,[2023](https://arxiv.org/html/2609.11955#bib.bib20)\)\.

Recent systems mitigate hallucination via retrieval\-augmented prompting, self\-revision, or consistency\-based aggregation, but these often couple evidence acquisition, verdict generation, and confidence estimation inside an LLM loop, limiting diagnosability and leaving abstention heuristics poorly calibrated\. In particular, retrieval mistakes such as wrong\-entity evidence can yield confident errors, and disagreement\-based signals do not directly enforce entailment or contradiction against cited passages\. We therefore propose R2VC, a modular retrieve, reason, verify, calibrate pipeline\.

Our design is motivated by the broader observation that modular verification pipelines remain attractive because they make system behavior more controllable and components easier to optimize in isolation\(Bestaet al\.,[2024](https://arxiv.org/html/2609.11955#bib.bib21)\)\. Recent interactive verification frameworks likewise emphasize that evidence acquisition and verification are major cost and latency drivers, making it useful to tune, replace, or budget these modules independently under different deployment constraints\(Xieet al\.,[2025](https://arxiv.org/html/2609.11955#bib.bib22)\)\. At the same time, tightly coupled search and reasoning loops must explicitly guard against retrieval\-induced errors, reinforcing the value of clean interfaces between retrieval, generation, and verification\(Xuet al\.,[2024](https://arxiv.org/html/2609.11955#bib.bib23)\)\. We evaluate R2VC on Wikipedia\-based fact verification, using VitaminC for supervised training and reporting zero\-shot and few\-shot \(3\-shot\) label accuracy on VitaminC and FEVER\. We compare against component ablations that remove SFT and DPO, verifier\-based selection, and calibration or abstention, alongside non\-LLM and heuristic baselines under a shared evaluation protocol\. We also report FEVER and VitaminC label accuracy across multiple LLMs in Table[1](https://arxiv.org/html/2609.11955#S3.T1)and analyze 250 annotated errors to characterize dominant failure modes\. Figure[A\.1](https://arxiv.org/html/2609.11955#A1.F1)summarizes the task interface and provides an illustrative example\.

## 2Related Work

##### Benchmarks and attribution\.

*FEVER*\(Thorneet al\.,[2018](https://arxiv.org/html/2609.11955#bib.bib2)\)introduced large\-scale claim verification with sentence\-level evidence, and*FEVEROUS*\(Alyet al\.,[2021](https://arxiv.org/html/2609.11955#bib.bib3)\)extends this setting to include tabular evidence\. WICE\(Kamoiet al\.,[2023](https://arxiv.org/html/2609.11955#bib.bib20)\)frames real\-world claim verification as document\-level entailment over Wikipedia claims and their cited sources, highlighting that retrieval and long\-context evidence selection remain key bottlenecks for entailment\-based verifiers\. Our primary training source is*VitaminC*, which provides contrastive claim–evidence pairs mined from Wikipedia revisions\(Schusteret al\.,[2021](https://arxiv.org/html/2609.11955#bib.bib1)\)\. We also connect to attribution work: AIS formalizes whether model statements are verifiable against identified passages and proposes automatic attribution metrics\(Rashkinet al\.,[2023](https://arxiv.org/html/2609.11955#bib.bib6)\)\.

##### Retrieval\-augmented fact\-checking\.

Retrieval\-augmented generation \(RAG\) conditions outputs on passages retrieved from corpora such as Wikipedia\(Lewiset al\.,[2020](https://arxiv.org/html/2609.11955#bib.bib4)\)\. Self\-RAG tightly interleaves retrieval, generation, and self\-critique to improve factuality and citation accuracy\(Asaiet al\.,[2024](https://arxiv.org/html/2609.11955#bib.bib5)\)\. RARR follows a “research and revise” paradigm that edits an initial response using retrieved evidence\(Gaoet al\.,[2023](https://arxiv.org/html/2609.11955#bib.bib8)\)\. Re2G \(Retrieve, Rerank, Generate\) extends the RAG paradigm with an explicit reranking stage before generation, enabling a multi\-stage pipeline that improves evidence selection for downstream generation\(Glasset al\.,[2022](https://arxiv.org/html/2609.11955#bib.bib19)\)\. Moreover, Re2G uses the reranker to merge candidate sets from retrievers with incomparable scoring functions \(e\.g\., BM25 and dense retrieval\), effectively supporting sparse\+dense ensembling under a unified ranking signal\.

##### Modular pipelines vs\. tightly\-coupled search loops\.

Interactive search paradigms interleave query planning with retrieval feedback and introduce explicit mechanisms \(e\.g\., confidence\-gated verification/completion\) to mitigate cases where retrieved evidence can mislead the model\(Xuet al\.,[2024](https://arxiv.org/html/2609.11955#bib.bib23)\)\. Agent\-style fact\-checking frameworks similarly couple retrieval and verification decisions to reduce unnecessary searches and improve efficiency\(Xieet al\.,[2025](https://arxiv.org/html/2609.11955#bib.bib22)\)\. Modular fact\-checking pipelines remain operationally attractive because they localize failures, support clean ablations, and enable incremental component upgrades without retraining the full system end\-to\-end\(Bestaet al\.,[2024](https://arxiv.org/html/2609.11955#bib.bib21)\)\.

##### Consistency, calibration, and alignment\.

Hallucination detection and confidence estimation are closely related in recent work\. Consistency\-based methods such as SelfCheckGPT\(Manakulet al\.,[2023](https://arxiv.org/html/2609.11955#bib.bib9)\)and ConFactCheck\(Guptaet al\.,[2025](https://arxiv.org/html/2609.11955#bib.bib10)\)use disagreement across stochastic or cross\-model generations as a signal of factual unreliability, while calibration work has shown that neural models are often miscalibrated and that post\-hoc fixes such as temperature scaling can remain fragile under distribution shift\(Guoet al\.,[2017](https://arxiv.org/html/2609.11955#bib.bib11); Desai and Durrett,[2020](https://arxiv.org/html/2609.11955#bib.bib12)\)\. Although LLMs can sometimes self\-report correctness, such estimates do not generalize reliably\(Kadavathet al\.,[2022](https://arxiv.org/html/2609.11955#bib.bib13)\)\. Our approach combines these lines of work by using multi\-sample candidate generation, but grounding each candidate in retrieved evidence and selecting among them with an external NLI verifier rather than self\-consistency alone\. To support abstention, we train a Sequence Likelihood Calibration \(SLC\) model over verifier and generation features, and for alignment we pair supervised fine\-tuning with Direct Preference Optimization \(DPO\)\(Rafailovet al\.,[2023](https://arxiv.org/html/2609.11955#bib.bib14)\)\. This also alings with work on aggregated prompting for improved reasoning\(Aroraet al\.,[2022](https://arxiv.org/html/2609.11955#bib.bib15)\)\.

![Refer to caption](https://arxiv.org/html/2609.11955v1/figs/architecture.png)Figure 1:R2VC pipeline: Retrieve→\\rightarrowReason→\\rightarrowVerify→\\rightarrowCalibrate\.

## 3Methodology

##### Overview\.

R2VC \(Figure[1](https://arxiv.org/html/2609.11955#S2.F1)\) decomposes fact checking into four stages: hybrid retrieval, candidate generation, external verification, and confidence calibration\. Given a claim, the system retrieves evidence from Wikipedia, generates multiple structured candidates, selects among them with an NLI\-based verifier, and then applies SLC to produce calibrated confidence and abstention\. Algorithmic pseudocode is provided in Appendix[C](https://arxiv.org/html/2609.11955#A3)\.

VitaminC exampleclaim:“The*Harry Potter and the Goblet of Fire*film was released in 2005\.”evidence:“…*Harry Potter and the Goblet of Fire*is a 2005 fantasy film directed by Mike Newell …”label:SUPPORTSFEVER exampleclaim:“The film*Inception*was directed by Christopher Nolan\.”evidence:“*Inception*is a 2010 science fiction action film written and directed by Christopher Nolan\.”label:SUPPORTS

Figure 2:Illustrative annotated examples from VitaminC and FEVER after standardization into the common claim, evidence, and label format used in our pipeline\.
### 3\.1Datasets

We work in the standard Wikipedia\-based fact\-verification setting, using VitaminC for supervised training, FEVER for evaluation, and a large English Wikipedia snapshot as the retrieval corpus\. VitaminC\(Schusteret al\.,[2021](https://arxiv.org/html/2609.11955#bib.bib1)\)is a contrastive benchmark built from Wikipedia revisions in which each example contains a claim, a single evidence sentence, and a label in\{SUPPORTS,REFUTES,NEI\}\\\{\\text\{SUPPORTS\},\\text\{REFUTES\},\\text\{NEI\}\\\}\. Many examples are paired with minimally different evidence sentences such that one supports the claim and another refutes it, making the dataset useful for testing sensitivity to subtle factual differences\. We use a public Hugging Face version and standardize it into the schema\{claim,evidence\_text,label\}\\\{\\texttt\{claim\},\\texttt\{evidence\\\_text\},\\texttt\{label\}\\\}, mapping related label variants into the shared ternary space and filtering examples with empty claims\. The resulting dataset contains317,672317\{,\}672training examples,54,01254\{,\}012validation examples, and47,92947\{,\}929test examples\. Figure[2](https://arxiv.org/html/2609.11955#S3.F2)shows an illustrative standardized example, while Figure[F\.1](https://arxiv.org/html/2609.11955#A6.F1)shows the joint distribution of claim and evidence lengths\.

FEVER\(Thorneet al\.,[2018](https://arxiv.org/html/2609.11955#bib.bib2)\)is a large\-scale Wikipedia\-based fact\-verification benchmark with 185,445 claims labeled asSUPPORTED,REFUTED, orNOT ENOUGH INFO, each paired with one or more evidence sentences\. We use FEVER as our principal evaluation benchmark and standardize it into the same ternary format as VitaminC by retainingNOT ENOUGH INFOexamples and collapsing each evidence set into a flatevidence\_textfield through heuristic concatenation\. This yields a common representation across training and evaluation data while preserving the task semantics\. An illustrative FEVER example is also shown in Figure[2](https://arxiv.org/html/2609.11955#S3.F2), and full results across multiple LLMs are reported in Table[1](https://arxiv.org/html/2609.11955#S3.T1)\. Beyond these labeled datasets, we build a large retrieval corpus from an English Wikipedia snapshot such as November 2023, storing each paragraph as a separate document with fields\(doc\_id,title,url,text\)\(\\texttt\{doc\\\_id\},\\texttt\{title\},\\texttt\{url\},\\texttt\{text\}\)and indexing it with both Lucene/BM25 and FAISS HNSW\. In practice, this produces several million paragraphs, roughly corresponding to the full English Wikipedia article set, all of which are available to the retriever at inference time\.

### 3\.2Hybrid retrieval

We use a Wikipedia evidence corpus𝒟\\mathcal\{D\}in which each retrieval unit is a paragraph paired with its page title and URL\. Letℛs\\mathcal\{R\}\_\{s\}andℛd\\mathcal\{R\}\_\{d\}denote the sparse and dense retrievers\. For a claimcc, they return scored listsEs=\{\(ej,rjs\)\}j=1NsE\_\{s\}=\\\{\(e\_\{j\},r^\{s\}\_\{j\}\)\\\}\_\{j=1\}^\{N\_\{s\}\}andEd=\{\(ej,rjd\)\}j=1NdE\_\{d\}=\\\{\(e\_\{j\},r^\{d\}\_\{j\}\)\\\}\_\{j=1\}^\{N\_\{d\}\}, whererjsr^\{s\}\_\{j\}andrjdr^\{d\}\_\{j\}are the sparse and dense retrieval scores for evidence itemeje\_\{j\}\.

We min–max normalize scores from each retriever independently:

r~jm=rjm−minℓ⁡rℓmmaxℓ⁡rℓm−minℓ⁡rℓm\+ϵ,m∈\{s,d\}\.\\tilde\{r\}^\{m\}\_\{j\}=\\frac\{r^\{m\}\_\{j\}\-\\min\_\{\\ell\}r^\{m\}\_\{\\ell\}\}\{\\max\_\{\\ell\}r^\{m\}\_\{\\ell\}\-\\min\_\{\\ell\}r^\{m\}\_\{\\ell\}\+\\epsilon\},\\qquad m\\in\\\{s,d\\\}\.For each evidence itemee, we compute the fused retrieval scorer​\(e∣c\)=α​r~s​\(e\)\+\(1−α\)​r~d​\(e\)r\(e\\mid c\)=\\alpha\\tilde\{r\}^\{s\}\(e\)\+\(1\-\\alpha\)\\tilde\{r\}^\{d\}\(e\), whereα∈\[0,1\]\\alpha\\in\[0,1\]is the interpolation weight b/w sparse and dense retrieval, and missing scores are treated as0\. We keep the top\-KKitems underr​\(e∣c\)r\(e\\mid c\)as the final evidence poolℰ​\(c\)\\mathcal\{E\}\(c\)\.

All later stages operate only on this fused evidence pool, which keeps inference efficient while preserving retrieval recall for downstream generation and verification\. In implementation, we fuse results by outer\-joining ondoc\_id, normalizing scores per retriever, and coalescing passage fields to ensure robust downstream prompting; pseudocode is given in Appendix[C](https://arxiv.org/html/2609.11955#A3), and detailed retriever models and hyperparameters are provided in Table[G\.1](https://arxiv.org/html/2609.11955#A7.T1)\.

### 3\.3Generator training: SFT and DPO

We do supervised finetuning on VitaminC and preference optimization via DPO on aLlama\-3\.1\-8B\-Nemotron\-Nano\. The training prompt template is provided in Appendix[B](https://arxiv.org/html/2609.11955#A2)\.

##### SFT stage\.

We build an SFT dataset by applying this template to all splits of the standardized VitaminC dataset, using theevidence\_textfield as the single evidence snippet\. We fine\-tune all model parameters without quantization, using bf16 full weights,flash\_attention\_2, maximum sequence length 8192, per\-device batch size 2, and gradient accumulation over 8 steps, giving an effective batch size of 16\. We use a learning rate of 2e\-4, cosine decay with warmup ratio 0\.03, weight decay 0\.1, thepaged\_adamw\_8bitoptimizer, gradient checkpointing, and train for 1 epoch\. The objective is next\-token likelihood over the concatenated prompt and response, with a masking collator that zeroes loss on user tokens and restricts learning to the assistant segment\.

##### Candidate diversity\.

For each claim, we first retrieve a fused pool of passagesE​\(c\)E\(c\)\. To encourage diverse candidates without increasing retrieval cost, we generatek=16k=16candidates by pairing random evidence subsets with a fixed bank of diverse decoding configurations\. For eachi∈\{1,…,k\}i\\in\\\{1,\\dots,k\\\}, we sampleSi⊆E​\(c\)S\_\{i\}\\subseteq E\(c\)with\|Si\|=min⁡\(16,\|E​\(c\)\|\)\|S\_\{i\}\|=\\min\(16,\|E\(c\)\|\)and decode\(y^i,r^i\)=𝒢​\(c,Si;θi\)\(\\hat\{y\}\_\{i\},\\hat\{r\}\_\{i\}\)=\\mathcal\{G\}\(c,S\_\{i\};\\theta\_\{i\}\), whereθi\\theta\_\{i\}denotes a decoding configuration such as temperature or top\-pp\. This produces candidates grounded in the same retrieved evidence pool but exposed to slightly different evidence mixtures and reasoning paths\.

##### DPO stage\.

To align the generator toward better citation usage and label correctness, we construct preference triples\(x,y\+,y−\)\(x,y^\{\+\},y^\{\-\}\)from VitaminC, wherexxis the fact\-checking prompt for a claim\-evidence pair,y\+y^\{\+\}is a templated chosen response with the correct True, False, or Uncertain label and a minimal evidence\-grounded rationale, andy−y^\{\-\}is a templated rejected response with either an incorrect label or clearly mismatched citations\. We optimize the SFT model with DPO usingβ=0\.2\\beta=0\.2and a learning rate of 4e\-6, encouraging the model to assign higher log\-probability toy\+y^\{\+\}than toy−y^\{\-\}\. This preserves the SFT\-induced response format while explicitly discouraging hallucinated or misaligned citations, yielding a final generatorGθG\_\{\\theta\}optimized for structured fact\-checking outputs conditioned on retrieved evidence\.

##### Candidate Generation\.

Given a claimccand fused evidenceℰ​\(c\)=\{e1,…,eK\}\\mathcal\{E\}\(c\)=\\\{e\_\{1\},\\dots,e\_\{K\}\\\}, we generatek=16k=16candidate answers using a fixed bank of diverse decoding configurations, each conditioned on a random subset of up to 16 evidence passages\. For each decode, we parse the raw text into a structured candidatezzconsisting of a labelz\.label∈\{True,False,Uncertain\}z\.\\texttt\{label\}\\in\\\{\\text\{True\},\\text\{False\},\\text\{Uncertain\}\\\}inferred from the leading token \(“True\.”, “False\.”, or other\), a rationalez\.rationalez\.\\texttt\{rationale\}given by the remaining sentence\(s\), citationsz\.citationsz\.\\texttt\{citations\}extracted from bracketed citation spans, and the evidence subsetz\.passagesz\.\\texttt\{passages\}seen by the generator\. This produces a candidate set𝒵​\(c\)=\{z1,…,zk\}\\mathcal\{Z\}\(c\)=\\\{z\_\{1\},\\dots,z\_\{k\}\\\}, which is then passed to the verifier for scoring\.

### 3\.4External verifier \(NLI \+ citation checks\)

To decouple generation from verification, we use a cross\-encoder NLI model \(cross\-encoder/nli\-deberta\-v3\-large\) as an external verifier\. For each candidatezz, the verifier evaluates every evidence passagep∈z\.passagesp\\in z\.\\texttt\{passages\}by scoring the pair \(claim, passage\) and producing probabilities\(pcontr,pneut,pent\)\(p\_\{\\text\{contr\}\},p\_\{\\text\{neut\}\},p\_\{\\text\{ent\}\}\)\. From these passage\-level scores, we derive the maximum entailment support and contradiction signals assent=maxp⁡pent​\(p\)s\_\{\\text\{ent\}\}=\\max\_\{p\}p\_\{\\text\{ent\}\}\(p\)andscontr=maxp⁡pcontr​\(p\)s\_\{\\text\{contr\}\}=\\max\_\{p\}p\_\{\\text\{contr\}\}\(p\)\. To capture support spread across multiple pieces of evidence, we sort passages by entailment score and average the top\-6 values to obtain a coverage termscovs\_\{\\text\{cov\}\}\. We also compute a citation\-validity scorescit∈\[0,1\]s\_\{\\text\{cit\}\}\\in\[0,1\]as the maximum heuristic similarity between the claim and each candidate passage, using string overlap and numeric matching\. Finally, using thresholdsTent=0\.60T\_\{\\text\{ent\}\}=0\.60,Tcontr=0\.60T\_\{\\text\{contr\}\}=0\.60, and a marginδ=0\.10\\delta=0\.10, we derive an NLI\-based labelyNLI∈\{True,False,Uncertain\}y\_\{\\text\{NLI\}\}\\in\\\{\\text\{True\},\\text\{False\},\\text\{Uncertain\}\\\}from\(sent,scontr\)\(s\_\{\\text\{ent\}\},s\_\{\\text\{contr\}\}\)\.

We then compare the generator labelygeny\_\{\\text\{gen\}\}withyNLIy\_\{\\text\{NLI\}\}to obtain a label\-consistency scorescons∈\{0,0\.5,1\}s\_\{\\text\{cons\}\}\\in\\\{0,0\.5,1\\\}, and measure citation alignment throughsalign∈\[0,1\]s\_\{\\text\{align\}\}\\in\[0,1\]by checking whether the passages cited by the candidate overlap with those receiving the highest entailment scores\. These signals are combined into a scalar verifier score,sver​\(z\)=0\.6​sent\+0\.2​scov−0\.3​max⁡\(0,scontr−sent\)\+0\.1​scons\+0\.1​salign−0\.1​\(1−scit\)s\_\{\\text\{ver\}\}\(z\)=0\.6\\,s\_\{\\text\{ent\}\}\+0\.2\\,s\_\{\\text\{cov\}\}\-0\.3\\,\\max\(0,s\_\{\\text\{contr\}\}\-s\_\{\\text\{ent\}\}\)\+0\.1\\,s\_\{\\text\{cons\}\}\+0\.1\\,s\_\{\\text\{align\}\}\-0\.1\\,\(1\-s\_\{\\text\{cit\}\}\), which rewards strong entailment, broader evidence coverage, and agreement between citations, NLI, and the generator’s label, while penalizing excess contradiction and weak citation quality\. The verifier also outputs a distribution over NLI labels,𝐩i=𝒱​\(c,y^i,Si\)=\(pient,pineu,picon\)\\mathbf\{p\}\_\{i\}=\\mathcal\{V\}\(c,\\hat\{y\}\_\{i\},S\_\{i\}\)=\\big\(p\_\{i\}^\{\\text\{ent\}\},p\_\{i\}^\{\\text\{neu\}\},p\_\{i\}^\{\\text\{con\}\}\\big\), withpient\+pineu\+picon=1p\_\{i\}^\{\\text\{ent\}\}\+p\_\{i\}^\{\\text\{neu\}\}\+p\_\{i\}^\{\\text\{con\}\}=1\. For each candidate, we retain bothsver​\(z\)s\_\{\\text\{ver\}\}\(z\)and its component signals, which are later used for calibration\.

### 3\.5Sequence Likelihood Calibration \(SLC\)

Whilesvers\_\{\\text\{ver\}\}is a useful internal score, it is not itself a calibrated probability of correctness\. We therefore train a shallow calibratorfϕf\_\{\\phi\}, implemented as logistic regression, on features extracted from a held\-out slice of 10,000 VitaminC examples\. These features include the mean log\-probability of generated tokens, the verifier scoresvers\_\{\\text\{ver\}\}, the agreement fraction across candidates in𝒵​\(c\)\\mathcal\{Z\}\(c\)that share the chosen label, the number of evidence passages used in the candidate, the rationale length in tokens, the label\-consistency scoresconss\_\{\\text\{cons\}\}, and the citation\-alignment scoresaligns\_\{\\text\{align\}\}\. For each training example, we simulate inference with up to 12 generated candidates, select the chosen candidatez​z^\{\\\*\}, assign a binary target indicating whether its predicted label matches the gold label, and fitfϕf\_\{\\phi\}to predict this target from the resulting feature vector\. We evaluate the calibrator using Brier score and expected calibration error \(ECE\), and store the best checkpoint for use at test time\.

Table 1:Comparison across models on FEVER and VitaminC\.†denotes Run on Local Machine \(RoLM\) in the same environment\. Calibration metrics \(Brier, ECE\) are reported only for RoLM runs; for cited baselines, these values were not reported in the original papers and are marked asn/r\.At inference, the calibrated confidence is computed asp=fϕ​\(features​\(z​,𝒵​\(c\)\)\)p=f\_\{\\phi\}\(\\text\{features\}\(z^\{\\\*\},\\mathcal\{Z\}\(c\)\)\)and we optionally override the predicted label toUncertainwheneverp<τcal=0\.60p<\\tau\_\{\\text\{cal\}\}=0\.60\. Together with the earlier verifier\-score thresholdτver=0\.55\\tau\_\{\\text\{ver\}\}=0\.55, this produces a two\-stage abstention mechanism that suppresses predictions when either verification strength or confidence is too low\. Concretely, we select the highest\-confidence candidatei⋆=arg⁡maxi⁡πii^\{\\star\}=\\arg\\max\_\{i\}\\pi\_\{i\}and predict

y^=\{y^i⋆,if​πi⋆≥τcal∧sver​\(i⋆\)≥τver,Uncertain,otherwise\.\\hat\{y\}=\\begin\{cases\}\\hat\{y\}\_\{i^\{\\star\}\},&\\text\{if \}\\pi\_\{i^\{\\star\}\}\\geq\\tau\_\{\\text\{cal\}\}\\ \\wedge\\ s\_\{\\text\{ver\}\}\(i^\{\\star\}\)\\geq\\tau\_\{\\text\{ver\}\},\\\\ \\textsc\{Uncertain\},&\\text\{otherwise\.\}\\end\{cases\}so only those passing both verifier and calibration are returned as final fact\-check decisions\.

This modular design supports clean ablations of retrieval, alignment, verification, and calibration, allowing component upgrades without retraining full pipeline\. We evaluate R2VC against several baselines and ablations, described in the following sections\.

## 4Results

### 4\.1Benchmark Performance on FEVER and VitaminC

Table[1](https://arxiv.org/html/2609.11955#S3.T1)reports accuracy, Macro\-F1, Brier score, and ECE on FEVER and VitaminC\. Entries marked with†were reproduced locally under a shared evaluation pipeline and prompt template, while the remaining numbers are taken from prior work\. Because Brier and ECE are available only for local runs, calibration comparisons are limited to those models\.

Within the 8B class,Llama\-3\.1\-Nemotron\-8B \+ R2VCis the strongest locally evaluated model on both benchmarks\. On FEVER, it reaches84\.71%84\.71\\%accuracy and85\.6885\.68Macro\-F1, compared with75\.14%75\.14\\%and74\.8774\.87forLlama\-3\.1\-8B, and74\.54%74\.54\\%and75\.4075\.40for the baseLlama\-3\.1\-Nemotron\-8B\. Calibration also improves, with Brier dropping from0\.1920\.192and0\.0950\.095to0\.0830\.083, and ECE from0\.03610\.0361and0\.01430\.0143to0\.01250\.0125\. The same trend appears for Qwen:Qwen3\-8B \+ R2VCreaches83\.44%83\.44\\%accuracy and83\.3683\.36Macro\-F1 on FEVER, far above the baseQwen3\-8Bat61\.53%61\.53\\%and60\.4260\.42, while improving calibration from0\.217/0\.04850\.217/0\.0485to0\.112/0\.01840\.112/0\.0184in Brier/ECE\.

R2VC also compares favorably to larger local baselines\. On FEVER,Qwen3\-8B \+ R2VCsubstantially outperformsQwen3\-14B, which attains63\.65%63\.65\\%accuracy,62\.5462\.54Macro\-F1,0\.1960\.196Brier, and0\.04380\.0438ECE\. This indicates that the gains come from the retrieval, verification, and calibration pipeline rather than simply scaling model size\. Across both Nemotron and Qwen backbones, lower Brier and ECE confirm that the improvements are not only in classification accuracy but also in how well confidence tracks empirical correctness\. Compared with larger open\-weight and proprietary systems,Llama\-3\.1\-Nemotron\-8B \+ R2VCremains competitive: on FEVER it trailsLlama2\-70B\(86\.4%86\.4\\%\) by only1\.691\.69points and OpenAI GPT\-4 \(89\.2%89\.2\\%\) by4\.494\.49points, while clearly outperformingLlama2\-13B\(76\.86%76\.86\\%\),Mistral\-7B\(69\.84%69\.84\\%\), andBLOOM\(71\.0%71\.0\\%\)\.

On VitaminC, the same pattern is even stronger\.Llama\-3\.1\-Nemotron\-8B \+ R2VCachieves99\.78%99\.78\\%accuracy,99\.8299\.82Macro\-F1,0\.0730\.073Brier, and0\.00770\.0077ECE, improving over the baseLlama\-3\.1\-Nemotron\-8Bat87\.81%87\.81\\%,87\.8487\.84,0\.0830\.083, and0\.00880\.0088, and overQwen3\-8Bat88\.76%88\.76\\%,88\.8488\.84,0\.1160\.116, and0\.03150\.0315\.Qwen3\-8B \+ R2VCalso performs strongly, reaching99\.21%99\.21\\%accuracy,99\.1599\.15Macro\-F1,0\.08120\.0812Brier, and0\.00950\.0095ECE\. Notably, the base Nemotron model is already better calibrated than the base Qwen model despite slightly lower raw accuracy, but the full R2VC pipeline improves both families further\.

These results show that the full R2VC architecture improves both predictive quality and confidence reliability; that too in a fixed budget \(8b params\)\. Lower Brier scores indicate more accurate probabilistic predictions, and lower ECE values indicate that predicted confidence is better matched to actual correctness, making the system more reliable for selective prediction and abstention\.

Table 2:Ablation study of the proposed framework on FEVER and VitaminC usingLlama\-3\.1\-Nemotron\-8B\. Higher is better for Accuracy and Macro\-F1, lower is better for Brier and ECE\.

## 5Ablation Study

Table[2](https://arxiv.org/html/2609.11955#S4.T2)shows that each major component of R2VC contributes to both predictive performance and calibration\. The full model performs best on all four metrics across both FEVER and VitaminC, reaching84\.71%84\.71\\%accuracy,85\.6885\.68Macro\-F1,0\.0830\.083Brier, and0\.01250\.0125ECE on FEVER, and99\.78%99\.78\\%accuracy,99\.8299\.82Macro\-F1,0\.0730\.073Brier, and0\.00770\.0077ECE on VitaminC\. Across ablations, the largest losses come from removing verifier\-based candidate selection, confidence calibration, or multi\-sample generation, indicating that these are the main drivers of the full system’s gains\.

Removing DPO causes a smaller but consistent degradation\. On FEVER, accuracy drops from84\.71%84\.71\\%to83\.02%83\.02\\%, Macro\-F1 from85\.6885\.68to83\.9783\.97, Brier worsens from0\.0830\.083to0\.0910\.091, and ECE from0\.01250\.0125to0\.01380\.0138\. On VitaminC, the same ablation reduces accuracy from99\.78%99\.78\\%to97\.78%97\.78\\%, Macro\-F1 from99\.8299\.82to97\.8297\.82, and calibration from0\.073/0\.00770\.073/0\.0077to0\.080/0\.00850\.080/0\.0085in Brier/ECE\. This suggests that DPO improves both label correctness and evidence\-label alignment, but its effect is smaller than that of selection and calibration\.

Removing verifier\-based candidate selection causes one of the largest drops in overall quality: FEVER falls to76\.24%76\.24\\%accuracy,77\.1177\.11Macro\-F1,0\.0930\.093Brier, and0\.01400\.0140ECE, while VitaminC drops to89\.80%89\.80\\%accuracy,89\.8489\.84Macro\-F1,0\.0820\.082Brier, and0\.00860\.0086ECE\. Single\-candidate generation also underperforms the full multi\-sample system, reaching only78\.78%78\.78\\%accuracy,79\.6879\.68Macro\-F1,0\.0880\.088Brier, and0\.01330\.0133ECE on FEVER, and92\.80%92\.80\\%accuracy,92\.8392\.83Macro\-F1,0\.0770\.077Brier, and0\.00820\.0082ECE on VitaminC\. Together, these results show that sampling multiple candidates and selecting among them with the verifier is critical for finding stronger evidence\-grounded outputs\.

Disabling confidence calibration has the clearest effect on reliability\. On FEVER, Brier nearly doubles from0\.0830\.083to0\.1610\.161and ECE rises from0\.01250\.0125to0\.03120\.0312, while accuracy and Macro\-F1 also drop to77\.48%77\.48\\%and77\.1677\.16\. On VitaminC, removing calibration reduces performance to96\.32%96\.32\\%accuracy,96\.5796\.57Macro\-F1, and worsens calibration from0\.073/0\.00770\.073/0\.0077to0\.085/0\.02270\.085/0\.0227\. These results show that the SLC module is not only improving confidence quality, but also helping final decision quality, especially on FEVER where confidence becomes much less aligned with correctness without calibration\.

The NLI\-only baseline is clearly weaker than the full pipeline, with74\.54%74\.54\\%accuracy and75\.4075\.40Macro\-F1 on FEVER and87\.81%87\.81\\%accuracy and87\.8487\.84Macro\-F1 on VitaminC, along with worse Brier and ECE on both datasets\. Relative to this baseline, the full model improves FEVER by\+10\.17\+10\.17accuracy points and\+10\.28\+10\.28Macro\-F1, while reducing Brier from0\.0950\.095to0\.0830\.083and ECE from0\.01430\.0143to0\.01250\.0125; on VitaminC, it improves accuracy by\+11\.97\+11\.97points and Macro\-F1 by\+11\.98\+11\.98, while reducing Brier from0\.0830\.083to0\.0730\.073and ECE from0\.00880\.0088to0\.00770\.0077\. These ablations show that the strongest gains come from verifier\-based candidate selection, multi\-sample generation, and calibration, while DPO provides a smaller but still consistent improvement\. More detailed variants are reported in Table[H\.1](https://arxiv.org/html/2609.11955#A8.T1)in Appendix[H](https://arxiv.org/html/2609.11955#A8)\.

![Refer to caption](https://arxiv.org/html/2609.11955v1/figs/error_primary_category.png)\(a\)Primary failure categories\.
![Refer to caption](https://arxiv.org/html/2609.11955v1/figs/error_primary_secondary_top20.png)\(b\)Top error subtypes\.
![Refer to caption](https://arxiv.org/html/2609.11955v1/figs/confidence_by_primary_error_category.png)\(c\)Confidence by primary error category\.
![Refer to caption](https://arxiv.org/html/2609.11955v1/figs/gold_to_pred_label_transitions.png)\(d\)Gold→\\rightarrowpredicted label transitions\.

Figure 3:Error analysis on the manually annotated error pool\. Top left: primary failure categories\. Top right: top primary/secondary subtypes, with retrieval errors dominated by wrong\-entity evidence and verifier errors often due to NLI label flips\. Bottom left: confidence by primary error category, showing overconfidence for retrieval failures and lower confidence for insufficient or ambiguous evidence\. Bottom right: gold→\\rightarrowpredicted label transitions\.
## 6Failure Modes and Confidence Diagnostics

##### Error sampling and annotation\.

We construct an error pool by stratified sampling across \(i\) calibrated confidence bins, \(ii\) verifier score bins, \(iii\) gold label, and \(iv\) dataset/source\. We manually annotated a stratified sample of 250 erroneous predictions using a two\-level taxonomy \(primary category \+ secondary subtype\), showcased in Appendix[E](https://arxiv.org/html/2609.11955#A5)with optional notes\. This procedure reduces the chance that the analysis over\-represents only low\-confidence or single\-dataset failures\.

Let𝒟=\{\(cn,yn\)\}n=1N\\mathcal\{D\}=\\\{\(c\_\{n\},y\_\{n\}\)\\\}\_\{n=1\}^\{N\}be a dataset andπn\\pi\_\{n\}the confidence for the system’s non\-abstained prediction\. In standard benchmark evaluation,UNCERTAINis scored as the third class, while in selective prediction analysis, only low\-confidence threshold\-triggered outputs are treated as abstentions\. For a thresholdτ\\tau, define the covered set𝒞​\(τ\)=\{n:πn≥τ\}\\mathcal\{C\}\(\\tau\)=\\\{n:\\pi\_\{n\}\\geq\\tau\\\}\. Coverage and risk are

Cov​\(τ\)=\|𝒞​\(τ\)\|N,Risk​\(τ\)=1−1\|𝒞​\(τ\)\|​∑n∈𝒞​\(τ\)𝕀​\[y^n=yn\]\.\\text\{Cov\}\(\\tau\)=\\frac\{\|\\mathcal\{C\}\(\\tau\)\|\}\{N\},\\qquad\\text\{Risk\}\(\\tau\)=1\-\\frac\{1\}\{\|\\mathcal\{C\}\(\\tau\)\|\}\\sum\_\{n\\in\\mathcal\{C\}\(\\tau\)\}\\mathbb\{I\}\[\\hat\{y\}\_\{n\}=y\_\{n\}\]\.\(1\)We summarize the trade\-off via the area under the risk–coverage curve \(AURC\)\.

Figures[3\(a\)](https://arxiv.org/html/2609.11955#S5.F3.sf1)–[3\(b\)](https://arxiv.org/html/2609.11955#S5.F3.sf2)report the category distribution and top subtypes\. Retrieval errors dominate \(Figure[3\(a\)](https://arxiv.org/html/2609.11955#S5.F3.sf1)\), indicating that evidence acquisition is the primary bottleneck relative to generation, verification, or citation selection\. Calibration/abstention errors are the next\-largest class, consistent with sensitivity to confidence mapping and decision thresholds\. Figure[3\(b\)](https://arxiv.org/html/2609.11955#S5.F3.sf2)shows that retrieval failures are primarily driven by*wrong\_entity\_page*, suggesting entity disambiguation as a key failure mode\. Verifier errors frequently involve*NLI label flips*, where the cross\-encoder misclassifies entailment/contradiction and induces incorrect candidate selection\. Generator and citation/attribution errors are less frequent but remain relevant for improving rationale faithfulness and citation precision\.

Figure[3\(c\)](https://arxiv.org/html/2609.11955#S5.F3.sf3)reports calibrated confidence distributions by primary category\. Retrieval failures are the most overconfident, consistent with strong internal evidence–claim alignment*conditional on the retrieved set*even when retrieval returns irrelevant or wrong\-entity passages\. Insufficient/ambiguous\-evidence errors occur at the lowest confidence, and many manifest as abstentions \(Figure[3\(d\)](https://arxiv.org/html/2609.11955#S5.F3.sf4)\)\. Verifier and calibration/abstention errors concentrate at higher confidence, consistent with confident but incorrect selection or thresholding\. Citation/attribution and generator\-given\-evidence errors occupy intermediate regimes\. Figure[3\(d\)](https://arxiv.org/html/2609.11955#S5.F3.sf4)summarizes gold\-to\-predicted transitions in the annotated error pool\. Most errors involve transitions intoUNCERTAIN, especially fromSUPPORTSorREFUTES, while directSUPPORTS↔REFUTES\\texttt\{SUPPORTS\}\\leftrightarrow\\texttt\{REFUTES\}polarity flips are less frequent\.

## 7Conclusion and Future Work

##### Conclusion\.

We presentedR2VC, a modular fact\-checking pipeline that equips an LLM with evidence\-grounded verification, citations, and calibrated abstention\. By combining hybrid Wikipedia retrieval, SFT\+DPO candidate generation, external NLI\-based selection, and sequence\-level calibration, R2VC achieves strong performance on VitaminC and FEVER while supporting reproducible local evaluation\. Our error analysis shows that retrieval remains the main bottleneck, especially when the system retrieves evidence for the wrong entity, leading to confident mistakes\. In contrast, insufficient or ambiguous evidence more often results in low confidence and abstention, suggesting that calibration is useful but cannot fully compensate for retrieval failures\. Overall, the results highlight retrieval quality, evidence selection, and verifier robustness as the main drivers of further improvement\.

##### Future Work\.

A natural next step is to evaluate the same pipeline across a broader range of model families, including reasoning\-oriented LLMs, mixture\-of\-experts architectures, and multimodal models that can use non\-textual evidence\. We also plan to test R2VC on more challenging fact\-checking benchmarks such as DEFAME, and FactLens, and to extend our manual error analysis using the richer taxonomy in Appendix[D](https://arxiv.org/html/2609.11955#A4)to better separate retrieval mismatches, generation and citation failures, verifier ranking errors, and calibration mistakes\. This expanded taxonomy can further support targeted training objectives, such as penalizing wrong\-entity retrieval, encouraging minimal sufficient citations, and rewarding calibrated abstention when evidence is insufficient, which could be incorporated into future preference optimization or reinforcement learning objectives\. By swapping fixed\-corpus retrieval module with web\-based dynamic evidence retriever, and test the architecture against oen\-domain benchmarks like FacTool\-QA, BingCheck, and FactCheck\-Bench\. For these benchmarks, an even more expanded error taxonomy will be required\.

## Acknowledgements

We thank Stevens Institute of Technology for providing GPU resources through the DuckUTE GPU Server, which were used to run all benchmark experiments reported in this paper\. We also thank the University of North Carolina at Chapel Hill for providing GPU resources through the Longleaf Cluster, which were used to run the ablation experiments\.

## Disclosure of LLM Use

The authors used an LLM\-based writing assistant to revise and improve portions of the manuscript prose based on text originally written by the authors\. The authors retained full responsibility for the scientific content of the paper, including the research ideas, technical approach, experiments, results, and conclusions\. All such revisions were carefully inspected by the authors for accuracy, and the manuscript was manually checked to ensure that no hallucinated, fabricated, or unsupported content was introduced\.

## References

- R\. Aly, Z\. Guo, M\. Schlichtkrull, J\. Thorne, A\. Vlachos, C\. Christodoulopoulos, O\. Cocarascu, and A\. Mittal \(2021\)Feverous: fact extraction and verification over unstructured and structured information\.arXiv preprint arXiv:2106\.05707\.Cited by:[§2](https://arxiv.org/html/2609.11955#S2.SS0.SSS0.Px1.p1.1)\.
- Anonymous \(2023\)Fact\-checking: generative LLMs don’t pay attention to details\.Note:OpenReview PDF \(ACL Rolling Review 2023 December Blind Submission\)External Links:[Link](https://openreview.net/pdf/815fcb232a8fd9b3951dc089ef25d6424d8b4f84.pdf)Cited by:[Table 1](https://arxiv.org/html/2609.11955#S3.T1.12.10.14.4.1),[Table 1](https://arxiv.org/html/2609.11955#S3.T1.12.10.15.5.1),[Table 1](https://arxiv.org/html/2609.11955#S3.T1.12.10.16.6.1)\.
- S\. Arora, A\. Narayan, M\. F\. Chen, L\. Orr, N\. Guha, K\. Bhatia, I\. Chami, F\. Sala, and C\. Ré \(2022\)Ask me anything: a simple strategy for prompting language models\.arXiv preprint arXiv:2210\.02441\.Cited by:[§2](https://arxiv.org/html/2609.11955#S2.SS0.SSS0.Px4.p1.1)\.
- A\. Asai, Z\. Wu, Y\. Wang, A\. Sil, and H\. Hajishirzi \(2024\)Self\-rag: learning to retrieve, generate, and critique through self\-reflection\.Cited by:[§2](https://arxiv.org/html/2609.11955#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Besta, N\. Blach, A\. Kubicek, R\. Gerstenberger, M\. Podstawski, L\. Gianinazzi, J\. Gajda, T\. Lehmann, H\. Niewiadomski, P\. Nyczyk,et al\.\(2024\)Graph of thoughts: solving elaborate problems with large language models\.InProceedings of the AAAI conference on artificial intelligence,Vol\.38,pp\. 17682–17690\.Cited by:[§1](https://arxiv.org/html/2609.11955#S1.p3.1),[§2](https://arxiv.org/html/2609.11955#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Desai and G\. Durrett \(2020\)Calibration of pre\-trained transformers\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 295–302\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.21/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.21)Cited by:[§2](https://arxiv.org/html/2609.11955#S2.SS0.SSS0.Px4.p1.1)\.
- L\. Gao, Z\. Dai, P\. Pasupat, A\. Chen, A\. T\. Chaganty, Y\. Fan, V\. Zhao, N\. Lao, H\. Lee, D\. Juan,et al\.\(2023\)Rarr: researching and revising what language models say, using language models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 16477–16508\.Cited by:[§2](https://arxiv.org/html/2609.11955#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Glass, G\. Rossiello, M\. F\. M\. Chowdhury, A\. Naik, P\. Cai, and A\. Gliozzo \(2022\)Re2G: retrieve, rerank, generate\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 2701–2715\.Cited by:[§1](https://arxiv.org/html/2609.11955#S1.p1.1),[§2](https://arxiv.org/html/2609.11955#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger \(2017\)On calibration of modern neural networks\.InInternational conference on machine learning,pp\. 1321–1330\.Cited by:[§2](https://arxiv.org/html/2609.11955#S2.SS0.SSS0.Px4.p1.1)\.
- R\. Gupta, P\. H\. Panicker, S\. Bhatia, and G\. Ramakrishnan \(2025\)Consistency is the key: detecting hallucinations in llm generated text by checking inconsistencies about key facts\.arXiv preprint arXiv:2511\.12236\.Cited by:[§2](https://arxiv.org/html/2609.11955#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DasSarma, E\. Tran\-Johnson,et al\.\(2022\)Language models \(mostly\) know what they know\.arXiv preprint arXiv:2207\.05221\.Cited by:[§2](https://arxiv.org/html/2609.11955#S2.SS0.SSS0.Px4.p1.1)\.
- R\. Kamoi, T\. Goyal, J\. D\. Rodriguez, and G\. Durrett \(2023\)Wice: real\-world entailment for claims in wikipedia\.arXiv preprint arXiv:2303\.01432\.Cited by:[§1](https://arxiv.org/html/2609.11955#S1.p1.1),[§2](https://arxiv.org/html/2609.11955#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.\(2020\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in neural information processing systems33,pp\. 9459–9474\.Cited by:[§2](https://arxiv.org/html/2609.11955#S2.SS0.SSS0.Px2.p1.1)\.
- P\. Manakul, A\. Liusie, and M\. Gales \(2023\)Selfcheckgpt: zero\-resource black\-box hallucination detection for generative large language models\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 9004–9017\.Cited by:[§2](https://arxiv.org/html/2609.11955#S2.SS0.SSS0.Px4.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.Advances in neural information processing systems36,pp\. 53728–53741\.Cited by:[§2](https://arxiv.org/html/2609.11955#S2.SS0.SSS0.Px4.p1.1)\.
- H\. Rashkin, V\. Nikolaev, M\. Lamm, L\. Aroyo, M\. Collins, D\. Das, S\. Petrov, G\. S\. Tomar, I\. Turc, and D\. Reitter \(2023\)Measuring attribution in natural language generation models\.Computational Linguistics49\(4\),pp\. 777–840\.External Links:[Link](https://aclanthology.org/2023.cl-4.2/),[Document](https://dx.doi.org/10.1162/coli%5Fa%5F00486)Cited by:[§2](https://arxiv.org/html/2609.11955#S2.SS0.SSS0.Px1.p1.1)\.
- T\. Schuster, A\. Fisch, and R\. Barzilay \(2021\)Get your vitamin c\! robust fact verification with contrastive evidence\.arXiv preprint arXiv:2103\.08541\.Cited by:[§2](https://arxiv.org/html/2609.11955#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2609.11955#S3.SS1.p1.5)\.
- C\. Thibault, J\. Tian, G\. Péloquin\-Skulski, T\. L\. Curtis, J\. Zhou, F\. Laflamme, L\. Y\. Guan, R\. Rabbany, J\. Godbout, and K\. Pelrine \(2025\)A guide to misinformation detection data and evaluation\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\. 2,pp\. 5801–5809\.Cited by:[Table 1](https://arxiv.org/html/2609.11955#S3.T1.12.10.17.7.1)\.
- J\. Thorne, A\. Vlachos, C\. Christodoulopoulos, and A\. Mittal \(2018\)FEVER: a large\-scale dataset for fact extraction and VERification\.InNAACL\-HLT,Cited by:[§2](https://arxiv.org/html/2609.11955#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2609.11955#S3.SS1.p2.1)\.
- R\. Wolfe, I\. Slaughter, B\. Han, B\. Wen, Y\. Yang, L\. Rosenblatt, B\. Herman, E\. Brown, Z\. Qu, N\. Weber,et al\.\(2024\)Laboratory\-scale ai: open\-weight models are competitive with chatgpt even in low\-resource settings\.InProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency,pp\. 1199–1210\.Cited by:[Table 1](https://arxiv.org/html/2609.11955#S3.T1.12.10.13.3.1)\.
- Z\. Xie, R\. Xing, Y\. Wang, J\. Geng, H\. Iqbal, D\. Sahnan, I\. Gurevych, and P\. Nakov \(2025\)FIRE: fact\-checking with iterative retrieval and verification\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 2901–2914\.Cited by:[§1](https://arxiv.org/html/2609.11955#S1.p3.1),[§2](https://arxiv.org/html/2609.11955#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Xu, L\. Pang, H\. Shen, X\. Cheng, and T\. Chua \(2024\)Search\-in\-the\-chain: interactively enhancing large language models with search for knowledge\-intensive tasks\.InProceedings of the ACM Web Conference 2024,pp\. 1362–1373\.Cited by:[§1](https://arxiv.org/html/2609.11955#S1.p3.1),[§2](https://arxiv.org/html/2609.11955#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix AFact Checking Illustrative Example

Task I/O\.Input:claimcc; evidence corpus𝒟\\mathcal\{D\}\(Wikipedia paragraphs\) indexed by a hybrid retriever \(at inference, onlyccis provided and retrieval over𝒟\\mathcal\{D\}is internal\)\.Output:\(y,E,r,p\)\(y,E,r,p\), wherey∈\{True,False,Uncertain\}y\\in\\\{\\text\{True\},\\text\{False\},\\text\{Uncertain\}\\\};E=\{e1,…,em\}E=\\\{e\_\{1\},\\dots,e\_\{m\}\\\}cited passages \(doc IDs \+ offsets\);rrevidence\-grounded rationale;p∈\[0,1\]p\\in\[0,1\]calibrated confidence for selective prediction/abstention\.Example\.Claim:“The Great Wall of China can be seen from the Moon with the naked eye\.”Output:y=Falsey=\\textbf\{False\};E=\{E=\\\{*Great Wall of China*\(Wikipedia\)\}\\\};rr: debunked myth; angular size below naked\-eye resolution at lunar distance \(with inline citations\);p=0\.87p=0\.87\(illustrative\)\.

Figure A\.1:Fact\-checking task interface and an illustrative example\.
## Appendix BTraining Prompt Template

Figure[B\.1](https://arxiv.org/html/2609.11955#A2.F1)shows the prompt–response template used to construct training examples for supervised fine\-tuning and preference optimization\. We includeUncertainas an allowed output to support abstention when retrieved evidence is missing or conflicting\.

Prompt<\|system\|\>You are a fact\-checking assistant\. Use evidence to decide\.<\|user\|\>Claim: \{claim\}Candidate evidence: \{evidence\_text\}Output True, False or Uncertain, then a one\-sentence justification with citations \[Title, §, line\]\.<\|assistant\|\>ResponseTrue\. Because …\[Title § line\]\.if label =SUPPORTSFalse\. Because …\[Title § line\]\.if label =REFUTESUncertain\. Because …\[Title § line\]\.if label =NEI

Figure B\.1:Prompt–response template used for generator training\.
## Appendix CInference and Retrieval Algorithms

This appendix provides pseudocode for the end\-to\-end R2VC inference procedure and the sparse\-dense retrieval fusion routine used to construct the final evidence pool\.

Algorithm C\.1Fact checking with fused retrieval, diverse generation, and entailment\-based verification1:claim

cc; sparse retriever

ℛs\\mathcal\{R\}\_\{s\}; dense retriever

ℛd\\mathcal\{R\}\_\{d\}; generator

𝒢\\mathcal\{G\}; entailment verifier

𝒱\\mathcal\{V\}; top\-

NsN\_\{s\}, top\-

NdN\_\{d\}, keep\-

MM; \#candidates

kk; sample size

ss\(e\.g\.,

≤6\\leq 6\); thresholds

τver,τcal\\tau\_\{\\text\{ver\}\},\\tau\_\{\\text\{cal\}\}; optional calibrator

g​\(⋅\)g\(\\cdot\)
2:predicted label

y^∈\{True,False,Uncertain\}\\hat\{y\}\\in\\\{\\textsc\{True\},\\textsc\{False\},\\textsc\{Uncertain\}\\\}and explanation

rr
3:

Es←ℛs​\(c,Ns\)E\_\{s\}\\leftarrow\\mathcal\{R\}\_\{s\}\(c,N\_\{s\}\)⊳\\trianglerightBM25 / sparse retrieval

4:

Ed←ℛd​\(c,Nd\)E\_\{d\}\\leftarrow\\mathcal\{R\}\_\{d\}\(c,N\_\{d\}\)⊳\\trianglerightdense retrieval

5:

E←Fuse​\(Es,Ed,M\)E\\leftarrow\\textsc\{Fuse\}\(E\_\{s\},E\_\{d\},M\)⊳\\trianglerightAlgorithm[C\.2](https://arxiv.org/html/2609.11955#alg2)

6:for

i←1i\\leftarrow 1to

kkdo

7:

Ei←Sample​\(E,s\)E\_\{i\}\\leftarrow\\textsc\{Sample\}\(E,s\)⊳\\trianglerightrandom subset for diversity

8:

θi←DecodingSchedule​\(i\)\\theta\_\{i\}\\leftarrow\\textsc\{DecodingSchedule\}\(i\)⊳\\trianglerighte\.g\., temperature/top\-ppsweep

9:

\(yi,ri\)←𝒢​\(c,Ei;θi\)\(y\_\{i\},r\_\{i\}\)\\leftarrow\\mathcal\{G\}\(c,E\_\{i\};\\theta\_\{i\}\)⊳\\trianglerightyi∈\{True,False,Uncertain\}y\_\{i\}\\in\\\{\\textsc\{True\},\\textsc\{False\},\\textsc\{Uncertain\}\\\}

10:

pi←𝒱​\(c,yi,Ei\)p\_\{i\}\\leftarrow\\mathcal\{V\}\(c,y\_\{i\},E\_\{i\}\)⊳\\trianglerightentail/neutral/contradict probs

11:

si←VerifierScore​\(pi\)s\_\{i\}\\leftarrow\\textsc\{VerifierScore\}\(p\_\{i\}\)⊳\\trianglerightpaper’ssvers\_\{\\text\{ver\}\}

12:

πi←\{g​\(si\)if calibrator is usedMonotoneMap​\(si\)otherwise\\pi\_\{i\}\\leftarrow\\begin\{cases\}g\(s\_\{i\}\)&\\text\{if calibrator is used\}\\\\ \\textsc\{MonotoneMap\}\(s\_\{i\}\)&\\text\{otherwise\}\\end\{cases\}
13:endfor

14:

i⋆←arg⁡maxi⁡πii^\{\\star\}\\leftarrow\\arg\\max\_\{i\}\\ \\pi\_\{i\}
15:if

si⋆<τvers\_\{i^\{\\star\}\}<\\tau\_\{\\text\{ver\}\}or

πi⋆<τcal\\pi\_\{i^\{\\star\}\}<\\tau\_\{\\text\{cal\}\}then

16:return\(Uncertain,\(\\textsc\{Uncertain\},“Insufficient / conflicting evidence\.”\)

17:else

18:return

\(yi⋆,ri⋆\)\(y\_\{i^\{\\star\}\},r\_\{i^\{\\star\}\}\)
19:endif

Algorithm C\.2Fusion of sparse and dense retrieval results1:sparse results

EsE\_\{s\}; dense results

EdE\_\{d\}; keep\-

MM
2:fused ranked evidence set

EE
3:

E←Es∪EdE\\leftarrow E\_\{s\}\\cup E\_\{d\}⊳\\trianglerightouter\-join / union on document id

4:Normalize sparse scores inEsE\_\{s\}to\[0,1\]\[0,1\]via min–max; normalize dense scores inEdE\_\{d\}to\[0,1\]\[0,1\]\.

5:foreach passage

e∈Ee\\in Edo

6:

ss​\(e\)←s\_\{s\}\(e\)\\leftarrownormalized sparse score if present else

0
7:

sd​\(e\)←s\_\{d\}\(e\)\\leftarrownormalized dense score if present else

0
8:

s​\(e\)←α⋅ss​\(e\)\+\(1−α\)⋅sd​\(e\)s\(e\)\\leftarrow\\alpha\\cdot s\_\{s\}\(e\)\+\(1\-\\alpha\)\\cdot s\_\{d\}\(e\)⊳\\trianglerightor any fixed fusion rule

9:CoalesceTextFields\(e\)\(e\)⊳\\trianglerightensure valid title/text for prompting

10:endfor

11:Sort

EEby

s​\(e\)s\(e\)descending and keep top

MM
12:return

EE

## Appendix DProposed Expanded Error Taxonomy

Table D\.1:Proposed expanded taxonomy for finer\-grained manual error analysis and subtype\-specific training signals\.Primary labelSecondary labelMeaningretrieval\_failurewrong\_entity\_pageRetrieved different entity with the same/similar name\.lexical\_missBM25 misses paraphrase or rare wording\.semantic\_driftDense retrieval returns topical but non\-answer passages\.multi\_hop\_neededRequires chaining; top\-KKmisses the needed hop\.timeliness\_mismatchEvidence is outdated or claim is time\-sensitive\.evidence\_buriedRelevant evidence is in top\-KKbut too low\-ranked/unused\.insufficient\_or\_ambiguous\_evidencenei\_likeClaim is not verifiable from the corpus; should abstain\.underspecified\_claimMissing referent or timeframe\.conflicting\_sourcesRetrieved evidence disagrees across sources\.requires\_world\_knowledgeRequires knowledge beyond the provided evidence\.requires\_definitionTerm unclear; needs definition/normalization\.generator\_error\_given\_evidencenegation\_scopeMisread negation/scope/qualifiers \(*not*,*only*,*never*, etc\.\)\.numeric\_date\_compareWrong numeric/date reasoning or comparison\.entity\_attribute\_swapCorrect entities retrieved but attributes swapped/misattributed\.coreference\_errorPronoun/mention refers to the wrong entity\.overgeneralizationConclusion goes beyond what evidence supports\.logical\_compositionErrors with AND/OR, quantifiers, conditionals, “at least/most”, etc\.hallucinated\_bridgeInvented intermediate fact not present in evidence\.citation\_or\_attribution\_errorirrelevant\_citationCited passage does not support the claim/rationale\.wrong\_spanRight document but wrong part/lines cited\.missing\_citationKey statement lacks supporting citation\.overcitationMany citations but none directly entail the decision\.misquoted\_paraphraseParaphrase distorts evidence meaning\.verifier\_selection\_errorlexical\_overlap\_biasSelects candidate via overlap rather than entailment\.nli\_label\_flipVerifier misclassifies entailment vs\. contradiction\.ranking\_bugCorrect candidate exists but scorer prefers the wrong one\.evidence\_subset\_issueCorrect candidate uses better evidence subset but is not selected\.calibration\_or\_abstention\_policy\_erroroverconfident\_wrongHigh confidence despite wrong label/citations\.underconfident\_correctLow confidence \(or abstains\) despite being correct\.threshold\_too\_lowAbstention threshold allows too many risky decisions\.threshold\_too\_highAbstention threshold rejects too many correct decisions\.uncertain\_misuseChooses SUPPORTS/REFUTES when it should abstain, or vice versa\.
## Appendix EError Labels Used in This Work

Table E\.1:Compact error taxonomy used for manual annotation in this work \(primary category with coarse subtypes\)\.
## Appendix FAdditional Figures

### F\.1Dataset Distribution Plots

![Refer to caption](https://arxiv.org/html/2609.11955v1/figs/VitaminC_claim_vs_evidence_length.png)Figure F\.1:Claim vs\. evidence length \(in words\) for a random subsample of VitaminC examples\.
### F\.2Training Dynamics Plots

SFT training is stable and convergent: Figure[2\(a\)](https://arxiv.org/html/2609.11955#A6.F2.sf1)shows a sharp early loss reduction \(roughly two orders of magnitude from an𝒪​\(1\)\\mathcal\{O\}\(1\)initialization into the10−210^\{\-2\}range\) followed by a smooth, steady decline with no signs of divergence or late\-stage overfitting, while Figure[2\(b\)](https://arxiv.org/html/2609.11955#A6.F2.sf2)confirms this behavior under a warmup–cosine schedule that ramps to≈2×10−4\\approx 2\\times 10^\{\-4\}by∼103\\sim 10^\{3\}steps and then decays smoothly toward zero over the remaining∼24\\sim 24k steps\.

#### F\.2\.1SFT training dynamics

![Refer to caption](https://arxiv.org/html/2609.11955v1/figs/sft_loss.png)\(a\)SFT training loss vs\. step\.
![Refer to caption](https://arxiv.org/html/2609.11955v1/figs/sft_lr_schedule.png)\(b\)SFT learning\-rate schedule\.

Figure F\.2:SFT training dynamics\.
#### F\.2\.2DPO training dynamics

DPO converges rapidly in a short run: Figure[3\(a\)](https://arxiv.org/html/2609.11955#A6.F3.sf1)drops from≈0\.16\\approx 0\.16at step 20 to near\-zero by step 40 and stays flat, with the final epoch\-level summary still small at step 125 \(around2\.5×10−22\.5\\times 10^\{\-2\}\), indicating strong separation between chosen and rejected responses\. This is reflected in reward behavior: Figure[4\(a\)](https://arxiv.org/html/2609.11955#A6.F4.sf1)shows chosen rewards increasing from roughly\+1\.5\+1\.5to about\+4\.0\+4\.0while rejected rewards decrease from roughly−1\.5\-1\.5to below−6\.5\-6\.5, yielding a widening margin \(Figure[4\(b\)](https://arxiv.org/html/2609.11955#A6.F4.sf2)\) from∼3\\sim 3to\>10\>10and near\-perfect preference satisfaction as reward accuracy rises from≈0\.95\\approx 0\.95to1\.01\.0by step 40 and remains at1\.01\.0\(Figure[5\(a\)](https://arxiv.org/html/2609.11955#A6.F5.sf1)\)\. Optimization signals are consistent with a brief alignment “polish” rather than destabilizing updates: Figure[3\(b\)](https://arxiv.org/html/2609.11955#A6.F3.sf2)shows gradient norms collapsing from roughly1\.1×10−11\.1\\times 10^\{\-1\}to𝒪​\(10−3\)\\mathcal\{O\}\(10^\{\-3\}\), while Figure[5\(b\)](https://arxiv.org/html/2609.11955#A6.F5.sf2)shows the chosen–rejected log\-probability gap increasing from about88to\>45\>45nats, all under a low learning rate that begins around4×10−64\\times 10^\{\-6\}at step 20 and linearly anneals toward zero over∼120\\sim 120steps\.

![Refer to caption](https://arxiv.org/html/2609.11955v1/figs/dpo_loss.png)\(a\)DPO training loss vs\. step\.
![Refer to caption](https://arxiv.org/html/2609.11955v1/figs/dpo_grad_norm.png)\(b\)Gradient norm over training\.

Figure F\.3:Optimization dynamics during DPO\.![Refer to caption](https://arxiv.org/html/2609.11955v1/figs/dpo_rewards_chosen_rejected.png)\(a\)Rewards for chosen vs\. rejected responses\.
![Refer to caption](https://arxiv.org/html/2609.11955v1/figs/dpo_reward_margin.png)\(b\)Reward margin over training\.

Figure F\.4:Preference signal during DPO\.![Refer to caption](https://arxiv.org/html/2609.11955v1/figs/dpo_reward_accuracy.png)\(a\)Reward accuracy over training\.
![Refer to caption](https://arxiv.org/html/2609.11955v1/figs/dpo_logprob_margin.png)\(b\)Log\-probability margin over training\.

Figure F\.5:Additional DPO training signals\.

## Appendix GHyper\-parameters and environment settings

Table G\.1:Hyperparameter configuration for FEVER run ofLlama\-3\.1\-Nemotron\-8Bon Nvidia H100\.ModuleSettingValueRetrievalWikipedia corpus capNo article capRetrievalParagraph cap per articleNo paragraph capRetrievalDense retrieverBAAI/bge\-large\-en\-v1\.5RetrievalBM25 top\-k200RetrievalDense top\-k200RetrievalFused passages kept24RetrievalPassages sampled per candidate16RetrievalFAISS HNSWMM64RetrievalFAISSefConstruction512RetrievalFAISSefSearch512RetrievalDense embedding chunk size8192RetrievalDense encoding batch size1024Generator / SFTFinetuning modeFull\-parameter finetuningGenerator / SFTQuantizationNoneGenerator / SFTWeights / compute dtypebf16 full weightsGenerator / SFTAttention implementationflash\_attention\_2Generator / SFTMax sequence length8192Generator / SFTPer\-device train batch size2Generator / SFTGradient accumulation8Generator / SFTEffective SFT batch size16Generator / SFTLearning rate2e\-4Generator / SFTWeight decay0\.1Generator / SFTOptimizerpaged\_adamw\_8bitGenerator / SFTGradient checkpointingonGenerator / SFTEpochs1InferenceRejection\-sampling candidateskk16InferenceDecoding configurations16 temperature and top\-p settingsInferenceMax new tokens160VerifierNLI cross\-encodercross\-encoder/nli\-deberta\-v3\-largeVerifierTop entailment passages used in scoring6VerifierVerifier threshold0\.55Verifier / SLCCalibration threshold0\.60Verifier / SLCSLC calibration set size10,000Verifier / SLCCandidates per claim during SLC fit12Verifier / SLCLogistic\-regression max iterations2000Verifier / SLCSLC used at inferenceonDPOPreference\-pair capFull VitaminC train splitDPOPer\-device train batch size2DPOGradient accumulation8DPOEffective DPO batch size16DPOMax target length512DPOMax sequence length8192DPOLearning rate4e\-6DPOWeight decay0\.1DPOOptimizerpaged\_adamw\_8bitDPODPOβ\\beta0\.2DPOEpochs1RuntimeAvg prompt tokens608\.3RuntimeAvg generated tokens64RuntimeGeneration tokens/sec50\.92RuntimeRetrieval latency per claim \(s\)0\.0316RuntimeGeneration latency per candidate \(s\)1\.2567RuntimeVerifier latency per candidate \(s\)0\.0251RuntimeMeasured fullkk\-sample latency per claim \(s\)20\.8248RuntimeMeasured fullkk\-sample claims/sec0\.0480
## Appendix HComplete Ablation Study

Table H\.1:Incremental and removal ablations of R2VC framework on FEVER and VitaminC\. Higher is better for Accuracy and Macro\-F1, lower is better for Brier and ECE\.
## Appendix IRebuttal Stage\-1

Table I\.1:Controlled comparison under the evaluation protocol usingLlama\-3\.1\-Nemotron\-8B\. All retrieval\-augmented rows use the same hybrid retriever and generator checkpoint\. Only the downstream selection and calibration mechanisms differ\.Table I\.2:Retrieval ablations usingLlama\-3\.1\-Nemotron\-8B\. Evidence recall is computed against annotated gold evidence where available\.Table I\.3:Accuracy\-latency trade\-off as the candidate budget changes usingLlama\-3\.1\-Nemotron\-8B\.Table I\.4:Backbone scaling analysis on FEVER\. Claim\-only LM receives no external evidence\. Single\-candidate RAG uses the shared hybrid retriever and one generated verdict\. Full R2VC adds multi\-candidate generation, NLI\-based selection, and SLC calibration\.Table I\.5:Generalization across fact\-checking benchmarks usingLlama\-3\.1\-Nemotron\-8B\. Open\-corpus retrieval evaluates the full pipeline\. Oracle\-document evaluation bypasses retrieval and isolates downstream reasoning and verification\. Evidence metrics follow the native annotation format of each benchmark\.Table I\.6:Qualitative positioning relative to modular and agentic fact\-checking systems\. Entries should be verified against the corresponding papers before posting\.Table I\.7:Effect of the fused retrieval budget on FEVER\. TheK=24K=24row is the measured Stage\-1 configuration; red entries are run\-time sanity\-check targets only\. These are placeholder values\.Table I\.8:Sensitivity to verifier and calibration thresholds on FEVER\. Coverage and selective risk are reported as percentages\. Red entries are run\-time sanity\-check targets only\. These are placeholder values\.Table I\.9:Ablation of the hand\-designed verifier score on FEVER\. Red entries are run\-time sanity\-check targets only\. These are placeholder values\.Table I\.10:Candidate\-budget and decoding\-strategy comparison on FEVER\. Existing diverse\-bank rows are measured; the red row is a run\-time sanity\-check target only\. These are placeholder values\.Table I\.11:Stability across random seeds on FEVER\. Values are reported as mean±\\pmstandard deviation\. Red entries are run\-time sanity\-check targets only\. These are placeholder values\.Table I\.12:Citation behavior on WiCE\. Metrics are computed from final returned citations or selected evidence sentences, not the full retrieval pool\. Red entries are run\-time sanity\-check targets only\. These are placeholder values\.