Decomposed Entailment for Factuality Checking and Hallucination Detection

# Decomposed Entailment for Factuality Checking and Hallucination Detection
Source: [https://arxiv.org/html/2608.05823](https://arxiv.org/html/2608.05823)
Achir Oukelmoun achir\.oukelmoun@gmail\.com&Nasredine Semmar CEA LIST NANO INNOV 2 Bd Thomas Gobert, 91120 Palaiseau, France nasredine\.semmar@cea\.fr&Gaël De Chalendar CEA LIST NANO INNOV 2 Bd Thomas Gobert, 91120 Palaiseau, France gael\.de\-chalendar@cea\.fr

###### Abstract

The reliability of Large Language Models \(LLMs\) is often compromised by factual inconsistencies, including hallucinations—cases where generated content is not supported by the underlying source\. We present HallDetect, a lightweight, reference\-free, and black\-box framework for hallucination detection that we evaluate not only on summarization but across a broader range of source\-grounded generation settings\. HallDetect builds on decomposition\-based factuality evaluation: generated content is decomposed into atomic claims, each verified by a compact encoder\-based entailment model through a contrastive formulation over a multi\-scale library of source chunks, and aggregated with an asymmetric score in which a single confidently contradicted claim flags the response\. Under a controlled protocol in which all methods share the same 4\-bit quantized backbones and consumer\-grade hardware budget, HallDetect outperforms comparably resourced generative and embedding\-based baselines on three of four benchmarks while remaining stable across backbone families, and yields a claim\-to\-span audit trail that localizes each error\.

Decomposed Entailment for Factuality Checking and Hallucination Detection

Achir Oukelmounachir\.oukelmoun@gmail\.comNasredine SemmarCEA LIST NANO INNOV2 Bd Thomas Gobert,91120 Palaiseau, Francenasredine\.semmar@cea\.frGaël De ChalendarCEA LIST NANO INNOV2 Bd Thomas Gobert,91120 Palaiseau, Francegael\.de\-chalendar@cea\.fr

## 1Introduction

The rapid advancement of Large Language Models \(LLMs\), driven in particular by the GPT familyAchiamet al\.\([2023](https://arxiv.org/html/2608.05823#bib.bib104)\), has fundamentally reshaped the landscape of Natural Language Processing\. As these models move from experimental settings to deployment in professional and decision\-critical contexts, a persistent limitation remains unresolved:hallucinations\. The tendency of LLMs to produce confident yet factually unsupported statements continues to hinder their reliable and safe useLiet al\.\([2024](https://arxiv.org/html/2608.05823#bib.bib109)\), and metrics based on surface overlap or embedding similarity fail to capture the logical errors exhibited by modern generative modelsMaynezet al\.\([2020](https://arxiv.org/html/2608.05823#bib.bib23)\)\.

Hallucination detection is also an operational challenge\. Many high\-performing LLMs are accessible exclusively through proprietary, black\-box APIs, precluding internal inspection\. At the same time, the “LLM\-as\-a\-judge” evaluation paradigm raises concerns regarding computational cost, latency, and reproducibility, particularly where local deployment or hardware constraints applySzymanskiet al\.\([2025](https://arxiv.org/html/2608.05823#bib.bib11)\); Dorneret al\.\([2024](https://arxiv.org/html/2608.05823#bib.bib12)\)\. Beyond a single scalar score, an evaluation system intended for auditing must identify*which*parts of an output are unsupported by the source and whyMiller \([2019](https://arxiv.org/html/2608.05823#bib.bib2)\); Fabbriet al\.\([2022](https://arxiv.org/html/2608.05823#bib.bib7)\)\.

In this work, we present HallDetect, a resource\-efficient and domain\-agnostic framework for hallucination detection\. HallDetect follows the established paradigm of decomposition\-based factuality evaluationMinet al\.\([2023](https://arxiv.org/html/2608.05823#bib.bib178)\); Chenet al\.\([2023](https://arxiv.org/html/2608.05823#bib.bib179)\): generated texts are decomposed into atomic propositions, which are then evaluated through contrastive entailment against the available evidence\. Our focus is on making this paradigm effective and stable under strict computational budgets—4\-bit quantized extractors and a sub\-1GB discriminative verifier on a single consumer GPU—and on characterizing its behavior across tasks, domains, and backbone families\. To that end, we evaluate on four benchmarks spanning news summarization \(QAGS\-CNN/DM,Wanget al\.,[2020](https://arxiv.org/html/2608.05823#bib.bib66)\), dialogue \(TofuEval,Tanget al\.,[2024b](https://arxiv.org/html/2608.05823#bib.bib78)\), adversarial perturbations \(FalseSum,Utamaet al\.,[2022](https://arxiv.org/html/2608.05823#bib.bib1)\), and biomedical QA \(PubMedQA,Jinet al\.,[2019](https://arxiv.org/html/2608.05823#bib.bib10)\)\.

Our main contributions are summarized as follows:

- •A frugal decomposition\-and\-verification pipeline\.Building on prior decomposition\-based factuality work \(Section[2\.2](https://arxiv.org/html/2608.05823#S2.SS2)\), HallDetect couples atomic claim extraction with a*contrastive, multi\-scale*NLI formulation: the best entailing chunk and the best contradicting chunk of the source are selected independently across five chunking granularities, and per\-claim scores are aggregated asymmetrically so that one confident contradiction flags the response\. The framework requires only the source\-generation pair and is compatible with arbitrary black\-box generators\.
- •Controlled evaluation under fixed computational budgets\.Under strict parameter and resource parity—all LLM components quantized to 4 bits and run on identical consumer hardware—HallDetect consistently outperforms comparably resourced generative, self\-consistency, and embedding baselines on three of four benchmarks\. We do not claim superiority over full\-precision or API\-based state\-of\-the\-art detectors, which operate under substantially larger budgets\.
- •Stability across backbones and domains\.Across three backbone families \(Mistral, Gemma, Llama\) and four datasets, HallDetect exhibits markedly lower variance than generative CoT judges, suggesting that delegating verification to a discriminative encoder shields the judgment from quantization artifacts\.
- •Claim\-level audit trail\.Like other decomposition\-based methods, HallDetect exposes a claim→\\rightarrowevidence\-span mapping; we illustrate with a worked long\-source example \(Appendix[B](https://arxiv.org/html/2608.05823#A2)\) how this trail localizes both contradictions and unsupported additions, and note that such localized verdicts are a natural fit for closed\-loop regeneration\.

## 2Related Work

Hallucinations—outputs inconsistent with or unsupported by a given source—impede the deployment of LLMs in high\-stakes settingsLiet al\.\([2024](https://arxiv.org/html/2608.05823#bib.bib109)\); Huanget al\.\([2025](https://arxiv.org/html/2608.05823#bib.bib24)\)\. A common distinction separatesfactuality\(consistency with world knowledge\) fromfaithfulness\(alignment with a specific input source\)Banerjeeet al\.\([2025](https://arxiv.org/html/2608.05823#bib.bib9)\); we target the latter, in summarization, question answering, and retrieval\-augmented generationTanget al\.\([2024b](https://arxiv.org/html/2608.05823#bib.bib78)\)\. Within this setting,intrinsichallucinations contradict the source directly, whileextrinsicones add unsupported content that may appear plausible in isolationWanget al\.\([2020](https://arxiv.org/html/2608.05823#bib.bib66)\); Maynezet al\.\([2020](https://arxiv.org/html/2608.05823#bib.bib23)\)\.

### 2\.1From Internal Signals to External Audits

White\-boxapproaches leverage internal signals such as token probabilities or hidden statesAzaria and Mitchell \([2023](https://arxiv.org/html/2608.05823#bib.bib59)\), but are impractical for proprietary APIs, motivating black\-box alternatives such as SelfCheckGPTManakulet al\.\([2023](https://arxiv.org/html/2608.05823#bib.bib67)\)that use outputs and stochastic sampling alone\. Surface\-level metrics such as ROUGELin \([2004](https://arxiv.org/html/2608.05823#bib.bib19)\)correlate weakly with factual correctnessMaynezet al\.\([2020](https://arxiv.org/html/2608.05823#bib.bib23)\), driving reference\-free strategies such as QAGSWanget al\.\([2020](https://arxiv.org/html/2608.05823#bib.bib66)\)that verify consistency through QA or NLI against the source\.

### 2\.2Decomposition\-Based Factuality Evaluation

HallDetect verifies generated text at the granularity of individual claims, a strategy shared by a growing body of work\. Our head\-to\-head baselines are deliberately restricted to*black\-box detectors of intrinsic, source\-grounded hallucinations that run under a matched frugal budget*\(Section[5](https://arxiv.org/html/2608.05823#S5)\); the related systems below fall outside this scope along one of three axes—*task*,*verifier type*, or*availability as a runnable detector*—and we position against each accordingly rather than benchmark it\.

*Extrinsic, knowledge\-grounded verification\.*FActScoreMinet al\.\([2023](https://arxiv.org/html/2608.05823#bib.bib178)\)decomposes generations into atomic facts and scores the fraction supported by an*external*knowledge source \(e\.g\., Wikipedia\), using a retrieval\-augmented*generative*LM as the validator rather than a discriminative entailment model\. It therefore measures*extrinsic*factual precision against world knowledge—not*intrinsic*faithfulness to a supplied document—with an LM\-based validator outside our frugal\-discriminative regime, making it orthogonal to our benchmark rather than a competitor on it\.

*Decomposition resources and analyses\.*PropSegmEntChenet al\.\([2023](https://arxiv.org/html/2608.05823#bib.bib179)\)is a corpus and task for proposition\-level segmentation and entailment recognition, whileWanneret al\.\([2024](https://arxiv.org/html/2608.05823#bib.bib180)\)and Molecular FactsGunjal and Durrett \([2024](https://arxiv.org/html/2608.05823#bib.bib181)\)analyze how decomposition granularity and decontextualization affect downstream verification\. These inform*how*we extract atomic claims \(Phase I\) but expose no source\-grounded faithfulness score to benchmark against on our datasets\.

*Generative atomic\-unit judges\.*ACUEvalWanet al\.\([2024](https://arxiv.org/html/2608.05823#bib.bib182)\)does target summarization faithfulness through atomic content units, but delegates each unit’s verification to an*LLM*judge \(and adds a correction stage\)\. It thus belongs to the generative “LLM\-as\-a\-judge” family, whose behavior under our quantized budget is already captured by ourLLM CoTbaseline\.

*Discriminative intrinsic detectors \(closest relatives\)\.*SummaCLabanet al\.\([2022](https://arxiv.org/html/2608.05823#bib.bib183)\)applies sentence\-level NLI over a source–summary pair matrix, but at a*fixed*sentence granularity and without generative atomic decomposition or a multi\-scale chunk library; ourNLI Onlybaseline is a document\-level analog of this family\. QAFactEvalFabbriet al\.\([2022](https://arxiv.org/html/2608.05823#bib.bib7)\)is a heavier multi\-stage QA\-plus\-entailment pipeline rather than a single\-pass NLI verifier\. MiniCheckTanget al\.\([2024a](https://arxiv.org/html/2608.05823#bib.bib184)\)is the most directly comparable system—a compact, trained fact\-checker with a DeBERTa variant aimed at exactly our low\-resource, source\-grounded setting—and we treat a controlled comparison against it as the priority next experiment \(Section[9](https://arxiv.org/html/2608.05823#S9)\)\.

Against this backdrop, HallDetect’s contribution is a specific, previously uncombined design for intrinsic detection under frugal budgets: \(i\)*generative*atomic\-claim extraction paired with a*specialized discriminative*NLI verifier used in a single forward pass per claim–chunk pair; \(ii\) a*multi\-scale*context library with*independent*best\-entailment / best\-contradiction selection; and \(iii\) a*contrastive*entailment\-minus\-contradiction score under an asymmetric, auditing\-oriented aggregation\. Unlike FActScore and ACUEval, verification never invokes a generative model; unlike SummaC, it verifies generatively\-extracted atomic propositions across multiple granularities rather than fixed source/summary sentence pairs\.

### 2\.3The Shift Toward Discriminative Verification

The “LLM\-as\-a\-judge” paradigmZhenget al\.\([2023](https://arxiv.org/html/2608.05823#bib.bib6)\); Liuet al\.\([2023](https://arxiv.org/html/2608.05823#bib.bib5)\); Eliavet al\.\([2025](https://arxiv.org/html/2608.05823#bib.bib30)\)prompts large generative models to assess other outputs, but risks self\-preference biasWataokaet al\.\([2024](https://arxiv.org/html/2608.05823#bib.bib8)\), evaluator hallucinations, and limited reproducibility at high cost\. These concerns have renewed interest in discriminative, encoder\-based verification, which frames factuality as a constrained classification problem with stronger logical structure and higher efficiency\. Benchmarks such as TofuEvalTanget al\.\([2024b](https://arxiv.org/html/2608.05823#bib.bib78)\)and HaluEvalHuet al\.\([2024](https://arxiv.org/html/2608.05823#bib.bib29)\)further stress the value of claim\-level judgments for human\-in\-the\-loop workflows\.

## 3Benchmarks and Multi\-Model Evaluation Strategy

### 3\.1Task Definition and Metrics

We addressbinary, response\-level hallucination detection: given a source documentDDand a generated responseRR, the system must decide whetherRRisfaithfultoDDorhallucinated\. All benchmarks are used with response\-level gold labels, and we report Precision, Recall, and F1 for thehallucinatedclass at a fixed decision threshold of0\.50\.5, reflecting out\-of\-the\-box utility without dataset\-specific tuning\. HallDetect additionally produces claim\-level verdicts; because the benchmarks above do not provide aligned gold labels at our claim granularity, we use these verdicts as an*audit trail*for qualitative analysis \(Appendix[B](https://arxiv.org/html/2608.05823#A2)\) rather than for quantitative localized\-error evaluation, which we leave to future work\.

### 3\.2Core Evaluation Benchmarks

To evaluateHallDetect, we adopt a multi\-model strategy using 4\-bit quantized variants ofLlama 3\.1\-8B,Gemma 2\-9B, andMistral\-7B, across four benchmarks representing distinct challenges:

- •QAGS\-CNN/DMWanget al\.\([2020](https://arxiv.org/html/2608.05823#bib.bib66)\)\(Primary Anchor\): high\-quality human faithfulness annotations for news summarization, our main testbed for atomic decomposition\.
- •TofuEvalTanget al\.\([2024b](https://arxiv.org/html/2608.05823#bib.bib78)\)\(Dialogue Robustness\): informal structure and speaker shifts test coreference stability and factual drift in dialogue summarization\.
- •FalseSumUtamaet al\.\([2022](https://arxiv.org/html/2608.05823#bib.bib1)\)\(Adversarial Stress Test\): fluent but factually perturbed summaries test sensitivity to subtle hallucinations that bypass coarse similarity checks\.
- •PubMedQAJinet al\.\([2019](https://arxiv.org/html/2608.05823#bib.bib10)\)\(Domain Generalization\): biomedical text requiring technical nomenclature handling and strict logical grounding\.

Compared to alternatives, SummEvalFabbriet al\.\([2021](https://arxiv.org/html/2608.05823#bib.bib63)\)conflates factuality with stylistic properties, FactCCKryścińskiet al\.\([2020](https://arxiv.org/html/2608.05823#bib.bib13)\)relies on synthetic perturbations that do not fully mirror organic LLM hallucinations, HaluEvalLiet al\.\([2023](https://arxiv.org/html/2608.05823#bib.bib14)\)lacks the long\-range document dependencies of QAGS, and FEVERThorneet al\.\([2018](https://arxiv.org/html/2608.05823#bib.bib16)\)targetsextrinsicfact\-checking against knowledge bases, whereas our framework evaluatesintrinsic consistencystrictly within the provided source\.

QAGS\-CNN/DM remains our primary reference point: it is reference\-free \(matching deployment, where gold summaries are unavailable\) and imposes long\-form complexity \(700–800\-word articles\) that exposes failure modes such as lead bias\. The additional datasets probe the framework’s task\- and domain\-agnostic behavior\.

### 3\.3Backbones and the Rationale for Quantized Evaluation

We evaluate across three architectures in quantized GGUF form \(Q4\_K\_M\):Llama\-3\.1\-8B\-Instruct\(dense Transformer with extended context\),Gemma\-2\-9B\-IT\(efficiency\-oriented attention design\), andNous\-Hermes\-2\-Mistral\-7B\-DPO\(DPO\-tuned for instruction following\)\. While newer model families exist, these backbones are widely deployed, run on consumer hardware, and—critically for our controlled protocol—are available in identical quantization formats across families; our claims concern the*framework’s*behavior under matched budgets rather than the absolute capability of any backbone\.

Restricting evaluation to 4\-bit quantization is deliberate: a framework stable under the added noise and capacity constraints demonstrates robustness not apparent at full precision, and all experiments remain reproducible on consumer\-grade hardware\.

## 4Methodology: The HallDetect Framework

Input:\{D,R\}\\\{D,R\\\}Claim Extractor\(on ResponseRR\)Context Splitter\(on DocumentDD\)Atomic Claims\{c1,…,cn\}\\\{c\_\{1\},\\dots,c\_\{n\}\\\}ContextLibrary𝒦\\mathcal\{K\}DeBERTa\-v3 NLI EngineFinalF​E​Ds​c​o​r​eFED\_\{score\}ExtractRRProcessDDFigure 1:The HallDetect architecture\. The pipeline bifurcates the input: the responseRRis decomposed into atomic claims, while the documentDDis indexed into a context library for NLI\-based verification\.HallDetectreconciles*diagnostic precision*,*auditability*, and*computational frugality*\. Rather than the proprietary “judge” models used by many systemsZhenget al\.\([2023](https://arxiv.org/html/2608.05823#bib.bib6)\); Liuet al\.\([2023](https://arxiv.org/html/2608.05823#bib.bib5)\), with their cost, transparency, and reproducibility trade\-offsChenet al\.\([2024](https://arxiv.org/html/2608.05823#bib.bib4)\), it adopts a local pipeline that formulates detection as logical verification grounded in Natural Language Inference \(NLI\)\.

### 4\.1Decompositional Verification

Assessing a response as a single, undifferentiated unit dilutes the error signal: isolated hallucinations are obscured by surrounding correct content, a known limitation of document\-level metricsFalkeet al\.\([2019](https://arxiv.org/html/2608.05823#bib.bib28)\)\. HallDetect therefore separates the*generative*extraction of atomic claims from the*discriminative*verification of each claim against the source, so that verification operates on focused, coherent hypotheses and gains sensitivity to localized contradictions and unsupported assertions\.

### 4\.2Phase I: Atomic Claim Extraction

Given a generated responseRR, the first stage decomposes it into a set of atomic claims\{c1,…,cn\}\\\{c\_\{1\},\\dots,c\_\{n\}\\\}, using locally deployed LLMs from the Llama 3\.1, Gemma 2, or Mistral families, all quantized to 4\-bit precision \(Q4\_K\_M\)\. Generative models provide the linguistic competence needed to surface implicit factual statements, while quantization keeps extraction deployable on consumer hardware\. The extraction prompt is*copy\-faithful*\(Appendix[A](https://arxiv.org/html/2608.05823#A1)\): claims are reproduced verbatim, so verification targets the model’s actual assertions rather than silently corrected paraphrases\.

### 4\.3Phase II: Multi\-Scale Contextual Chunking

Relevant evidence in long documents may be distributed across distant regions or diluted at a single context scaleLiuet al\.\([2024](https://arxiv.org/html/2608.05823#bib.bib15)\)\. HallDetect constructs a hierarchical context library𝒦\\mathcal\{K\}by partitioning the source documentDDat multiple granularities \(m∈\{1,2,4,8,16\}m\\in\\\{1,2,4,8,16\\\}chunks\)\. This multi\-scale representation increases the likelihood that, for any claim, some context window renders the relevant evidence both present and salient\.

The granularities are chosen*dyadically*\(mmdoubling from11to1616\) so that context sizes are spaced exponentially rather than linearly\. Two considerations motivate this\. First, a purely linear schedule \(m=1,2,3,…m=1,2,3,\\dots\) would spend most of its passes at coarse scales that differ only marginally, whereas the failure modes we care about—diffuse, multi\-sentence support versus sharply localized single\-sentence contradictions—live at scales that are*orders of magnitude*apart in span length\. A dyadic schedule places roughly equal representational effort at each order of magnitude of context length\. Second, the schedule is cost\-bounded: across all five granularities the library contains at most1\+2\+4\+8\+16=311\{\+\}2\{\+\}4\{\+\}8\{\+\}16=31chunks per document \(Appendix[A](https://arxiv.org/html/2608.05823#A1)\), so the entire multi\-scale view costs at most3131NLI passes per claim regardless of document length\. This keeps the verifier’s cost proportional to the \(small\) number of claims rather than to the document, while still ensuring that whole\-document, section\-, window\-, and sentence\-level evidence are all represented for every claim\.

### 4\.4Phase III: Contrastive NLI Verification

The core verification stage replaces open\-ended “LLM\-as\-a\-judge” prompting with a three\-way NLI formulation\. We employDeBERTa\-v3\-LargeHeet al\.\([2021](https://arxiv.org/html/2608.05823#bib.bib17)\)in its NLI\-fine\-tuned variant \(trained on MNLI, FEVER\-NLI, ANLI, LingNLI, and WANLI\), i\.e\., a dedicated entailment classifier rather than a generic encoder\.

##### Encoder\-only verification\.

This is a deliberate design choice: encoders jointly attend over premise and hypothesis, enabling fine\-grained detection of logical inconsistencies, and the model is trained explicitly for entailment classification, providing strong zero\-shot verification performance\.

##### Contrastive scoring\.

For each atomic claimcic\_\{i\}, HallDetect computes a contrastive score balancing evidence of support against evidence of contradiction, derived directly from the three\-way softmax of the NLI engine:

S​c​o​r​e​\(ci\)=maxk∈𝒦⁡PNLI​\(E∣k,ci\)−maxk∈𝒦⁡PNLI​\(C∣k,ci\),\\begin\{split\}Score\(c\_\{i\}\)=\{\}&\\max\_\{k\\in\\mathcal\{K\}\}P\_\{\\text\{NLI\}\}\(E\\mid k,c\_\{i\}\)\\\\ &\-\\max\_\{k\\in\\mathcal\{K\}\}P\_\{\\text\{NLI\}\}\(C\\mid k,c\_\{i\}\),\\end\{split\}\(1\)wherePNLI​\(E\)P\_\{\\text\{NLI\}\}\(E\)andPNLI​\(C\)P\_\{\\text\{NLI\}\}\(C\)denote the predicted probabilities of the*Entailment*and*Contradiction*classes given chunkkkand claimcic\_\{i\}, and the maximizing chunks forEEandCCare selected*independently*\. Intuitively, the twomax\\maxoperators ask complementary questions:*is there any window of the source under which this claim is strongly supported?*and, separately,*is there any window under which it is strongly contradicted?*A claim is treated as well\-grounded only when the best available support outweighs the best available contradiction, so a single strongly contradicting window suffices to driveS​c​o​r​e​\(ci\)Score\(c\_\{i\}\)negative even if the claim is loosely compatible with much of the document\.

##### Treatment of the Neutral class\.

The*Neutral*probability does not enter the score explicitly but shapes it implicitly through the softmax: a claim for which every chunk is predominantly neutral obtains both lowP​\(E\)P\(E\)and lowP​\(C\)P\(C\), yielding a score near zero\. Such claims are treated as*unsupported*\(extrinsic additions\) rather than*contradicted*, which is exactly the distinction illustrated by claimsc2c\_\{2\}andc3c\_\{3\}in the worked example of Appendix[B](https://arxiv.org/html/2608.05823#A2)\.

##### Computational profile\.

HallDetect is frugal in a specific and quantifiable sense\. The expensive operation in every pipeline we consider is autoregressive LLM*decoding*; HallDetect invokes it*exactly once*—for claim extraction—and, crucially, only over the short responseRR\(capped at512512new tokens\),*never*placing the long source document in a generative context\. All source\-grounded verification is delegated to the DeBERTa\-v3 encoder, which at435435M parameters is nearly20×20\\timessmaller than the88B\-class extractor, occupies<1<1GB of VRAM, and performs a*single, non\-autoregressive*forward pass per claim–chunk pair—at most\|𝒦\|⋅n≤31​n\|\\mathcal\{K\}\|\\cdot n\\leq 31npasses fornnclaims, running concurrently with the extractor on one consumer GPU \(e\.g\., RTX 3060/4060\)\. By contrast, theLLM CoTjudge must ingest the*entire*700700–800800\-word document into the generative context and decode a full reasoning chain, andSelfCheckrequiresN=5N\{=\}5such generations\. Counting generative LLM invocations per instance, HallDetect uses11\(over a short input\), LLM CoT11\(over a long input*and*a long output\), and SelfCheck55; the discriminative verifier adds only a sub\-dominant∼5%\\sim\\\!5\\%of the extractor’s parameters at single\-pass cost\. HallDetect is thus not the absolute cheapest scorer—the embedding baselines are lighter but markedly weaker—but it attains decomposition\-level accuracy*without*the repeated or long\-context generative passes that dominate the cost of competitive generative judges\. Appendix[A](https://arxiv.org/html/2608.05823#A1)\(Table[6](https://arxiv.org/html/2608.05823#A1.T6)\) gives the full accounting\.

### 4\.5Phase IV: Aggregation and theF​E​Ds​c​o​r​eFED\_\{score\}

The final output is theFactuality Entailment Decomposition score\(F​E​Ds​c​o​r​eFED\_\{score\}\), aggregating per\-claim scores via a geometric mean:

F​E​Ds​c​o​r​e=1−∏i=1n\(S​c​o​r​e​\(ci\)\)nFED\_\{score\}=1\-\\sqrt\[n\]\{\\prod\_\{i=1\}^\{n\}\\left\(Score\(c\_\{i\}\)\\right\)\}\(2\)The aggregation is intentionally asymmetric: whenever anyS​c​o​r​e​\(ci\)≤0Score\(c\_\{i\}\)\\leq 0, the geometric mean is defined as0and the response is flagged \(F​E​Ds​c​o​r​e=1FED\_\{score\}=1\)\. In auditing scenarios a single unsupported or false statement may invalidate an entire output, and the scoring function reflects this recall\-oriented design; its consequences for the precision–recall balance are analyzed in Section[6\.1\.3](https://arxiv.org/html/2608.05823#S6.SS1.SSS3)\.

##### Properties of the aggregation\.

Two features ofF​E​Ds​c​o​r​eFED\_\{score\}are worth making explicit\. First, because the geometric mean is dominated by its smallest factors, a single weakly\-supported claim \(small but strictly positiveS​c​o​r​e​\(ci\)Score\(c\_\{i\}\)\) depresses the aggregate far more than it would under an arithmetic mean; the geometric form thus already leans toward flagging even before the hard contradiction cutoff applies\. Concretely, over the regime in which all claim scores are strictly positive,F​E​Ds​c​o​r​e=1−\(∏iS​c​o​r​e​\(ci\)\)1/nFED\_\{score\}=1\-\\big\(\\prod\_\{i\}Score\(c\_\{i\}\)\\big\)^\{1/n\}is monotonically non\-increasing in everyS​c​o​r​e​\(ci\)Score\(c\_\{i\}\)and bounded in\[0,1\)\[0,1\), so strengthening the support of any claim can only lower the hallucination score, and no single well\-supported claim can by itself rescue a response whose other claims are weak\. Second, the edge caseS​c​o​r​e​\(ci\)≤0⇒F​E​Ds​c​o​r​e=1Score\(c\_\{i\}\)\\leq 0\\Rightarrow FED\_\{score\}=1makes the penalty discontinuous at the point where contradiction evidence overtakes entailment evidence for*any*claim: one confidently contradicted proposition saturates the score irrespective of how well the remaining claims are supported\. Taken together, the two regimes encode an auditing intuition in which faithfulness behaves more like a conjunction over claims than an average—a summary is unfaithful if*any*of its assertions is contradicted—while the geometric mean supplies a graded, monotone signal over the non\-contradicted regime rather than a bare binary flag\.

## 5Baselines and Comparative Evaluation Framework

To isolate the efficacy of HallDetect from raw model capacity, we enforce acontrolled\-resource protocol\. All LLM\-based baselines use identical 4\-bit \(Q4\_K\_M\) quantized engines:Llama\-3\.1\-8B,Gemma\-2\-9B, andMistral\-7B\. Embedding\-based methods useQwen2\.5\-0\.5B\-Embedto maintain a consistent, frugal budget\. These baselines are representative of the main black\-box detection paradigms under matched resources; they are not full\-precision state\-of\-the\-art systems, and our claims are scoped accordingly\.

##### Monolithic NLI Baseline\.

To quantify the gains attributable to decomposition and multi\-scale chunking jointly, we define a document\-level NLI baseline using the sameDeBERTa\-v3\-Largeengine, applied holistically to the full pair\(D,R\)\(D,R\):

HNLI​\(D,R\)=1−PNLI​\(Entailment∣D,R\)H\_\{\\text\{NLI\}\}\(D,R\)=1\-P\_\{\\text\{NLI\}\}\(\\text\{Entailment\}\\mid D,R\)\(3\)This baseline assesses whether holistic verification “averages away” localized hallucinations\. Note that it differs from HallDetect in*both*decomposition and chunking; disentangling the two effects requires ablations we identify as future work \(see Limitations\)\.

##### Generative Chain\-of\-Thought \(CoT\)\.

A representative “LLM\-as\-a\-judge” baseline in which the backbone is prompted to reason step\-by\-step before a binary verdict:

HDirect=\{1if​T​indicatesHallucinated0if​T​indicatesFaithfulH\_\{\\text\{Direct\}\}=\\begin\{cases\}1&\\text\{if \}T\\text\{ indicates \{Hallucinated\}\}\\\\ 0&\\text\{if \}T\\text\{ indicates \{Faithful\}\}\\end\{cases\}\(4\)whereTTis the generated judgment\. This tests whether unstructured generative reasoning in mid\-sized quantized models can match the explicit logical constraints of HallDetect\.

##### Consistency\-Based Scoring \(SelfCheck\)\.

Following the SelfCheckGPT paradigmManakulet al\.\([2023](https://arxiv.org/html/2608.05823#bib.bib67)\), we generateN=5N=5alternative summaries\{Si\}i=1N\\\{S\_\{i\}\\\}\_\{i=1\}^\{N\}and compute the average semantic divergence from the original response:

HSelfCheck=1N​∑i=1N\(1−cos⁡\(𝐞R,𝐞Si\)\)H\_\{\\text\{SelfCheck\}\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\(1\-\\cos\(\\mathbf\{e\}\_\{R\},\\mathbf\{e\}\_\{S\_\{i\}\}\)\\right\)\(5\)where𝐞\\mathbf\{e\}denotes the embedding vector\.

##### Semantic Divergence \(QwenScore\)\.

A lightweight, non\-generative baseline measuring coarse semantic distance between source and response:HQwen=1−cos⁡\(𝐞D,𝐞R\)H\_\{\\text\{Qwen\}\}=1\-\\cos\(\\mathbf\{e\}\_\{D\},\\mathbf\{e\}\_\{R\}\)\.

## 6Quantitative Results and Analysis

We evaluate HallDetect \(F​E​Ds​c​o​r​eFED\_\{score\}\) against the comparably resourced baselines of Section[5](https://arxiv.org/html/2608.05823#S5)across four benchmarks\. All LLM backbones \(Mistral\-7B, Llama\-3\.1\-8B, Gemma\-2\-9B\) are executed under 4\-bit \(Q4\_K\_M\) quantization\.QwenScoreandNLI OnlyareNon\-Generative Baselines: fixed, backbone\-independent reference points using no LLM extraction or prompting\. We report Precision, Recall, and F1 for the hallucinated class at a fixed threshold of0\.5\(Section[3\.1](https://arxiv.org/html/2608.05823#S3.SS1)\)\.

Table 1:Primary Results onQAGS\-CNN/DM\.Table 2:Cross\-Domain Results onTofuEval\(Dialogue Summarization\)\.Table 3:Adversarial Results onFalseSum\.Table 4:Specialized Results onPubMedQA\.### 6\.1Comparative Analysis and Key Findings

#### 6\.1\.1Architectural Resilience and Quantization Stability

The clearest finding \(Table[1](https://arxiv.org/html/2608.05823#S6.T1)\) is thestability gapbetween generative judges and HallDetect\. Under 4\-bit quantization LLM CoT is volatile—its F1 collapses from 0\.589 on Gemma\-2 to 0\.321 on Llama\-3\.1—suggesting that unstructured generative reasoning is acutely sensitive to quantization, or that some architectures falter at discriminative verification in a CoT format\. HallDetect instead stays within a narrow envelope \(F1 0\.734–0\.771\): decoupling claim extraction from verification \(delegated to a discriminative encoder\) shields the judgment from the decoding biases that compromise autoregressive reasoning in compressed LLMsLiet al\.\([2025](https://arxiv.org/html/2608.05823#bib.bib3)\)\.

#### 6\.1\.2The Decomposition Premium: Unmasking Localized Errors

The NLI Only baseline is our counter\-factual\. A monolithic check on QAGS\-CNN/DM reaches high precision \(0\.833\) but limited recall \(0\.517\): when claims are aggregated into one premise, localized contradictions are averaged away \(the “dilution effect”\)\. Operating atomically, HallDetect recovers these signals, reaching near\-saturation recall across all backbones without the precision collapse of simpler heuristics\. This comparison bundles decomposition with multi\-scale chunking; isolating each requires the ablations noted in Limitations\.

#### 6\.1\.3Recall Saturation and Threshold Sensitivity

HallDetect’s recall frequently approaches 0\.95–1\.0 at more modest precision—a direct, intended consequence of the asymmetric aggregation, which flags a response as soon as one claim is confidently contradicted\. This recall\-oriented operating point suits auditing, where a missed hallucination costs more than a false alarm, but it makes fixed\-0\.5 F1 most favorable on datasets with high hallucination base rates and does not characterize the full precision–recall trade\-off\. We therefore read the fixed\-threshold results as out\-of\-the\-box utility at one operating point, not dominance across all; threshold\-free analysis \(PR curves, AUROC\) is left to future work\.

#### 6\.1\.4Cross\-Domain Behavior: TofuEval and PubMedQA

On TofuEval, absolute performance is low for*all*methods \(best F1 0\.362 at precision 0\.224\): frugal\-budget faithfulness detection in dialogue remains hard, and HallDetect’s edge, while consistent, is modest rather than practically sufficient\. On PubMedQA, QwenScore and SelfCheck occasionally match or exceed HallDetect, likely because these scorers exploit biomedical knowledge in the backbones’ pre\-training as a prior for plausible statements\. HallDetect stays competitive while additionally localizing which medical claim lacks grounding—valuable in high\-stakes auditing—though we claim no superiority in this domain\.

## 7Discussion: Robustness and Interpretability

Generative evaluators often exhibitself\-consistency bias, validating hallucinations that resemble their own output patternsWataokaet al\.\([2024](https://arxiv.org/html/2608.05823#bib.bib8)\); assigning the final judgment to a discriminative encoder subjects each factual unit to an independent, logically constrained audit, which our results indicate also confers stability under quantization\. Beyond a binary flag, HallDetect produces anaudit trail—a mapping between atomic propositions and source\-evidence spans \(shared with other decomposition\-based evaluators\)—so reviewers can focus only on flagged claims; Appendix[B](https://arxiv.org/html/2608.05823#A2)traces this on a long\-source example, showing how the contrastive formulation separates contradicted claims from unsupported additions\.

## 8Conclusion

We presented HallDetect,111The source code for HallDetect and theF​E​Ds​c​o​r​eFED\_\{score\}implementation are publicly available at[https://anonymous\.4open\.science/r/HallDetect\_code\_review\-E702/](https://anonymous.4open.science/r/HallDetect_code_review-E702/)\.a reference\-free framework for interpretable hallucination detection under strict computational constraints\. Under a matched 4\-bit consumer\-hardware protocol, claim\-level decomposition with contrastive multi\-scale verification improves recall over holistic NLI and remains stable where generative judges are volatile, while exposing a claim\-to\-span audit trail\. Future work will include direct comparison to specialized fact\-checkers such as MiniCheck and FActScore\-style pipelines, ablations isolating decomposition from multi\-scale chunking, threshold\-free evaluation, Claim Grouping to reduce NLI overhead, and external knowledge for extrinsic verification\.

## 9Limitations

##### Missing comparisons to discriminative fact\-checkers\.

Our baselines cover the main black\-box paradigms under matched frugal budgets, but not the discriminative intrinsic detectors nearest to ours: MiniCheckTanget al\.\([2024a](https://arxiv.org/html/2608.05823#bib.bib184)\)—which shares our low\-cost, source\-grounded goal and offers a DeBERTa variant—and, secondarily, SummaCLabanet al\.\([2022](https://arxiv.org/html/2608.05823#bib.bib183)\)and QAFactEvalFabbriet al\.\([2022](https://arxiv.org/html/2608.05823#bib.bib7)\)\. Section[2\.2](https://arxiv.org/html/2608.05823#S2.SS2)draws the conceptual distinctions, but these are genuinely comparable systems; a controlled comparison—MiniCheck first—is the single most important next experiment\. Our claims are accordingly scoped to superiority over comparably resourced*generic and generative*baselines, not over the state of the art in fact\-checking\.

##### Confounded ablation\.

The document\-level NLI baseline differs from HallDetect in both decomposition and multi\-scale chunking, so their individual contributions to the recall gain cannot be separated here; ablations holding each factor fixed are needed to isolate the two effects\.

##### Recall\-oriented operating point\.

The asymmetric aggregation deliberately biases toward flagging, yielding near\-saturated recall at lower precision \(Section[6\.1\.3](https://arxiv.org/html/2608.05823#S6.SS1.SSS3)\)\. Fixed\-threshold F1 may thus flatter the method on high\-base\-rate datasets, and we report no precision–recall curves or threshold\-sensitivity analysis beyond the fixed 0\.5 cut\.

##### Efficiency is characterized architecturally, not measured\.

We report parameter counts, VRAM, and pass counts, but no wall\-clock or energy figures\. Verification requiresO​\(n⋅\|𝒦\|\)O\(n\\cdot\|\\mathcal\{K\}\|\)NLI passes and re\-encodes overlapping content across granularities\. A rigorous comparison should include the alternative of a single long\-context LLM call over the full source, where KV caching amortizes much of the \(already required\) extraction context; whether our chunked discriminative verification remains cheaper in wall\-clock terms is an open question\.

##### Single runs, backbones, and scope\.

All results are single deterministic runs, reported without variance or significance tests\. The backbones \(Llama\-3\.1, Gemma\-2, Mistral\-7B\) date to 2024; replication on newer families and broader suites such as LLM\-AggreFact would strengthen the evidence\. The main pipeline bottleneck is coreference during extraction—pronouns severed from distant antecedents by chunking propagate as spurious contradictions—and the evaluation targetsintrinsichallucinations only, since HallDetect lacks the world knowledge to verifyextrinsicclaims absent from the source\.

## 10Ethical Considerations

##### Data and licensing\.

All experiments rely exclusively on publicly available research benchmarks—QAGS\-CNN/DMWanget al\.\([2020](https://arxiv.org/html/2608.05823#bib.bib66)\), TofuEvalTanget al\.\([2024b](https://arxiv.org/html/2608.05823#bib.bib78)\), FalseSumUtamaet al\.\([2022](https://arxiv.org/html/2608.05823#bib.bib1)\), and PubMedQAJinet al\.\([2019](https://arxiv.org/html/2608.05823#bib.bib10)\)—and on publicly released pretrained models\. These artifacts are distributed for research purposes, and our use is consistent with their intended research use\. We did not collect any new data, and the corpora consist of public news articles, dialogue summaries, and biomedical abstracts that are not known to contain private or personally identifying information\. The HallDetect code andF​E​Ds​c​o​r​eFED\_\{score\}implementation we release are intended for research use only, consistent with the access conditions of the underlying datasets and models\.

##### Intended use and dual\-use risk\.

HallDetect is designed as a research tool for auditing the factual consistency of generated text\. It is not a certified safety system\. In high\-stakes domains such as the biomedical setting evaluated here, an over\-reliance on automated faithfulness scores could create a false sense of security: a highF​E​Ds​c​o​r​eFED\_\{score\}indicates intrinsic consistency with the provided source, not real\-world truth, and the framework verifies claims only against the supplied document rather than against external world knowledge\. We therefore recommend that HallDetect be used to support, not replace, human oversight in any decision\-critical deployment\.

##### Bias and limitations of the components\.

HallDetect inherits the biases of its constituent models: the generative claim extractors may omit or distort propositions, and the DeBERTa\-v3 NLI engine may encode systematic errors learned during pretraining\. Because all benchmarks are in English, our conclusions may not transfer to other languages or writing conventions\. Results are reported from single deterministic runs and should be interpreted accordingly\.

##### Environmental impact\.

A central design goal of HallDetect is computational frugality\. All experiments use 4\-bit quantized models that run on a single consumer\-grade GPU \(e\.g\., an RTX 3060/4060\), and the verification engine requires under 1 GB of VRAM\. This substantially lowers the energy footprint and hardware barrier relative to API\-based or full\-precision “LLM\-as\-a\-judge” approaches\.

##### Use of AI assistants\.

AI\-based coding and writing assistants were used for code scaffolding and for language editing of this manuscript\. All scientific claims, experimental designs, and reported results were produced and verified by the authors\.

## Acknowledgments

We thank the anonymous reviewers of the ACL Rolling Review \(ARR\) for their careful and constructive feedback\. Their comments—in particular on positioning this work relative to the decomposition\-based factuality literature \(e\.g\., FActScore, SummaC, QAFactEval, ACUEval, and MiniCheck\), on disentangling the contributions of atomic decomposition and multi\-scale chunking, on the recall\-saturation behavior induced by the asymmetric aggregation, on the treatment of the Neutral NLI class, and on making the computational\-frugality argument explicit rather than asserted—substantially improved the framing, scope, and rigor of this paper\. Any remaining shortcomings are our own\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2608.05823#S1.p1.1)\.
- The internal state of an llm knows when it‘s lying\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Cited by:[§2\.1](https://arxiv.org/html/2608.05823#S2.SS1.p1.1)\.
- S\. Banerjee, A\. Agarwal, and S\. Singla \(2025\)Llms will always hallucinate, and we need to live with this\.InIntelligent Systems Conference,pp\. 624–648\.Cited by:[§2](https://arxiv.org/html/2608.05823#S2.p1.1)\.
- L\. Chen, M\. Zaharia, and J\. Zou \(2024\)How is chatgpt’s behavior changing over time?\.Harvard Data Science Review6\(2\)\.Cited by:[§4](https://arxiv.org/html/2608.05823#S4.p1.1)\.
- S\. Chen, S\. Buthpitiya, A\. Fabrikant, D\. Roth, and T\. Schuster \(2023\)PropSegmEnt: a large\-scale corpus for proposition\-level segmentation and entailment recognition\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 8874–8893\.Cited by:[§1](https://arxiv.org/html/2608.05823#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.05823#S2.SS2.p3.1)\.
- F\. E\. Dorner, V\. Y\. Nastl, and M\. Hardt \(2024\)Limits to scalable evaluation at the frontier: llm as judge won’t beat twice the data\.arXiv preprint arXiv:2410\.13341\.Cited by:[§1](https://arxiv.org/html/2608.05823#S1.p2.1)\.
- R\. Eliav, A\. Cattan, E\. Hirsch, S\. Bassan, E\. Stengel\-Eskin, M\. Bansal, and I\. Dagan \(2025\)CLATTER: comprehensive entailment reasoning for hallucination detection\.arXiv preprint arXiv:2506\.05243\.Cited by:[§2\.3](https://arxiv.org/html/2608.05823#S2.SS3.p1.1)\.
- A\. R\. Fabbri, W\. Kryściński, B\. McCann, C\. Xiong, R\. Socher, and D\. Radev \(2021\)Summeval: re\-evaluating summarization evaluation\.Transactions of the Association for Computational Linguistics9,pp\. 391–409\.Cited by:[§3\.2](https://arxiv.org/html/2608.05823#S3.SS2.p3.1)\.
- A\. R\. Fabbri, C\. Wu, W\. Liu, and C\. Xiong \(2022\)QAFactEval: improved qa\-based factual consistency evaluation for summarization\.InProceedings of the 2022 conference of the north american chapter of the association for computational linguistics: Human language technologies,pp\. 2587–2601\.Cited by:[§1](https://arxiv.org/html/2608.05823#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.05823#S2.SS2.p5.1),[§9](https://arxiv.org/html/2608.05823#S9.SS0.SSS0.Px1.p1.1)\.
- T\. Falke, L\. F\. Ribeiro, P\. A\. Utama, I\. Dagan, and I\. Gurevych \(2019\)Ranking generated summaries by correctness: an interesting but challenging application for natural language inference\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 2214–2220\.Cited by:[§4\.1](https://arxiv.org/html/2608.05823#S4.SS1.p1.1)\.
- A\. Gunjal and G\. Durrett \(2024\)Molecular facts: desiderata for decontextualization in LLM fact verification\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 3751–3768\.Cited by:[§2\.2](https://arxiv.org/html/2608.05823#S2.SS2.p3.1)\.
- P\. He, J\. Gao, and W\. Chen \(2021\)Debertav3: improving deberta using electra\-style pre\-training with gradient\-disentangled embedding sharing\.arXiv preprint arXiv:2111\.09543\.Cited by:[§4\.4](https://arxiv.org/html/2608.05823#S4.SS4.p1.1)\.
- M\. Hu, B\. He, Y\. Wang, L\. Li, C\. Ma, and I\. King \(2024\)Mitigating large language model hallucination with faithful finetuning\.arXiv preprint arXiv:2406\.11267\.Cited by:[§2\.3](https://arxiv.org/html/2608.05823#S2.SS3.p1.1)\.
- L\. Huang, W\. Yu, W\. Ma, W\. Zhong, Z\. Feng, H\. Wang, Q\. Chen, W\. Peng, X\. Feng, B\. Qin,et al\.\(2025\)A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions\.ACM Transactions on Information Systems43\(2\),pp\. 1–55\.Cited by:[§2](https://arxiv.org/html/2608.05823#S2.p1.1)\.
- Q\. Jin, B\. Dhingra, Z\. Liu, W\. Cohen, and X\. Lu \(2019\)Pubmedqa: a dataset for biomedical research question answering\.InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing \(EMNLP\-IJCNLP\),pp\. 2567–2577\.Cited by:[§1](https://arxiv.org/html/2608.05823#S1.p3.1),[§10](https://arxiv.org/html/2608.05823#S10.SS0.SSS0.Px1.p1.1),[4th item](https://arxiv.org/html/2608.05823#S3.I1.i4.p1.1)\.
- W\. Kryściński, B\. McCann, C\. Xiong, and R\. Socher \(2020\)Evaluating the factual consistency of abstractive text summarization\.InProceedings of the 2020 conference on empirical methods in natural language processing \(EMNLP\),pp\. 9332–9346\.Cited by:[§3\.2](https://arxiv.org/html/2608.05823#S3.SS2.p3.1)\.
- P\. Laban, T\. Schnabel, P\. N\. Bennett, and M\. A\. Hearst \(2022\)SummaC: re\-visiting NLI\-based models for inconsistency detection in summarization\.Transactions of the Association for Computational Linguistics10,pp\. 163–177\.Cited by:[§2\.2](https://arxiv.org/html/2608.05823#S2.SS2.p5.1),[§9](https://arxiv.org/html/2608.05823#S9.SS0.SSS0.Px1.p1.1)\.
- H\. Li, H\. Wang, X\. Sun, H\. He, and J\. Feng \(2024\)Prompt\-Guided Generation of Structured Chest X\-Ray Report Using a Pre\-trained LLM\.In2024 IEEE International Conference on Multimedia and Expo \(ICME\),pp\. 1–6\.External Links:ISSN 1945\-788X,[Document](https://dx.doi.org/10.1109/ICME57554.2024.10687707)Cited by:[§1](https://arxiv.org/html/2608.05823#S1.p1.1),[§2](https://arxiv.org/html/2608.05823#S2.p1.1)\.
- J\. Li, X\. Cheng, W\. X\. Zhao, J\. Nie, and J\. Wen \(2023\)Halueval: a large\-scale hallucination evaluation benchmark for large language models\.arXiv preprint arXiv:2305\.11747\.Cited by:[§3\.2](https://arxiv.org/html/2608.05823#S3.SS2.p3.1)\.
- Z\. Li, Y\. Su, S\. Wang, R\. Yang, C\. Xie, A\. Liu, M\. Li, J\. Cao, N\. Wong, and H\. Yang \(2025\)QUANTIZATION meets reasoning: exploring and mitigating degradation of low\-bit llms in mathematical reasoning\.arXiv preprint arXiv:2505\.11574\.Cited by:[§6\.1\.1](https://arxiv.org/html/2608.05823#S6.SS1.SSS1.p1.1)\.
- C\. Lin \(2004\)Rouge: a package for automatic evaluation of summaries\.InText summarization branches out,pp\. 74–81\.Cited by:[§2\.1](https://arxiv.org/html/2608.05823#S2.SS1.p1.1)\.
- N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang \(2024\)Lost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.Cited by:[§4\.3](https://arxiv.org/html/2608.05823#S4.SS3.p1.3)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023\)G\-eval: nlg evaluation using gpt\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 2511–2522\.Cited by:[§2\.3](https://arxiv.org/html/2608.05823#S2.SS3.p1.1),[§4](https://arxiv.org/html/2608.05823#S4.p1.1)\.
- P\. Manakul, A\. Liusie, and M\. Gales \(2023\)SelfCheckGPT: zero\-resource black\-box hallucination detection for generative large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 9004–9017\.Cited by:[§2\.1](https://arxiv.org/html/2608.05823#S2.SS1.p1.1),[§5](https://arxiv.org/html/2608.05823#S5.SS0.SSS0.Px3.p1.2)\.
- J\. Maynez, S\. Narayan, B\. Bohnet, and R\. McDonald \(2020\)On faithfulness and factuality in abstractive summarization\.arXiv preprint arXiv:2005\.00661\.Cited by:[§1](https://arxiv.org/html/2608.05823#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.05823#S2.SS1.p1.1),[§2](https://arxiv.org/html/2608.05823#S2.p1.1)\.
- T\. Miller \(2019\)Explanation in artificial intelligence: insights from the social sciences\.Artificial intelligence267,pp\. 1–38\.Cited by:[§1](https://arxiv.org/html/2608.05823#S1.p2.1)\.
- S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\. Yih, P\. W\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. Hajishirzi \(2023\)FActScore: fine\-grained atomic evaluation of factual precision in long form text generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 12076–12100\.Cited by:[§1](https://arxiv.org/html/2608.05823#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.05823#S2.SS2.p2.1)\.
- A\. Szymanski, N\. Ziems, H\. A\. Eicher\-Miller, T\. J\. Li, M\. Jiang, and R\. A\. Metoyer \(2025\)Limitations of the llm\-as\-a\-judge approach for evaluating llm outputs in expert knowledge tasks\.InProceedings of the 30th International Conference on Intelligent User Interfaces,pp\. 952–966\.Cited by:[§1](https://arxiv.org/html/2608.05823#S1.p2.1)\.
- L\. Tang, P\. Laban, and G\. Durrett \(2024a\)MiniCheck: efficient fact\-checking of LLMs on grounding documents\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 8818–8847\.Cited by:[§A\.2](https://arxiv.org/html/2608.05823#A1.SS2.p1.8),[§2\.2](https://arxiv.org/html/2608.05823#S2.SS2.p5.1),[§9](https://arxiv.org/html/2608.05823#S9.SS0.SSS0.Px1.p1.1)\.
- L\. Tang, I\. Shalyminov, A\. Wong, J\. Burnsky, J\. Vincent, Y\. Yang, S\. Singh, S\. Feng, H\. Song, H\. Su,et al\.\(2024b\)Tofueval: evaluating hallucinations of llms on topic\-focused dialogue summarization\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 4455–4480\.Cited by:[§1](https://arxiv.org/html/2608.05823#S1.p3.1),[§10](https://arxiv.org/html/2608.05823#S10.SS0.SSS0.Px1.p1.1),[§2\.3](https://arxiv.org/html/2608.05823#S2.SS3.p1.1),[§2](https://arxiv.org/html/2608.05823#S2.p1.1),[2nd item](https://arxiv.org/html/2608.05823#S3.I1.i2.p1.1)\.
- J\. Thorne, A\. Vlachos, C\. Christodoulopoulos, and A\. Mittal \(2018\)FEVER: a large\-scale dataset for fact extraction and verification\.arXiv preprint arXiv:1803\.05355\.Cited by:[§3\.2](https://arxiv.org/html/2608.05823#S3.SS2.p3.1)\.
- P\. Utama, J\. Bambrick, N\. S\. Moosavi, and I\. Gurevych \(2022\)Falsesum: generating document\-level nli examples for recognizing factual inconsistency in summarization\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 2763–2776\.Cited by:[§1](https://arxiv.org/html/2608.05823#S1.p3.1),[§10](https://arxiv.org/html/2608.05823#S10.SS0.SSS0.Px1.p1.1),[3rd item](https://arxiv.org/html/2608.05823#S3.I1.i3.p1.1)\.
- D\. Wan, K\. Sinha, S\. Iyer, A\. Celikyilmaz, M\. Bansal, and R\. Pasunuru \(2024\)ACUEval: fine\-grained hallucination evaluation and correction for abstractive summarization\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 10036–10056\.Cited by:[§2\.2](https://arxiv.org/html/2608.05823#S2.SS2.p4.1)\.
- A\. Wang, K\. Cho, and M\. Lewis \(2020\)Asking and answering questions to evaluate the factual consistency of summaries\.Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics\.External Links:[Link](http://dx.doi.org/10.18653/v1/2020.acl-main.450),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.450)Cited by:[§1](https://arxiv.org/html/2608.05823#S1.p3.1),[§10](https://arxiv.org/html/2608.05823#S10.SS0.SSS0.Px1.p1.1),[§2\.1](https://arxiv.org/html/2608.05823#S2.SS1.p1.1),[§2](https://arxiv.org/html/2608.05823#S2.p1.1),[1st item](https://arxiv.org/html/2608.05823#S3.I1.i1.p1.1)\.
- M\. Wanner, S\. Ebner, Z\. Jiang, M\. Dredze, and B\. Van Durme \(2024\)A closer look at claim decomposition\.InProceedings of the 13th Joint Conference on Lexical and Computational Semantics \(\*SEM 2024\),pp\. 153–175\.Cited by:[§2\.2](https://arxiv.org/html/2608.05823#S2.SS2.p3.1)\.
- K\. Wataoka, T\. Takahashi, and R\. Ri \(2024\)Self\-preference bias in llm\-as\-a\-judge\.arXiv preprint arXiv:2410\.21819\.Cited by:[§2\.3](https://arxiv.org/html/2608.05823#S2.SS3.p1.1),[§7](https://arxiv.org/html/2608.05823#S7.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§2\.3](https://arxiv.org/html/2608.05823#S2.SS3.p1.1),[§4](https://arxiv.org/html/2608.05823#S4.p1.1)\.

## Appendix AImplementation Details

This appendix documents the concrete configuration used in all experiments so that the results in Section[6](https://arxiv.org/html/2608.05823#S6)are fully reproducible\. All components run locally on a single consumer\-grade GPU\.

### A\.1Model and Runtime Configuration

Table[5](https://arxiv.org/html/2608.05823#A1.T5)summarizes the settings of the two model components of the pipeline: the generative claim extractor \(Phase I\) and the discriminative NLI engine \(Phase III\)\.

Table 5:Hyperparameters and runtime configuration of the HallDetect pipeline\.
### A\.2Computational Cost Accounting

Table[6](https://arxiv.org/html/2608.05823#A1.T6)makes the frugality comparison of Section[6](https://arxiv.org/html/2608.05823#S6)explicit\. The dominant cost across pipelines is autoregressive LLM*decoding*, so we report, per evaluated instance, the number of such generative passes, what the LLM must read, and the auxiliary \(embedding or NLI\) scorer with its parameter count and pass type\. Two contrasts stand out\. First, HallDetect issues a*single*generative pass, and only over the short responseRR; unlike the LLM CoT and SelfCheck judges, the long source document is never placed in a generative context—it is seen only by the435435M encoder, one non\-autoregressive forward pass at a time\. Second, the verifier’s parameter footprint \(435435M,<1<1GB VRAM\) is roughly5%5\\%of the88B\-class extractor and runs on the same consumer GPU, so the verification stage is sub\-dominant despite scaling asO​\(n⋅\|𝒦\|\)O\(n\\cdot\|\\mathcal\{K\}\|\)passes \(\|𝒦\|≤31\|\\mathcal\{K\}\|\\leq 31\)\. We note that a trained compact checker such as MiniCheckTanget al\.\([2024a](https://arxiv.org/html/2608.05823#bib.bib184)\)occupies a comparably lightweight point in this space—indeed it can skip generative extraction entirely—which is why it is the natural target of the controlled comparison we flag in Section[9](https://arxiv.org/html/2608.05823#S9); we do not claim a frugality advantage over it, only over the generative judges\.

Table 6:Per\-instance cost accounting\. “Gen\. passes” counts autoregressive LLM decodes \(the dominant cost\); “Scorer / passes” gives the auxiliary embedding or NLI model and its number of single \(non\-autoregressive\) forward passes, withnnthe claim count and\|𝒦\|≤31\|\\mathcal\{K\}\|\\leq 31the chunk\-library size\. HallDetect confines generation to one short\-input pass and offloads all source verification to a sub\-1 GB encoder\.
### A\.3Claim Extraction Prompt

The number of extracted claims is capped dynamically asmin⁡\(10,4⋅nsent\)\\min\(10,\\,4\{\\cdot\}n\_\{\\text\{sent\}\}\), wherensentn\_\{\\text\{sent\}\}is the sentence count of the response\. The prompt is deliberately*copy\-faithful*: the extractor is instructed to reproduce claims verbatim, including implausible or self\-contradictory statements, so that verification is performed on the model’s actual assertions rather than on a silently corrected paraphrase\.

> Extract the factual claims from the following answer\. Each claim must be taken exactly as it is written in the answer, even if it appears implausible, incorrect, or contradictory\. Do not use outside knowledge\. Do not fix, rephrase, or interpret claims\. Your task is to copy the claims into standalone sentences exactly as stated\. List up to \{max\_claims\} claims\. No explanations, no corrections\. Answer: """\{answer\}""" Claims:

Raw output is post\-processed by stripping enumeration artifacts \(leading digits, bullets, hyphens, and trailing punctuation\) and discarding empty lines, yielding the atomic claim set\{c1,…,cn\}\\\{c\_\{1\},\\dots,c\_\{n\}\\\}\.

### A\.4Context Library Construction

For each granularitym∈\{1,2,4,8,16\}m\\in\\\{1,2,4,8,16\\\}, the source documentDDis split intommroughly equal sentence groups \(chunk size=⌈nsent/m⌉=\\lceil n\_\{\\text\{sent\}\}/m\\rceil\), and the union of all groups forms the context library𝒦\\mathcal\{K\}\. For a document ofnsentn\_\{\\text\{sent\}\}sentences this yields up to1\+2\+4\+8\+16=311\{\+\}2\{\+\}4\{\+\}8\{\+\}16=31overlapping chunks\. Crucially, the best entailment chunk and the best contradiction chunk are selected*independently*: for each claim the engine retains the chunk that most strongly supports it and, separately, the chunk that most strongly contradicts it\. This multi\-scale, dual\-selection view ensures that the relevant evidence is salient at*some*granularity, mitigating the dilution that arises when verifying against the full document at a single scale\.

### A\.5Scoring and Aggregation

For each claimcic\_\{i\}, the NLI engine is run against every chunkk∈𝒦k\\in\\mathcal\{K\}, and the best entailment and contradiction probabilities are retained independently:

S​c​o​r​e​\(ci\)=maxk∈𝒦⁡PNLI​\(E∣k,ci\)−maxk∈𝒦⁡PNLI​\(C∣k,ci\)\\begin\{split\}Score\(c\_\{i\}\)=\{\}&\\max\_\{k\\in\\mathcal\{K\}\}P\_\{\\text\{NLI\}\}\(E\\mid k,c\_\{i\}\)\\\\ &\-\\max\_\{k\\in\\mathcal\{K\}\}P\_\{\\text\{NLI\}\}\(C\\mid k,c\_\{i\}\)\\end\{split\}\(6\)
The per\-claim scores are aggregated with a geometric mean to obtain the finalF​E​Ds​c​o​r​eFED\_\{score\}\(Eq\. 2\)\. A practically important detail is the treatment of the aggregation edge case: whenever*any*S​c​o​r​e​\(ci\)≤0Score\(c\_\{i\}\)\\leq 0\(i\.e\., contradiction evidence outweighs entailment evidence for at least one claim\), the geometric mean is defined to be0, so thatF​E​Ds​c​o​r​e=1FED\_\{score\}=1\. This realizes the asymmetric penalty discussed in Section[6](https://arxiv.org/html/2608.05823#S6): a single confidently contradicted claim is sufficient to flag the entire response as unfaithful\. A binary verdict is obtained by thresholding at0\.50\.5\(hallucinatedifF​E​Ds​c​o​r​e≥0\.5FED\_\{score\}\\geq 0\.5\), with no per\-dataset tuning\.

## Appendix BWorked Example: A Long\-Source Audit Trail

To show how HallDetect operates on realistic, multi\-sentence inputs and why its output is auditable, we trace the full pipeline on a single news\-style source document and a generated summary\. Probabilities are reported on the\[0,1\]\[0,1\]scale \(the implementation stores them as percentages internally\) and are illustrative of the model’s behavior rather than logged values\.

##### Source documentDD\(nsent=8n\_\{\\text\{sent\}\}=8\)\.

> \(s1\)The city council approved a new transit plan on Tuesday after months of debate\.\(s2\)The plan allocates $4\.2 million to extend the eastern light\-rail line by three stations\.\(s3\)Mayor Lena Ortiz said the expansion would cut average commute times in the eastern districts by roughly fifteen minutes\.\(s4\)Construction is scheduled to begin in March 2026 and is expected to last two years\.\(s5\)Council member David Hsu voted against the proposal, citing concerns about the municipal budget\.\(s6\)The council also set aside $800,000 for accessibility upgrades at existing stations\.\(s7\)Local business owners welcomed the decision, anticipating increased foot traffic\.\(s8\)The plan does not include any changes to existing bus routes\.

##### Generated responseRR\.

> The council approved a $4\.2 million plan to extend the light\-rail line\. Mayor Ortiz opposed the expansion\. The new stations will feature underground parking\.

##### Phase I — Decomposition\.

The extractor returns three atomic claims:c1=c\_\{1\}=“The council approved a $4\.2 million plan to extend the light\-rail line\.”;c2=c\_\{2\}=“Mayor Ortiz opposed the expansion\.”;c3=c\_\{3\}=“The new stations will feature underground parking\.”

##### Phase II — Chunking\.

Withnsent=8n\_\{\\text\{sent\}\}=8, the granularitiesm∈\{1,2,4,8,16\}m\\in\\\{1,2,4,8,16\\\}produce chunk sizes\{8,4,2,1,1\}\\\{8,4,2,1,1\\\}, i\.e\. one whole\-document chunk, two 4\-sentence halves, four 2\-sentence windows, and \(for bothm=8m\{=\}8andm=16m\{=\}16\) eight single\-sentence chunks, for1\+2\+4\+8\+8=231\{\+\}2\{\+\}4\{\+\}8\{\+\}8=23distinct windows in𝒦\\mathcal\{K\}\.

##### Phase III/IV — Per\-claim verification and audit trail\.

Table[7](https://arxiv.org/html/2608.05823#A2.T7)reports, for each claim, the*independently*selected best\-entailing and best\-contradicting chunks together with their probabilities and the resultingS​c​o​r​e​\(ci\)Score\(c\_\{i\}\)\. This claim→\\rightarrowspan mapping*is*the audit trail: a human reviewer can read why each claim passed or failed by inspecting the cited sentence rather than re\-reading the whole document\.

Table 7:Audit trail for the long\-source example\.P​\(E\)P\(E\)andP​\(C\)P\(C\)are the best entailment / contradiction probabilities, and the “span” columns give the chunk \(sentence, granularitymm\) that achieved each, selected independently across all granularities\.c1c\_\{1\}is grounded in s2;c2c\_\{2\}is contradicted by s3 \(the mayor in fact*supported*the plan, while the opposition came from council member Hsu in s5\);c3c\_\{3\}is unsupported, with no chunk providing entailment at any granularity\.
##### Aggregation\.

Bothc2c\_\{2\}andc3c\_\{3\}yieldS​c​o​r​e​\(ci\)<0Score\(c\_\{i\}\)<0, so the geometric\-mean aggregation returns0and the response is flagged:F​E​Ds​c​o​r​e=1−0=1\.0≥0\.5FED\_\{score\}=1\-0=1\.0\\geq 0\.5\. The verdict is not an opaque scalar: the trail localizes the failure to a confident*contradiction*\(c2c\_\{2\}, traced to s3\) and an*unsupported addition*\(c3c\_\{3\}, no entailing span\), the two distinct error modes targeted by the framework\. The correctly supported claimc1c\_\{1\}is left untouched, so a reviewer’s attention is directed only to the two problematic spans\.

##### Why multi\-scale chunking matters here\.

Had verification been performed only atm=1m\{=\}1\(the monolithic NLI baseline of Section[5](https://arxiv.org/html/2608.05823#S5)\), the contradiction inc2c\_\{2\}would have been diluted: averaged across all eight sentences, the document is broadly*about*an approved expansion, so a holistic entailment check returns a moderate support signal and the localized “opposed” error is masked\. Selecting the best contradicting window atm=8m\{=\}8isolates s3, recovering the signal\. This is the mechanism behind the recall gains reported in Section[6](https://arxiv.org/html/2608.05823#S6)and the reason HallDetect’s judgments remain inspectable on long\-form inputs\.