Evidence Graph Consistency in Retrieval-Augmented Generation: A Model-Dependent Analysis of Hallucination Detection

# Evidence Graph Consistency in Retrieval-Augmented Generation: A Model-Dependent Analysis of Hallucination Detection
Source: [https://arxiv.org/html/2606.06748](https://arxiv.org/html/2606.06748)
###### Abstract

Retrieval\-Augmented Generation \(RAG\) reduces but does not eliminate hallucination in large language models\. Existing detection methods rely on flat similarity between generated answers and retrieved passages, ignoring structural relationships among evidence pieces and answer claims\. We propose Evidence Graph Consistency \(EGC\), a framework that constructs a local evidence graph per response and computes five structural consistency measures as hallucination indicators\. Evaluated on the full question answering split of RAGTruth across six LLMs \(5,767 responses\), EGC reveals a consistent model\-family split: graph consistency features show the expected diagnostic direction for hallucinations in Llama\-2 models but exhibit systematic reversal in GPT\-4, GPT\-3\.5, and Mistral\-7B\. This reversal suggests qualitatively different hallucination patterns across model families and indicates that embedding\-based graph consistency cannot serve as a model\-independent hallucination detection signal\.

## IIntroduction

Retrieval\-Augmented Generation \(RAG\) has become a standard technique for grounding large language model \(LLM\) outputs in external knowledge\[[1](https://arxiv.org/html/2606.06748#bib.bib1),[18](https://arxiv.org/html/2606.06748#bib.bib18)\]\. By conditioning generation on retrieved passages, RAG reduces the frequency of factually unsupported outputs\[[16](https://arxiv.org/html/2606.06748#bib.bib16)\]compared to parametric generation alone\. Nevertheless, RAG does not eliminate hallucination: LLMs continue to produce claims that contradict or extend beyond the retrieved evidence\[[2](https://arxiv.org/html/2606.06748#bib.bib2),[11](https://arxiv.org/html/2606.06748#bib.bib11),[22](https://arxiv.org/html/2606.06748#bib.bib22)\]\.

Existing approaches to hallucination detection in RAG settings largely rely on flat similarity between the generated answer and the retrieved context\[[3](https://arxiv.org/html/2606.06748#bib.bib3)\], or on prompting a separate LLM to judge faithfulness\[[4](https://arxiv.org/html/2606.06748#bib.bib4)\]\. Both approaches treat the relationship between answer and evidence as a single scalar signal, ignoring the structural relationships among individual evidence passages and answer claims\.

We propose Evidence Graph Consistency \(EGC\), a lightweight framework that constructs a local graph over the question, retrieved passages, and answer claims, then computes structural consistency features as hallucination indicators\. Applying EGC to the full question answering split of RAGTruth\[[2](https://arxiv.org/html/2606.06748#bib.bib2)\]across six LLMs, we find that its effectiveness is model\-dependent: EGC shows the expected diagnostic direction for Llama\-2 models but exhibits systematic reversal for GPT\-4, GPT\-3\.5, and Mistral\-7B, indicating qualitatively different hallucination patterns across model families\.

The contributions of this paper are as follows\. We use EGC as a structural probe to investigate when graph\-based consistency is and is not a valid hallucination signal across model families, rather than to propose a competitive detector\. We conduct a systematic evaluation across six LLMs on 5,767 RAG responses and reveal a consistent model\-family split with implications for the design of hallucination detection systems\. We analyse the structural patterns underlying this split and demonstrate that embedding\-based graph consistency alone is not reliable as a model\-independent hallucination signal\.

## IIRelated Work

Hallucination detection in RAG\.RAGAs\[[3](https://arxiv.org/html/2606.06748#bib.bib3)\]decomposes generated answers into atomic claims and checks each claim against the retrieved context using an LLM judge, computing a faithfulness score as the fraction of supported claims\. ARES\[[25](https://arxiv.org/html/2606.06748#bib.bib25)\]extends this paradigm by training lightweight classifiers to evaluate RAG systems without requiring human annotations\. FActScore\[[4](https://arxiv.org/html/2606.06748#bib.bib4)\]applies a similar decomposition strategy to long\-form generation, verifying claims against a retrieval corpus\. SelfCheckGPT\[[5](https://arxiv.org/html/2606.06748#bib.bib5)\]detects hallucinations by sampling multiple responses and measuring self\-consistency, without requiring external evidence\. LRP4RAG\[[21](https://arxiv.org/html/2606.06748#bib.bib21)\]detects hallucinations via layer\-wise relevance propagation over the model’s internal states, providing an interpretable signal that does not require external annotations\. RAG\-HAT\[[27](https://arxiv.org/html/2606.06748#bib.bib27)\]takes a complementary approach by fine\-tuning LLMs with hallucination\-aware preference data to reduce hallucination at generation time\. Unlike these approaches, EGC operates on graph topology derived from embedding similarity rather than LLM\-generated judgements or stochastic sampling, making it computationally lightweight and model\-agnostic\.

Graph\-based approaches in NLP\.Graph structures have been used for multi\-hop question answering\[[6](https://arxiv.org/html/2606.06748#bib.bib6),[20](https://arxiv.org/html/2606.06748#bib.bib20)\], knowledge\-grounded dialogue\[[12](https://arxiv.org/html/2606.06748#bib.bib12)\], and document summarisation\[[13](https://arxiv.org/html/2606.06748#bib.bib13),[26](https://arxiv.org/html/2606.06748#bib.bib26)\]\. These approaches typically rely on large pre\-constructed knowledge graphs\. EGC instead constructs a lightweight local graph per response at inference time, requiring no external graph infrastructure or pre\-processing\.

The RAGTruth corpus\.RAGTruth\[[2](https://arxiv.org/html/2606.06748#bib.bib2)\]provides 17,790 naturally generated responses from six LLMs across QA, data\-to\-text, and summarisation tasks, with word\-level hallucination annotations produced by human labellers\. It is a large\-scale RAG hallucination corpus with span\-level annotations across multiple model families\[[31](https://arxiv.org/html/2606.06748#bib.bib31)\], making it suitable for the cross\-model analysis we conduct\.

Model\-dependent evaluation\.Prior work has noted that hallucination rates vary substantially across model families\[[2](https://arxiv.org/html/2606.06748#bib.bib2),[11](https://arxiv.org/html/2606.06748#bib.bib11),[22](https://arxiv.org/html/2606.06748#bib.bib22),[28](https://arxiv.org/html/2606.06748#bib.bib28),[29](https://arxiv.org/html/2606.06748#bib.bib29),[30](https://arxiv.org/html/2606.06748#bib.bib30)\], but the downstream effect on detection method effectiveness has not been systematically studied\. Our results show that this variation is qualitative: the diagnostic direction of EGC measures differs across the evaluated model families\. Unlike prior work that aims to maximise detection accuracy, this paper uses EGC as a diagnostic lens to examine when structural consistency is and is not a valid hallucination signal\.

## IIIMethod

We propose Evidence Graph Consistency \(EGC\), a lightweight framework that constructs a local evidence graph per RAG response and derives structural consistency features as hallucination indicators\.

### III\-AGraph Construction

Given a questionqq, a set of retrieved passages𝒫=\{p1,…,pk\}\\mathcal\{P\}=\\\{p\_\{1\},\\ldots,p\_\{k\}\\\}, and a generated answeraa, we construct an undirected graphG=\(V,ℰ\)G=\(V,\\mathcal\{E\}\)with three node types\.

Nodes\.A singlequestion nodevqv\_\{q\}represents the input question\. Each passagepip\_\{i\}becomes anevidence nodeveiv\_\{e\_\{i\}\}\. The answeraais segmented into sentences using spaCy\[[8](https://arxiv.org/html/2606.06748#bib.bib8)\], and each sentence of length greater than ten tokens becomes aclaim nodevcjv\_\{c\_\{j\}\}\. All nodes are encoded withall\-MiniLM\-L6\-v2\[[7](https://arxiv.org/html/2606.06748#bib.bib7),[19](https://arxiv.org/html/2606.06748#bib.bib19)\]\. Throughout,ℰ\\mathcal\{E\}denotes the edge set ofGG, whileVEV\_\{E\}andVCV\_\{C\}denote the sets of evidence and claim nodes respectively\.

Edges\.We define three edge types based on cosine similaritysim​\(⋅,⋅\)\\text\{sim\}\(\\cdot,\\cdot\)with thresholdτ=0\.4\\tau=0\.4:

Cosine similarity is chosen because all nodes are encoded as fixed\-dimensional dense vectors byall\-MiniLM\-L6\-v2; the similarity between any two nodes is therefore independent of the original text length and is defined assim​\(𝐮,𝐯\)=𝐮⋅𝐯‖𝐮‖​‖𝐯‖\\text\{sim\}\(\\mathbf\{u\},\\mathbf\{v\}\)=\\frac\{\\mathbf\{u\}\\cdot\\mathbf\{v\}\}\{\\\|\\mathbf\{u\}\\\|\\\|\\mathbf\{v\}\\\|\}, where𝐮,𝐯∈ℝ384\\mathbf\{u\},\\mathbf\{v\}\\in\\mathbb\{R\}^\{384\}are the corresponding embeddings\[[24](https://arxiv.org/html/2606.06748#bib.bib24)\]\.

- •Q–E edges: connectvqv\_\{q\}toveiv\_\{e\_\{i\}\}ifsim​\(vq,vei\)≥τ\\text\{sim\}\(v\_\{q\},v\_\{e\_\{i\}\}\)\\geq\\tau
- •E–C edges: connectveiv\_\{e\_\{i\}\}tovcjv\_\{c\_\{j\}\}ifsim​\(vei,vcj\)≥τ\\text\{sim\}\(v\_\{e\_\{i\}\},v\_\{c\_\{j\}\}\)\\geq\\tau
- •E–E edges: connectveiv\_\{e\_\{i\}\}tovelv\_\{e\_\{l\}\}ifsim​\(vei,vel\)≥τ\\text\{sim\}\(v\_\{e\_\{i\}\},v\_\{e\_\{l\}\}\)\\geq\\tauandi≠li\\neq l

Fig\.[1](https://arxiv.org/html/2606.06748#S3.F1)illustrates the resulting graph structure for a grounded and a hallucinated answer\.

![Refer to caption](https://arxiv.org/html/2606.06748v1/figure2.png)

Figure 1:Evidence graph structure for a grounded answer \(left\) and a hallucinated answer \(right\) from Llama\-2\-13B\. In the grounded case all claim nodes connect to evidence; in the hallucinated case claim nodes are fully isolated\.
### III\-BConsistency Features

We compute five scalar measures fromGG, whered​\(v\)d\(v\)denotes the degree of nodevv\.

Coveragemeasures the fraction of claims supported by at least one evidence node:

cov=\|\{vc∈VC:∃ve∈VE,\(vc,ve\)∈ℰ\}\|\|VC\|\\text\{cov\}=\\frac\{\|\\\{v\_\{c\}\\in V\_\{C\}:\\exists\\,v\_\{e\}\\in V\_\{E\},\\,\(v\_\{c\},v\_\{e\}\)\\in\\mathcal\{E\}\\\}\|\}\{\|V\_\{C\}\|\}\(1\)
Support densitymeasures the average number of evidence nodes per claim, normalised by the total number of evidence nodes:

sup=1\|VC\|​∑vc∈VC\|\{ve∈VE:\(vc,ve\)∈ℰ\}\|\|VE\|\\text\{sup\}=\\frac\{1\}\{\|V\_\{C\}\|\}\\sum\_\{v\_\{c\}\\in V\_\{C\}\}\\frac\{\|\\\{v\_\{e\}\\in V\_\{E\}:\(v\_\{c\},v\_\{e\}\)\\in\\mathcal\{E\}\\\}\|\}\{\|V\_\{E\}\|\}\(2\)
Cross\-evidence agreementis the mean weight of all E–E edges, capturing whether retrieved passages are mutually consistent:

agr=∑\(ve,ve′\)∈ℰEEsim​\(ve,ve′\)\|ℰEE\|\\text\{agr\}=\\frac\{\\sum\_\{\(v\_\{e\},v\_\{e^\{\\prime\}\}\)\\in\\mathcal\{E\}\_\{\\text\{EE\}\}\}\\text\{sim\}\(v\_\{e\},v\_\{e^\{\\prime\}\}\)\}\{\|\\mathcal\{E\}\_\{\\text\{EE\}\}\|\}\(3\)whereℰEE\\mathcal\{E\}\_\{\\text\{EE\}\}is the set of E–E edges\. If no E–E edges exist,agr=0\\text\{agr\}=0\.

Connectivitymeasures the fraction of claim nodes reachable from the question node via any path inGG:

conn=\|\{vc∈VC:vq↝vc​in​G\}\|\|VC\|\\text\{conn\}=\\frac\{\|\\\{v\_\{c\}\\in V\_\{C\}:v\_\{q\}\\leadsto v\_\{c\}\\text\{ in \}G\\\}\|\}\{\|V\_\{C\}\|\}\(4\)
Isolation penaltyis the fraction of claim nodes with no edges, indicating claims entirely unsupported by retrieved evidence:

iso=\|\{vc∈VC:d​\(vc\)=0\}\|\|VC\|\\text\{iso\}=\\frac\{\|\\\{v\_\{c\}\\in V\_\{C\}:d\(v\_\{c\}\)=0\\\}\|\}\{\|V\_\{C\}\|\}\(5\)
Because claim nodes connect only to evidence nodes, the isolation penalty is the complement of coverage, that is,iso=1−cov\\mathrm\{iso\}=1\-\\mathrm\{cov\}\. We retain both quantities because they offer complementary interpretations: coverage emphasises supported claims, whereas isolation directly highlights unsupported ones\. They are not independent structural signals\.

Algorithm 1EGC Graph Construction0:Question

qq, passages

𝒫=\{p1,…,pk\}\\mathcal\{P\}=\\\{p\_\{1\},\\ldots,p\_\{k\}\\\}, answer

aa, similarity threshold

τ\\tau
0:Evidence graph

G=\(V,ℰ\)G=\(V,\\mathcal\{E\}\)
1:Encode

qq, each

pi∈𝒫p\_\{i\}\\in\\mathcal\{P\}, and each sentence

cjc\_\{j\}of

aawithall\-MiniLM\-L6\-v2to obtain embeddings

𝐞q\\mathbf\{e\}\_\{q\},

\{𝐞pi\}\\\{\\mathbf\{e\}\_\{p\_\{i\}\}\\\},

\{𝐞cj\}\\\{\\mathbf\{e\}\_\{c\_\{j\}\}\\\}
2:

V←\{vq\}∪\{vei\}∪\{vcj\}V\\leftarrow\\\{v\_\{q\}\\\}\\cup\\\{v\_\{e\_\{i\}\}\\\}\\cup\\\{v\_\{c\_\{j\}\}\\\}
3:

ℰ←∅\\mathcal\{E\}\\leftarrow\\emptyset
4:foreach

vei∈VEv\_\{e\_\{i\}\}\\in V\_\{E\}do

5:if

sim​\(𝐞q,𝐞pi\)≥τ\\text\{sim\}\(\\mathbf\{e\}\_\{q\},\\mathbf\{e\}\_\{p\_\{i\}\}\)\\geq\\tauthen

6:

ℰ←ℰ∪\{\(vq,vei\)\}\\mathcal\{E\}\\leftarrow\\mathcal\{E\}\\cup\\\{\(v\_\{q\},v\_\{e\_\{i\}\}\)\\\}
7:endif

8:endfor

9:foreach

vei∈VEv\_\{e\_\{i\}\}\\in V\_\{E\}, each

vcj∈VCv\_\{c\_\{j\}\}\\in V\_\{C\}do

10:if

sim​\(𝐞pi,𝐞cj\)≥τ\\text\{sim\}\(\\mathbf\{e\}\_\{p\_\{i\}\},\\mathbf\{e\}\_\{c\_\{j\}\}\)\\geq\\tauthen

11:

ℰ←ℰ∪\{\(vei,vcj\)\}\\mathcal\{E\}\\leftarrow\\mathcal\{E\}\\cup\\\{\(v\_\{e\_\{i\}\},v\_\{c\_\{j\}\}\)\\\}
12:endif

13:endfor

14:foreach pair

vei,vel∈VEv\_\{e\_\{i\}\},v\_\{e\_\{l\}\}\\in V\_\{E\},

i≠li\\neq ldo

15:if

sim​\(𝐞pi,𝐞pl\)≥τ\\text\{sim\}\(\\mathbf\{e\}\_\{p\_\{i\}\},\\mathbf\{e\}\_\{p\_\{l\}\}\)\\geq\\tauthen

16:

ℰ←ℰ∪\{\(vei,vel\)\}\\mathcal\{E\}\\leftarrow\\mathcal\{E\}\\cup\\\{\(v\_\{e\_\{i\}\},v\_\{e\_\{l\}\}\)\\\}
17:endif

18:endfor

19:return

G=\(V,ℰ\)G=\(V,\\mathcal\{E\}\)

Algorithm 2EGC Feature Computation0:Evidence graph

G=\(V,ℰ\)G=\(V,\\mathcal\{E\}\), node sets

VEV\_\{E\},

VCV\_\{C\}, question node

vqv\_\{q\}
0:Feature vector

𝐟=\[cov,sup,agr,conn,iso\]\\mathbf\{f\}=\[\\text\{cov\},\\text\{sup\},\\text\{agr\},\\text\{conn\},\\text\{iso\}\]
1:

nc←\|VC\|n\_\{c\}\\leftarrow\|V\_\{C\}\|
2:

cov←1nc∑vc∈VC𝟏\[∃ve∈VE:\(vc,ve\)∈ℰ\]\\text\{cov\}\\leftarrow\\frac\{1\}\{n\_\{c\}\}\\sum\_\{v\_\{c\}\\in V\_\{C\}\}\\mathbf\{1\}\[\\exists\\,v\_\{e\}\\in V\_\{E\}:\(v\_\{c\},v\_\{e\}\)\\in\\mathcal\{E\}\]
3:

sup←1nc⋅\|VE\|​∑vc∈VC\|\{ve∈VE:\(vc,ve\)∈ℰ\}\|\\text\{sup\}\\leftarrow\\frac\{1\}\{n\_\{c\}\\cdot\|V\_\{E\}\|\}\\sum\_\{v\_\{c\}\\in V\_\{C\}\}\|\\\{v\_\{e\}\\in V\_\{E\}:\(v\_\{c\},v\_\{e\}\)\\in\\mathcal\{E\}\\\}\|
4:

ℰEE←\{\(ve,ve′\)∈ℰ:ve,ve′∈VE\}\\mathcal\{E\}\_\{\\text\{EE\}\}\\leftarrow\\\{\(v\_\{e\},v\_\{e^\{\\prime\}\}\)\\in\\mathcal\{E\}:v\_\{e\},v\_\{e^\{\\prime\}\}\\in V\_\{E\}\\\}
5:if

\|ℰEE\|\>0\|\\mathcal\{E\}\_\{\\text\{EE\}\}\|\>0then

6:

agr←1\|ℰEE\|​∑\(ve,ve′\)∈ℰEEsim​\(𝐞ve,𝐞ve′\)\\text\{agr\}\\leftarrow\\frac\{1\}\{\|\\mathcal\{E\}\_\{\\text\{EE\}\}\|\}\\sum\_\{\(v\_\{e\},v\_\{e^\{\\prime\}\}\)\\in\\mathcal\{E\}\_\{\\text\{EE\}\}\}\\text\{sim\}\(\\mathbf\{e\}\_\{v\_\{e\}\},\\mathbf\{e\}\_\{v\_\{e^\{\\prime\}\}\}\)
7:else

8:

agr←0\\text\{agr\}\\leftarrow 0
9:endif

10:

conn←1nc​∑vc∈VC𝟏​\[vq↝vc​in​G\]\\text\{conn\}\\leftarrow\\frac\{1\}\{n\_\{c\}\}\\sum\_\{v\_\{c\}\\in V\_\{C\}\}\\mathbf\{1\}\[v\_\{q\}\\leadsto v\_\{c\}\\text\{ in \}G\]
11:

iso←1nc​∑vc∈VC𝟏​\[d​\(vc\)=0\]\\text\{iso\}\\leftarrow\\frac\{1\}\{n\_\{c\}\}\\sum\_\{v\_\{c\}\\in V\_\{C\}\}\\mathbf\{1\}\[d\(v\_\{c\}\)=0\]
12:return

𝐟=\[cov,sup,agr,conn,iso\]\\mathbf\{f\}=\[\\text\{cov\},\\text\{sup\},\\text\{agr\},\\text\{conn\},\\text\{iso\}\]

### III\-CHallucination Diagnosis

The five measures𝐟=\[cov,sup,agr,conn,iso\]\\mathbf\{f\}=\[\\text\{cov\},\\text\{sup\},\\text\{agr\},\\text\{conn\},\\text\{iso\}\]are standardised and passed to a logistic regression classifier\[[14](https://arxiv.org/html/2606.06748#bib.bib14),[23](https://arxiv.org/html/2606.06748#bib.bib23)\]with balanced class weights\. We use the official train and test splits of RAGTruth\[[2](https://arxiv.org/html/2606.06748#bib.bib2)\]and report AUROC, F1, and the per\-model EGC diagnostic gap:

Δm=EGC¯grounded\(m\)−EGC¯hallucinated\(m\)\\Delta\_\{m\}=\\overline\{\\text\{EGC\}\}\_\{\\text\{grounded\}\}^\{\(m\)\}\-\\overline\{\\text\{EGC\}\}\_\{\\text\{hallucinated\}\}^\{\(m\)\}\(6\)whereEGC¯\(m\)\\overline\{\\text\{EGC\}\}^\{\(m\)\}is the mean EGC score for modelmmand the composite EGC score is defined as:

EGC​\(a\)=cov\+sup\+conn−iso3\\text\{EGC\}\(a\)=\\frac\{\\text\{cov\}\+\\text\{sup\}\+\\text\{conn\}\-\\text\{iso\}\}\{3\}\(7\)
The score combines coverage, support density, and connectivity as positive indicators of evidence grounding, and subtracts isolation penalty as a negative indicator of unsupported claims\. Becauseiso=1−cov\\mathrm\{iso\}=1\-\\mathrm\{cov\}, the composite score places additional emphasis on whether claims are supported by the retrieved evidence\. The denominator normalises the score to the range\[−13,1\]\[\-\\frac\{1\}\{3\},1\]\. Cross\-evidence agreement is excluded from the composite score because it measures inter\-passage consistency rather than claim\-level grounding\. Although it provides limited aggregate diagnostic value, Table[V](https://arxiv.org/html/2606.06748#S5.T5)shows that it can capture model\-specific behaviour, including a strong reversed signal for GPT\-4\.

A positiveΔm\\Delta\_\{m\}indicates that EGC assigns higher scores to grounded answers, consistent with the intended diagnostic direction\.

## IVExperimental Setup

### IV\-ADataset

We evaluate on the question answering subset of RAGTruth\[[2](https://arxiv.org/html/2606.06748#bib.bib2)\], a manually annotated hallucination corpus for RAG applications\. The QA subset contains 5,767 responses generated by six LLMs across 989 unique questions drawn from MS MARCO\[[10](https://arxiv.org/html/2606.06748#bib.bib10)\]\. RAGTruth provides word\-level hallucination annotations produced by human labellers, categorising each hallucination span into one of four types: evident conflict, subtle conflict, evident baseless information, and subtle baseless information\. We derive a binary response\-level label by treating any response containing at least one annotated span of any type as hallucinated\. This response\-level label is appropriate because EGC features are computed over the full response graph rather than at the token or span level; a binary label aligns the graph\-level structural signal with the response\-level hallucination status\. We acknowledge that this binarisation does not exploit the severity or type of individual hallucination spans; future work could leverage span\-level annotations for claim\-level or edge\-level supervision\. Under this definition, 1,706 of 5,767 responses \(29\.6%\) are hallucinated, yielding a class ratio of approximately 1:2\.4\. We use the official train and test splits provided by the dataset authors \(4,892 train, 875 test\)\.

Table[I](https://arxiv.org/html/2606.06748#S4.T1)summarises the per\-model sample distribution across train and test splits\. Hallucination rates vary substantially across models, from 4\.4% for GPT\-4 to 51\.2% for Llama\-2\-7B, reflecting the well\-documented relationship between model capability and hallucination frequency\[[2](https://arxiv.org/html/2606.06748#bib.bib2)\]\.

TABLE I:Per\-Model Sample Distribution in RAGTruth QA SplitModelTotalTrainTestHal\.%GPT\-49618151464\.4GPT\-3\.59307891418\.1Mistral\-7B93979714240\.3Llama\-2\-70B98283414832\.4Llama\-2\-13B98483514940\.2Llama\-2\-7B97182214951\.2Total5767489287529\.6Hal\.% = percentage of hallucinated responses\.
### IV\-BModels and Implementation

All text representations are computed withall\-MiniLM\-L6\-v2\[[7](https://arxiv.org/html/2606.06748#bib.bib7)\], a 22M\-parameter sentence encoder trained on over one billion sentence pairs\. Sentence segmentation uses spaCyen\_core\_web\_sm\[[8](https://arxiv.org/html/2606.06748#bib.bib8)\]; sentences of ten tokens or fewer are discarded to avoid short, potentially uninformative claim nodes\. The similarity threshold is set toτ=0\.4\\tau=0\.4throughout all experiments\. This fixed value is not claimed to be universally optimal across all models or domains; rather, a single threshold is used deliberately so that any observed differences in diagnostic direction across model families can be attributed to the models themselves rather than to threshold variation\. Graph construction and feature computation are implemented with NetworkX\[[9](https://arxiv.org/html/2606.06748#bib.bib9)\]\.

The classifier is logistic regression\[[14](https://arxiv.org/html/2606.06748#bib.bib14)\]withclass\_weight=balanced, which reweights each training sample by the inverse frequency of its class\. This addresses the 1:2\.4 class imbalance without requiring oversampling or synthetic data generation\. Features are standardised to zero mean and unit variance before classification\. Because this study uses the released RAGTruth responses, no additional generation prompts are introduced in our experiments\. Given the fixed dataset, sentence segmenter, encoder, and similarity threshold, graph construction is fully deterministic\. This makes the reported EGC features reproducible without requiring access to the original LLM inference endpoints\. The only learned component is the logistic regression classifier, which is trained on the official RAGTruth training split and evaluated on the official test split\.

### IV\-CEvaluation

We report three metrics\. Area under the ROC curve \(AUROC\) is the primary metric as it is threshold\-independent and measures discrimination ability across the full range of classification thresholds, making it appropriate for imbalanced binary classification\. Macro F1\[[17](https://arxiv.org/html/2606.06748#bib.bib17)\], the unweighted mean of per\-class F1 scores, is reported as a secondary metric; it treats each class equally and is appropriate for imbalanced classification where both precision and recall matter\. The per\-model diagnostic gapΔm\\Delta\_\{m\}, defined in Section[III](https://arxiv.org/html/2606.06748#S3), quantifies the direction and magnitude of EGC effectiveness for each model family\. All metrics are computed on the official test split only; the train split is used exclusively for fitting the logistic regression classifier\.

## VResults

### V\-AOverall Diagnostic Performance

Table[II](https://arxiv.org/html/2606.06748#S5.T2)reports AUROC and F1 on the full test set\. The coverage\-only baseline achieves the strongest aggregate AUROC of 0\.589, while EGC reaches 0\.556\. This suggests that aggregate pooling obscures the model\-dependent behaviour examined in the following sections\.

Aggregate AUROC assumes a monotonic relationship between EGC score and hallucination label across all samples\. When that relationship flips sign between model families, as our per\-model analysis shows, pooling across families cancels the signal rather than measuring it\. The appropriate diagnostic lens for this question is therefore the per\-model direction and magnitude ofΔm\\Delta\_\{m\}, which directly captures whether structural graph consistency aligns with the expected hallucination signal for each model family\.

TABLE II:Overall Hallucination Detection Performance on RAGTruth QA Test SetTable[III](https://arxiv.org/html/2606.06748#S5.T3)further decomposes this result by model family and applies a direction correction\. We group Mistral\-7B with GPT\-class models because it exhibits the same reversed diagnostic direction as confirmed by the per\-model analysis in Section[V\-B](https://arxiv.org/html/2606.06748#S5.SS2)\. When models are grouped by family, Llama models alone yield AUROC 0\.524 and GPT/Mistral models alone yield AUROC 0\.509, both modest\. However, when the EGC score is sign\-flipped for the GPT/Mistral family before pooling, reflecting the reversal documented in Section[V\-B](https://arxiv.org/html/2606.06748#S5.SS2), the direction\-corrected AUROC rises to 0\.669, an improvement of\+0\.113\+0\.113over the uncorrected aggregate\. This is not a performance claim but an interpretive one: it demonstrates that the aggregate suppression is a direct consequence of combining opposing signals, and that EGC carries direction\-heterogeneous diagnostic information across model families rather than a uniformly weak signal\. The pooled metric therefore underestimates the diagnostic value of a signal that is consistent within families but opposed between them\.

TABLE III:Family\-Level and Direction\-Corrected AUROCEvaluationAUROCnnLlama family only0\.524448GPT \+ Mistral only0\.509427All models \(overall\)0\.556875All models \(direction\-corrected\)0\.669875Direction\-corrected: EGC scores sign\-flipped for GPT/Mistralbefore pooling, reflecting the observed family\-level reversal\.
### V\-BPer\-Model Analysis

Table[IV](https://arxiv.org/html/2606.06748#S5.T4)breaks down AUROC and the diagnostic gapΔm\\Delta\_\{m\}by model\. A clear split emerges between the Llama family and the remaining models\.

TABLE IV:Per\-Model EGC Diagnostic Gap and AUROCModelHal\.%AUROC𝚫𝒎\\boldsymbol\{\\Delta\_\{m\}\}DirectionLlama\-2\-13B40\.20\.536\+\+0\.029CorrectLlama\-2\-7B51\.20\.493\+\+0\.016CorrectLlama\-2\-70B32\.40\.528\+\+0\.007CorrectMistral\-7B40\.30\.505−\-0\.010ReversedGPT\-3\.58\.10\.244−\-0\.008ReversedGPT\-44\.40\.456−\-0\.038ReversedHal\.% = overall hallucination rate\.Direction is determined by the sign ofΔm\\Delta\_\{m\}, not by whetherAUROC is above or below 0\.5\.For all three Llama models,Δm\>0\\Delta\_\{m\}\>0: grounded answers receive higher EGC scores than hallucinated ones, consistent with the intended diagnostic direction\. GPT\-4 shows the strongest reversal atΔm=−0\.038\\Delta\_\{m\}=\-0\.038, meaning its hallucinated answers score systematically higher than its grounded ones\. GPT\-3\.5 and Mistral\-7B exhibit smaller but consistent reversals\.

Fig\.[2](https://arxiv.org/html/2606.06748#S5.F2)visualises the EGC score distributions and per\-model gap, showing that Llama hallucinations are structurally sparser while GPT\-4 hallucinations are not\.

![Refer to caption](https://arxiv.org/html/2606.06748v1/figure1.png)

Figure 2:Left: mean EGC score by model and label\. Right: per\-model diagnostic gapΔm\\Delta\_\{m\}\. Blue bars indicate models where EGC is effective; orange bars indicate reversal\.
### V\-CFeature Distributions

Fig\.[3](https://arxiv.org/html/2606.06748#S5.F3)shows the distribution of all five EGC features across the full test set\. Coverage and connectivity are bimodal, with a large mass at 1\.0 for grounded answers and a heavier tail near 0 for hallucinated ones\. Cross\-evidence agreement shows similar aggregate distributions for both classes, although the per\-model analysis in Table V reveals a strong reversed signal for GPT\-4\. Isolation penalty mirrors the coverage distribution as expected by construction\.

![Refer to caption](https://arxiv.org/html/2606.06748v1/fig3.png)

Figure 3:EGC feature distributions for grounded and hallucinated answers across all models \(top five panels\), and per\-model AUROC \(bottom right\)\. GPT\-3\.5 AUROC of 0\.244 reflects the systematic reversal described in Section[V\-B](https://arxiv.org/html/2606.06748#S5.SS2)\.
### V\-DCase Study

We examine two representative cases from Llama\-2\-13B to illustrate when EGC succeeds and fails\.

Grounded answer\(EGC score 1\.00\)\. For the question“butcher shop phone number”, the model correctly extracts a phone number from the retrieved passages\. All three claim nodes connect to at least one evidence node, yielding coverage = 1\.00, connectivity = 1\.00, and isolation = 0\.00\.

Hallucinated answer\(EGC score−\-0\.33\)\. For the question“do not turn off target tizen”, the model produces an answer referencing a SIM card, a detail absent from the retrieved passages\. Both claim nodes are fully isolated with no evidence connections, yielding coverage = 0\.00, connectivity = 0\.00, and isolation = 1\.00\. Fig\.[1](https://arxiv.org/html/2606.06748#S3.F1)illustrates the structural difference between these two cases\.

For GPT\-4, the reversal pattern is demonstrated by the question“how to clean under laptop keyboard without removing keys”\. The hallucinated answer scores EGC = 1\.00 because its phrasing closely mirrors the retrieved passages even though it introduces an unsupported procedural claim\. This linguistic alignment with the evidence corpus causes all five features to signal grounded behaviour, showing that EGC is unable to detect hallucinations that are semantically proximate to the evidence\.

### V\-ERobustness Analysis

Per\-feature analysis\.Table[V](https://arxiv.org/html/2606.06748#S5.T5)reports per\-feature AUROC for each model on the test split\. Coverage and connectivity show consistent behaviour across Llama models, with AUROC modestly above chance, while values fall below chance for GPT\-3\.5, reflecting the reversal pattern observed in the aggregate EGC score\. Notably, cross\-evidence agreement shows an elevated AUROC of 0\.772 for GPT\-4, but in the reversed direction: hallucinated GPT\-4 answers exhibit higher inter\-passage agreement than grounded ones\. Rather than contradicting the failure mode interpretation, this reinforces it\. GPT\-4 hallucinations closely mirror the retrieved passages, producing dense and mutually consistent E–E structure even when introducing unsupported claims\.

TABLE V:Per\-Feature AUROC by Model on Test SplitCov\.=Coverage, Sup\.=Support density, Agr\.=Agreement, Conn\.=Connectivity, Iso\.=Isolation penalty\. Coverage and isolation penalty are complementary; their per\-feature AUROC values coincide by construction\.

Threshold sensitivity\.A potential concern is that the model\-family split is an artifact of the fixed thresholdτ=0\.4\\tau=0\.4rather than a structural signal\. Table[VI](https://arxiv.org/html/2606.06748#S5.T6)reports the diagnostic gapΔm\\Delta\_\{m\}across five thresholds from 0\.3 to 0\.7, along with the direction consistency score: the fraction of models whose gap sign matches the expected direction \(positive for Llama, negative for GPT and Mistral\)\.

Atτ=0\.3\\tau=0\.3, graphs are overly dense because nearly all node pairs exceed the low threshold, compressing structural differences and yielding a consistency score of 0\.833, with Llama\-2\-70B showing an incorrect direction\. Atτ=0\.4\\tau=0\.4, the model\-family split is most pronounced: all six models show the expected direction, yielding a perfect consistency score of 1\.000\. Atτ≥0\.5\\tau\\geq 0\.5, graphs become increasingly sparse as fewer edges meet the higher threshold\. Most claim nodes become isolated regardless of model family, which causes hallucinated and grounded answers to converge in EGC score and progressively degrades directional consistency to 0\.667 atτ=0\.5\\tau=0\.5and 0\.500 atτ≥0\.6\\tau\\geq 0\.6\.

These results confirm two things\. First,τ=0\.4\\tau=0\.4is empirically the most discriminative operating point in our study: it is the unique threshold at which all six models show the expected directional split simultaneously\. This supports that the model\-family reversal is unlikely to be a trivial artifact of threshold choice\. Second, the degradation pattern is structurally predictable: at lower thresholds graphs become too dense to preserve discriminative structure, while at higher thresholds graphs become too sparse and all models converge toward high isolation regardless of label\. The fact that consistency degrades in opposite directions for opposite reasons further supportsτ=0\.4\\tau=0\.4as a meaningful operating point rather than an arbitrary one\.

TABLE VI:Diagnostic GapΔm\\Delta\_\{m\}and Direction Consistency Across Similarity ThresholdsTable[VII](https://arxiv.org/html/2606.06748#S5.T7)shows the per\-model directional correctness at each threshold\. Atτ=0\.4\\tau=0\.4, all six models show the correct direction\. Atτ=0\.3\\tau=0\.3, Llama\-2\-70B is the only exception\. Atτ≥0\.5\\tau\\geq 0\.5, GPT\-3\.5 and GPT\-4 shift to incorrect directions; Mistral\-7B follows atτ≥0\.6\\tau\\geq 0\.6\.

TABLE VII:Per\-Model Directional Correctness Across ThresholdsNotably,τ=0\.4\\tau=0\.4is the only threshold at which all six models show the correct diagnostic direction simultaneously, providing empirical justification for this choice and indicating that the split is most clearly observable at an intermediate operating point\.

## VIDiscussion and Conclusion

### VI\-AWhy EGC Works for Llama but Reverses for GPT

The model\-family split observed in Table[IV](https://arxiv.org/html/2606.06748#S5.T4)reflects a difference in how these models hallucinate\. Llama\-2 models, particularly at 7B and 13B scale, tend to produce hallucinated answers that are structurally disconnected from the retrieved evidence: the generated claims use vocabulary and phrasing that diverges from the source passages, resulting in sparse or absent E–C edges and low EGC scores\.

GPT\-4 and GPT\-3\.5 exhibit a different failure mode\. Their hallucinated answers are linguistically fluent and closely mirror the retrieved passages, often paraphrasing or recombining evidence\-level phrasing even when introducing unsupported claims\. This causes high cosine similarity between claim and evidence nodes, dense E–C connectivity, and consequently high EGC scores despite factual inaccuracy\. The reversal is strongest for GPT\-4, which has the lowest hallucination rate \(4\.4%\) and may produce comparatively subtle hallucinations\. Mistral\-7B sits between the two families: instruction\-tuned at 7B scale, it produces fluent outputs that partially mirror evidence phrasing, yielding a small but consistent reversal\.

This finding has a practical implication for hallucination detection system design\. Embedding\-based graph consistency signals are more informative for the evaluated Llama\-2 models but inadequate for GPT\-class models whose hallucinations remain semantically proximate to the retrieved evidence\. Detecting such cases requires entailment\-level or factual verification beyond embedding similarity, such as natural language inference\[[15](https://arxiv.org/html/2606.06748#bib.bib15)\]or verification against external knowledge\.

These results suggest a practical deployment guideline\. EGC should not be used as a universal stand\-alone hallucination detector\. Instead, it is best interpreted as a lightweight structural diagnostic layer whose reliability depends on the hallucination style of the underlying model\. For models whose unsupported claims tend to drift lexically or structurally away from the retrieved evidence, low EGC scores provide a useful warning signal\. For stronger models whose unsupported claims remain fluent and evidence\-proximate, high EGC scores should not be treated as evidence of factual grounding; rather, such cases should be passed to semantic verification methods such as natural language inference\[[15](https://arxiv.org/html/2606.06748#bib.bib15)\]or external factual verification\.

More broadly, these results have implications for how hallucination detection methods should be evaluated\. A single pooled benchmark score can obscure model\-dependent behaviour when the same signal has opposite diagnostic directions across model families\. In this setting, a modest aggregate AUROC does not necessarily imply that the structural signal is uninformative; it may instead indicate that the signal is heterogeneous across models, as demonstrated by the direction\-corrected AUROC of 0\.669 in Table[III](https://arxiv.org/html/2606.06748#S5.T3)\. Reporting per\-model or per\-family diagnostic behaviour is therefore important for evaluating RAG hallucination detectors, especially when benchmark responses are generated by multiple LLM families with qualitatively different hallucination patterns\.

### VI\-BLimitations

Several limitations bound the interpretation of these results\. First, although Section[V\-E](https://arxiv.org/html/2606.06748#S5.SS5)shows thatτ=0\.4\\tau=0\.4achieves perfect directional consistency across all six models, the threshold is not learned and may not generalise to other domains or embedding models\. Second, claim segmentation by sentence boundary is a coarse approximation; atomic claim extraction via natural language inference would provide finer\-grained claim nodes\. Third, cross\-evidence agreement provides limited aggregate signal, although its strong reversed association for GPT\-4 suggests that E–E structure may capture model\-specific behaviour not reflected in pooled evaluation\. Finally, all results are reported on a single dataset from a single domain \(open\-domain QA\); generalisation to other RAG tasks such as summarisation or data\-to\-text generation is not established\. A further limitation is that our evaluation converts RAGTruth’s span\-level annotations into a binary response\-level label\. While this is appropriate for the present graph\-level analysis, it discards information about hallucination span length, span type, and the number of unsupported claims within a response\. Future extensions could use the original span\-level annotations to define claim\-level labels or edge\-level supervision, enabling evaluation of whether specific claim–evidence links are individually faithful rather than treating the entire response as a single unit\.

### VI\-CFuture Work

The findings in this paper open several directions for future investigation, spanning both methodological improvements to the EGC framework and broader questions about the generalisability of the model\-family split observed here\.

Model\-adaptive thresholds\.The fixed thresholdτ=0\.4\\tau=0\.4achieves perfect directional consistency across all six models in our study, but it is not learned and may not generalise to other domains or embedding models\. A model\-adaptive or learned threshold that accounts for the embedding similarity distribution of the underlying model could improve feature separation, particularly for instruction\-tuned models whose outputs tend to be more fluent and evidence\-proximate\. Learningτ\\tauas a function of model family characteristics rather than treating it as a global hyperparameter is a natural extension of the current framework\.

Finer\-grained claim segmentation\.The current implementation segments answers at the sentence level, which is a coarse approximation that may conflate supported and unsupported claims within a single sentence\. Atomic claim extraction via natural language inference, as used in FActScore\[[4](https://arxiv.org/html/2606.06748#bib.bib4)\], would provide finer\-grained claim nodes and potentially stronger E–C edge signals\. This is especially relevant for GPT\-class models, whose hallucinations tend to be sub\-sentence level insertions into otherwise well\-supported responses\.

Cross\-task generalisation\.All results in this paper are from the RAGTruth QA split\. Extending EGC to the summarisation and data\-to\-text splits of RAGTruth would test whether the model\-family split generalises beyond open\-domain question answering\. These tasks involve longer responses and different hallucination patterns, and may reveal whether the reversal phenomenon is specific to QA or a broader characteristic of how different model families respond to retrieved context\.

Combining structural and semantic signals\.The fundamental limitation of EGC for GPT\-class models is that embedding\-based similarity alone cannot distinguish fluent paraphrasing from factual grounding\. Combining EGC features with internal model states, token\-level confidence scores, or NLI\-based entailment signals may address this limitation\. A hybrid detector that uses structural probes for weaker models and semantic verification for stronger ones could realise the model\-aware deployment strategy suggested by our findings\.

### VI\-DConclusion

The model\-family split revealed by EGC carries a broader implication: hallucination is not a monolithic phenomenon\. The evaluated Llama\-2 models tend to hallucinate by generating claims that are structurally disconnected from the evidence, making the gap detectable through graph topology\. The evaluated GPT\-class models hallucinate differently, producing fluent, evidence\-proximate text that defeats embedding\-based structural checks\. This distinction suggests that hallucination detection cannot rely on a single signal across all deployment contexts\. Effective systems will likely need to match their detection strategy to the characteristics of the underlying model\. For lightweight deployment on smaller open\-source RAG systems, structural graph probes of the kind proposed here remain a practical and computationally inexpensive option\. For stronger proprietary systems such as GPT\-4, where hallucinations are linguistically fluent and evidence\-proximate, semantic verification or natural language inference approaches are more appropriate\. This model\-aware framing may offer a more principled foundation for hallucination detection system design than a one\-size\-fits\-all signal\.

## References

- \[1\]P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal,et al\., “Retrieval\-augmented generation for knowledge\-intensive NLP tasks,” inAdvances in Neural Information Processing Systems, 2020, pp\. 9459–9474\.
- \[2\]C\. Niu, Y\. Wu, J\. Zhu, S\. Xu, K\. Shum, R\. Zhong, J\. Song, and T\. Zhang, “RAGTruth: A hallucination corpus for developing trustworthy retrieval\-augmented language models,” inProc\. 62nd Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand, 2024, pp\. 10862–10878\.
- \[3\]S\. Es, J\. James, L\. Espinosa Anke, and S\. Schockaert, “RAGAs: Automated evaluation of retrieval augmented generation,” inProc\. 18th Conference of the European Chapter of the Association for Computational Linguistics, 2024, pp\. 150–158\.
- \[4\]S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\. Tau Yih, P\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. Hajishirzi, “FActScore: Fine\-grained atomic evaluation of factual precision in long form text generation,” inProc\. 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, 2023, pp\. 12076–12100\.
- \[5\]P\. Manakul, A\. Liusie, and M\. Gales, “SelfCheckGPT: Zero\-resource black\-box hallucination detection for generative large language models,” inProc\. 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, 2023, pp\. 9004–9017\.
- \[6\]Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning, “HotpotQA: A dataset for diverse, explainable multi\-hop question answering,” inProc\. 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 2018, pp\. 2369–2380\.
- \[7\]N\. Reimers and I\. Gurevych, “Sentence\-BERT: Sentence embeddings using Siamese BERT\-networks,” inProc\. 2019 Conference on Empirical Methods in Natural Language Processing, Hong Kong, China, 2019, pp\. 3982–3992\.
- \[8\]M\. Honnibal, I\. Montani, S\. Van Landeghem, and A\. Boyd, “spaCy: Industrial\-strength natural language processing in Python,” Explosion AI, Tech\. Rep\., 2020\.
- \[9\]A\. A\. Hagberg, D\. A\. Schult, and P\. J\. Swart, “Exploring network structure, dynamics, and function using NetworkX,” inProc\. 7th Python in Science Conference, 2008\.
- \[10\]T\. Nguyen, M\. Rosenberg, X\. Song, J\. Gao, S\. Tiwary, R\. Majumder, and L\. Deng, “MS MARCO: A human generated machine reading comprehension dataset,” inProc\. Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches, vol\. 1773, 2016\.
- \[11\]Z\. Ji, N\. Lee, R\. Frieske, T\. Yu, D\. Su, Y\. Xu, E\. Ishii, Y\. Bang, A\. Madotto, and P\. Fung, “Survey of hallucination in natural language generation,”ACM Computing Surveys, vol\. 55, no\. 12, pp\. 1–38, 2023\.
- \[12\]E\. Dinan, S\. Roller, K\. Shuster, A\. Fan, M\. Auli, and J\. Weston, “Wizard of Wikipedia: Knowledge\-powered conversational agents,” inProc\. International Conference on Learning Representations, 2019\.
- \[13\]J\. Maynez, S\. Narayan, B\. Bohnet, and R\. McDonald, “On faithfulness and factuality in abstractive summarisation,” inProc\. 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp\. 1906–1919\.
- \[14\]F\. Pedregosa, G\. Varoquaux, A\. Gramfort, V\. Michel, B\. Thirion, O\. Grisel, M\. Blondel, P\. Prettenhofer, R\. Weiss, V\. Dubourg, J\. Vanderplas, A\. Passos, D\. Cournapeau, M\. Brucher, M\. Perrot, and E\. Duchesnay, “Scikit\-learn: Machine learning in Python,”Journal of Machine Learning Research, vol\. 12, 2011\.
- \[15\]S\. R\. Bowman, G\. Angeli, C\. Potts, and C\. D\. Manning, “A large annotated corpus for learning natural language inference,” inProc\. 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, 2015, pp\. 632–642\.
- \[16\]K\. Shuster, S\. Poff, M\. Chen, D\. Kiela, and J\. Weston, “Retrieval augmentation reduces hallucination in conversation,” inFindings of the Association for Computational Linguistics: EMNLP 2021, 2021, pp\. 3784–3803\.
- \[17\]C\. J\. Van Rijsbergen,Information Retrieval, 2nd ed\. Butterworth\-Heinemann, 1979\.
- \[18\]Y\. Gao, Y\. Xiong, X\. Gao, K\. Jia, J\. Pan, Y\. Bi, Y\. Dai, J\. Sun, M\. Wang, and H\. Wang, “Retrieval\-augmented generation for large language models: A survey,”arXiv preprint arXiv:2312\.10997, 2023\.
- \[19\]J\. Devlin, M\.\-W\. Chang, K\. Lee, and K\. Toutanova, “BERT: Pre\-training of deep bidirectional transformers for language understanding,” inProc\. 2019 Conference of the North American Chapter of the Association for Computational Linguistics, Minneapolis, MN, 2019, pp\. 4171–4186\.
- \[20\]M\. Yasunaga, H\. Ren, A\. Bosselut, P\. Liang, and J\. Leskovec, “QA\-GNN: Reasoning with language models and knowledge graphs for question answering,” inProc\. 2021 Conference of the North American Chapter of the Association for Computational Linguistics, 2021, pp\. 535–546\.
- \[21\]H\. Hu, C\. He, X\. Xie, and Q\. Zhang, “LRP4RAG: Detecting hallucinations in retrieval\-augmented generation via layer\-wise relevance propagation,” unpublished, arXiv:2408\.15533, 2024\.
- \[22\]L\. Huang, W\. Yu, W\. Ma, W\. Zhong, Z\. Feng, H\. Wang, Q\. Chen, W\. Peng, X\. Feng, B\. Qin, and T\. Liu, “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,”arXiv preprint arXiv:2311\.05232, 2023\.
- \[23\]D\. W\. Hosmer, S\. Lemeshow, and R\. X\. Sturdivant,Applied Logistic Regression, 3rd ed\. Wiley, 2013\.
- \[24\]T\. Mikolov, I\. Sutskever, K\. Chen, G\. Corrado, and J\. Dean, “Distributed representations of words and phrases and their compositionality,” inAdvances in Neural Information Processing Systems, vol\. 2, 2013, pp\. 3111–3119\.
- \[25\]J\. Saad\-Falcon, O\. Khattab, C\. Potts, and M\. Zaharia, “ARES: An automated evaluation framework for retrieval\-augmented generation systems,”arXiv preprint arXiv:2311\.09476, 2023\.
- \[26\]T\. Falke, L\. F\. R\. Ribeiro, P\. A\. Utama, I\. Dagan, and I\. Gurevych, “Ranking generated summaries by correctness: An interesting but challenging application for natural language inference,” inProc\. 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 2019, pp\. 2214–2220\.
- \[27\]J\. Song, X\. Wang, J\. Zhu, Y\. Wu, X\. Cheng, R\. Zhong, and C\. Niu, “RAG\-HAT: A hallucination\-aware tuning pipeline for LLM in retrieval\-augmented generation,” inProc\. 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, Miami, FL, 2024, pp\. 1548–1558\.
- \[28\]H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei,et al\., “Llama 2: Open foundation and fine\-tuned chat models,”arXiv preprintarXiv:2307\.09288, 2023\.
- \[29\]A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier,et al\., “Mistral 7B,”arXiv preprintarXiv:2310\.06825, 2023\.
- \[30\]OpenAI, “GPT\-4 technical report,”arXiv preprintarXiv:2303\.08774, 2023\.
- \[31\]W\. Fan, Y\. Ding, L\. Ning, S\. Wang, H\. Li, D\. Yin, T\.\-S\. Chua, and Q\. Li, “A survey on RAG meeting LLMs: Towards retrieval\-augmented large language models,” inProc\. 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp\. 6491–6501\.