A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization

# A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization
Source: [https://arxiv.org/html/2608.08180](https://arxiv.org/html/2608.08180)
2ndRakesh Chandra Balabantaray3rdKali Prasad Vittala4thNaman Kabadi

###### Abstract

Abstractive text summarization systems frequently generate fluent yet unfaithful summaries by fabricating or distorting relationships between entities and events\. Such relation\-level hallucinations undermine the reliability of generated summaries, particularly in high\-stakes domains\. In this work, we present a refined and grounded framework for evaluating relation hallucination in abstractive summarization\.

We present the empirical Relation Hallucination Index \(RHI\) by introducing a dependency\-aware relation extraction algorithm that incorporates lemmatization\-based normalization, named entity grounded subject resolution, passive agent recovery, negation\-aware verb modeling, reporting verb filtering, nominal relation fallback, clausal propagation, and systematic deduplication\. These enhancements improve the structural fidelity of extracted relation triples and reduce spurious matches during evaluation\.

In addition, we introduce a normalized formulation of RHI to ensure scale\-invariant comparison between datasets and models\. The revised metric decomposes hallucination into interpretable components, aggregates relation hallucination metric into a normalized relation faithfulness score\.

Extensive evaluation across multiple state\-of\-the\-art summarization models demonstrates that the grounded extraction process yields more stable and discriminative hallucination measurements\. The proposed framework advances automated relation\-level faithfulness evaluation and supports coherence\-aware, hallucination\-sensitive model analysis\.

## IIntroduction

Recent advances in natural text generation have significantly improved the fluency and readability of automatically generated summaries\. However, fluency does not guarantee factual reliability\. Abstractive summarization systems frequently introduce relational distortions, where entities are preserved but the connections between them are inaccurately constructed\. Such relation\-level hallucinations can subtly alter the meaning of source content, raising serious concerns in applications where correctness is critical\.

Although existing evaluation metrics emphasize lexical overlap or entity matching, they often fail to capture inconsistencies in structured relationships\. Assessing relational faithfulness therefore requires explicit modeling of subject–verb–object structures and systematic comparison between source documents, references, and generated summaries\.

In this work, we present a grounded and normalized framework for evaluating relation\-level hallucination\. We refine relation extraction using linguistically informed mechanisms and introduce a normalized formulation of the Relation Hallucination Index \(RHI\) to enable stable comparison across models and datasets\. Our approach provides a structured and interpretable mechanism for analyzing relational fidelity and coherence\[[20](https://arxiv.org/html/2608.08180#bib.bib20)\]in abstractive summarization systems\.

## IIBackground and Problem Formulation

### II\-ARelation\-Level Hallucination

Relation\-level hallucination arises when a summarization system preserves surface entities from the source document, but incorrectly constructs or alters the relationships among them\. Modern transformer\-based models and large language models generate summaries by predicting tokens conditioned on contextual representations rather than explicitly modeling structured relational dependencies\. As a result, these systems may inadvertently introduce fabricated interactions, misattribute actions, or exaggerate causal links between entities\. Unlike simple factual omissions, relational distortions modify the semantic structure of the original content and may lead to misleading interpretations\. Detecting such inconsistencies, therefore, requires a structured comparison of subject–predicate–object relations across source texts, references, and generated summaries\. So, relation\-aware extraction and evaluation frameworks are essential to quantify these structural deviations and assess summary faithfulness beyond lexical similarity\.

### II\-BLimitations of Existing Metrics

Widely adopted evaluation measures such as ROUGE primarily rely on n\-gram overlap, which captures surface similarity but does not reflect relational correctness\. Embedding\-based metrics, including BERT\-derived similarity scores, assess semantic proximity yet lack explicit modeling of structured entity interactions\. Entity\-centric metrics improve entity coverage evaluation but remain insensitive to incorrect predicate assignments or distorted connections between entities\. Consequently, existing approaches often fail to identify relation\-level inconsistencies, motivating the need for structured and normalized relational evaluation mechanisms\.

## IIIRelated Work

### III\-AEvaluation Metrics for Summarization

Automatic evaluation of summarization systems has historically relied on lexical overlap measures, most prominently ROUGE\[[4](https://arxiv.org/html/2608.08180#bib.bib4)\]\. These metrics quantify n\-gram correspondence between system outputs and reference summaries and remain widely adopted for benchmarking informativeness\. However, lexical similarity does not guarantee factual correctness, and high overlap scores may still accompany structurally distorted content\. As neural generation models became more expressive, limitations of purely surface\-based metrics became increasingly apparent, motivating research into faithfulness\-oriented evaluation strategies\.

### III\-BApproaches to Mitigating Hallucination

To improve the reliability of abstractive summaries, several modeling strategies have been proposed\. Entity\-aware decoding mechanisms incorporate named entity signals during generation to encourage preservation of salient source entities\[[15](https://arxiv.org/html/2608.08180#bib.bib15)\]\. Knowledge\-grounded frameworks further attempt to constrain generation by referencing structured repositories such as knowledge graphs\[[16](https://arxiv.org/html/2608.08180#bib.bib16)\]\. Other approaches employ adversarial training, where a discriminator guides the generator toward producing outputs that are not only fluent but also factually aligned with source content\[[7](https://arxiv.org/html/2608.08180#bib.bib7)\]\. While these methods enhance generation quality, they focus primarily on model training rather than post\-hoc structural evaluation\.

### III\-CEntity\-Level Faithfulness Metrics

Beyond generation strategies, evaluation metrics specifically targeting hallucination have been introduced\. Entity\-level measures, such as Entity F1\[[17](https://arxiv.org/html/2608.08180#bib.bib17)\], compare named entities across source and generated summaries to quantify factual preservation\. Subsequent work expanded this perspective by categorizing hallucinations into subject, object, and relation dimensions\[[19](https://arxiv.org/html/2608.08180#bib.bib19)\]\. Entity\-centric indices, including the Entity Hallucination Index \(EHI\)\[[5](https://arxiv.org/html/2608.08180#bib.bib5)\], provide finer\-grained insight into entity consistency\. Nevertheless, entity\-level comparison does not guarantee relational correctness\. A summary may preserve correct entities while misassigning roles, altering predicates, or fabricating interactions between otherwise accurate mentions\.

### III\-DStructured and Knowledge\-Based Evaluation

Structured evaluation methods model textual content as relational triples or knowledge graph representations\. Benchmarks such as Text2KG frameworks\[[18](https://arxiv.org/html/2608.08180#bib.bib18)\]evaluate extraction quality under predefined ontologies\. While effective for structured prediction tasks, such approaches typically require domain\-specific schemas or world knowledge alignment, limiting their applicability to general summarization evaluation\. Moreover, many existing relation\-based pipelines rely on basic extraction heuristics that are sensitive to syntactic variation and duplication artifacts, potentially affecting hallucination measurements\.

### III\-EGap and Motivation

Existing research demonstrates substantial progress in entity preservation and semantic similarity evaluation; however, systematic quantification of relation\-level hallucination without reliance on external ontologies remains underexplored\. In particular, prior methods lack linguistically grounded extraction refinements and normalized scoring mechanisms that ensure stable cross\-dataset comparison\. The present work addresses these gaps by integrating dependency\-aware relation extraction, refined hallucination decomposition, normalized RHI computation, and coherence\-aware relational continuity analysis within a unified evaluation framework\.

## IVMethodology

The proposed framework consists of four major components: dataset preparation, summary generation, relation extraction, and relation\-level hallucination evaluation\. The overall pipeline constructs structured relational representations from input, reference, and generated summaries, and evaluates relational faithfulness using a normalized Relation Hallucination Index\.

### IV\-ADataset Selection and Preprocessing

#### IV\-A1Dataset Selection

To evaluate relation\-level hallucination across varying abstraction styles, we employ three widely used summarization benchmarks covering complementary news domains\. The experimental dataset consists ofXSUM \(200 instances\),XLSUM \(200 instances\), and theCNN/DailyMail SumEval subset \(400 instances\), resulting in a total of800 evaluation samples\.

XSUM is selected for its highly abstractive single\-sentence summaries, which frequently induce factual distortions\. XLSUM introduces stylistic and linguistic diversity through cross\-domain news reporting, enabling robustness assessment under varied writing structures\. The CNN/DailyMail SumEval dataset is incorporated due to its established use in factual consistency evaluation \(e\.g\., SummaC\), providing longer multi\-sentence summaries with stronger extractive grounding\.

This combination enables balanced evaluation across extreme abstraction, moderate abstraction, and factual consistency\-oriented summarization settings, improving generalizability of relation hallucination analysis\.

#### IV\-A2Dataset Preparation

All documents undergo normalization, removal of noisy symbols, and sentence\-level segmentation prior to processing\. Inputs exceeding transformer token limits are truncated while preserving contextual coherence\. The cleaned texts are subsequently passed to the relation extraction pipeline for Subject–Verb–Object \(SVO\) tuple generation, as illustrated in Fig\.[1](https://arxiv.org/html/2608.08180#S4.F1)\.

### IV\-BSummary Generation

#### IV\-B1Model Selection

We evaluate four representative transformer\-based abstractive summarization models, namelyBART\-large\-CNN,PEGASUS,T5\-large, andGPT\-3\.5, covering diverse pretraining objectives and abstraction behaviors\. The human reference summary is additionally considered as a pseudo\-model \(RefSum\) to establish an upper\-bound benchmark for relational consistency\.

![Refer to caption](https://arxiv.org/html/2608.08180v1/design_flow_rhi.drawio.png)Figure 1:Process flow diagram illustrating dataset preparation, summary generation, relation extraction, and metric evaluation pipeline\.
#### IV\-B2Model Output Generation

Each model generates zero\-shot summaries for the selected dataset\. The generated summaries, along with input documents and reference summaries, are processed to extract structured relation triples\. These triples form the basis for computing relational overlap and hallucination factors\.

### IV\-CRelation Extraction Framework

We propose a dependency\-aware relation extraction framework designed to enhance structural robustness and reduce spurious tuple generation\. Relations are represented as normalized subject–verb–object triples extracted using syntactic dependency parsing\.

The overall relation extraction is formalized in Algorithm[1](https://arxiv.org/html/2608.08180#alg1)\. The verb\-centric extraction procedure, detailed in Algorithm[2](https://arxiv.org/html/2608.08180#alg2), identifies predicate\-driven relational structures while incorporating multiple linguistic constraints\. Nominal fallback extraction for prepositional noun patterns is described in Algorithm[3](https://arxiv.org/html/2608.08180#alg3), and normalization together with subject grounding mechanisms are specified in Algorithm[4](https://arxiv.org/html/2608.08180#alg4)\.

The framework integrates several linguistic refinements to improve structural stability\. Lemmatization\-based normalization reduces morphological variation across predicates and arguments\. To mitigate underspecified subject ambiguity, named entity grounded subject resolution replaces generic subjects with entity\-aligned mentions when available\. Passive constructions are addressed through agent recovery to restore implicit actors, while negation\-aware verb modeling preserves relational polarity\. Reporting verbs are filtered to exclude meta\-discourse structures that do not contribute to factual content\. Furthermore, nominal fallback extraction captures noun–preposition–object configurations, and clausal propagation enables relation recovery from embedded predicate structures\. Finally, systematic deduplication prevents redundant triples from inflating hallucination measurements\.

These refinements collectively improve structural consistency of extracted relations and reduce artificial inflation of hallucination factors\. The resulting triples serve as structured inputs for computing the refined evaluation metrics, including EF1 and the normalized Relation Hallucination Index \(RHI\), described in subsequent sections\.

![Refer to caption](https://arxiv.org/html/2608.08180v1/Venn_diagram.png)Figure 2:Venn Diagram representing the six factors to identify the relation hallucinationAlgorithm 1Grounded Relation Extraction Algorithm1:functionExtractRelationsWithGrounding\(

t​e​x​ttext\)

2:

R​e​l​a​t​i​o​n​s←∅Relations\\leftarrow\\emptyset
3:

t​e​x​t←C​l​e​a​n​T​e​x​t​\(t​e​x​t\)text\\leftarrow CleanText\(text\)
4:

C​h​u​n​k​s←S​e​n​t​e​n​c​e​C​h​u​n​k​s​\(t​e​x​t\)Chunks\\leftarrow SentenceChunks\(text\)
5:foreach

c​h​u​n​k∈C​h​u​n​k​schunk\\in Chunksdo

6:Parse

c​h​u​n​kchunkusing dependency parser

7:foreach

s​e​n​t​e​n​c​esentencein

c​h​u​n​kchunkdo

8:

R​e​l​a​t​i​o​n​s←R​e​l​a​t​i​o​n​s∪Relations\\leftarrow Relations\\cup
9:ExtractVerbRelations\(

s​e​n​t​e​n​c​esentence\)

10:

R​e​l​a​t​i​o​n​s←R​e​l​a​t​i​o​n​s∪Relations\\leftarrow Relations\\cup
11:ExtractNominalRelations\(

s​e​n​t​e​n​c​esentence\)

12:endfor

13:endfor

14:Remove duplicate triples

15:return

R​e​l​a​t​i​o​n​sRelations
16:endfunction

Algorithm 2Verb\-Based Relation Extraction1:functionExtractVerbRelations\(

s​e​n​t​e​n​c​esentence\)

2:

R←∅R\\leftarrow\\emptyset
3:foreach

t​o​k​e​ntokenin

s​e​n​t​e​n​c​esentencedo

4:if

t​o​k​e​n\.p​o​s≠token\.pos\\neqVERBthen

5:continue

6:endif

7:

v←L​e​m​m​a​\(t​o​k​e​n\)v\\leftarrow Lemma\(token\)
8:if

v∈R​e​p​o​r​t​i​n​g​V​e​r​b​sv\\in ReportingVerbsthen

9:continue

10:endif

11:iftoken has negationthen

12:

vf​i​n​a​l←v\_\{final\}\\leftarrow“not\_”

\+\+vv
13:else

14:

vf​i​n​a​l←vv\_\{final\}\\leftarrow v
15:endif

16:Identify

S​u​b​j​e​c​t​sSubjectsfrom \{nsubj,nsubjpass\}

17:Identify

O​b​j​e​c​t​sObjectsfrom \{dobj,obj,attr\}⊳\\trianglerightPassive Agent Recovery

18:foreach agent dependencydo

19:Recover subject

20:endfor

21:foreach

s∈S​u​b​j​e​c​t​ss\\in Subjectsdo

22:if

s\.p​o​s∉\{N​O​U​N,P​R​O​P​N\}s\.pos\\notin\\\{NOUN,PROPN\\\}then

23:continue

24:endif

25:

sn​o​r​m←s\_\{norm\}\\leftarrowGroundSubject\(

s,s​e​n​t​e​n​c​es,sentence\)

26:if

sn​o​r​m=N​U​L​Ls\_\{norm\}=NULLthen

27:continue

28:endif

29:foreach

o∈O​b​j​e​c​t​so\\in Objectsdo

30:

on​o​r​m←o\_\{norm\}\\leftarrowNormalize\(

oo\)

31:if

on​o​r​m≠N​U​L​Lo\_\{norm\}\\neq NULLthen

32:Add

\(sn​o​r​m,vf​i​n​a​l,on​o​r​m\)\(s\_\{norm\},v\_\{final\},o\_\{norm\}\)to

RR
33:endif

34:endfor⊳\\trianglerightClausal Propagation

35:foreach child

ccwith dep

∈\{x​c​o​m​p,c​c​o​m​p\}\\in\\\{xcomp,ccomp\\\}do

36:Extract inner verb and object

37:Add propagated relation

38:endfor⊳\\trianglerightVerb Conjunction Handling

39:foreach conjunct verbdo

40:Add conjunction relation

41:endfor

42:endfor

43:endfor

44:return

RR
45:endfunction

Algorithm 3Nominal Relation Fallback1:functionExtractNominalRelations\(

s​e​n​t​e​n​c​esentence\)

2:

R←∅R\\leftarrow\\emptyset
3:foreach noun

nnin

s​e​n​t​e​n​c​esentencedo

4:foreach child with dep = prepdo

5:foreach grandchild with dep = pobjdo

6:

s←s\\leftarrowNormalize\(

nn\)

7:

v←L​e​m​m​a​\(p​r​e​p\)v\\leftarrow Lemma\(prep\)
8:

o←o\\leftarrowNormalize\(

p​o​b​jpobj\)

9:if

s≠N​U​L​Ls\\neq NULLand

o≠N​U​L​Lo\\neq NULLthen

10:Add

\(s,v,o\)\(s,v,o\)to

RR
11:endif

12:endfor

13:endfor

14:endfor

15:return

RR
16:endfunction

Algorithm 4Normalization and Subject Grounding1:functionNormalize\(

t​o​k​e​ntoken\)

2:if

t​o​k​e​ntokenis stopword or punctuationthen

3:returnNULL

4:endif

5:if

t​o​k​e​n\.p​o​s∉\{N​O​U​N,P​R​O​P​N,V​E​R​B,A​D​J\}token\.pos\\notin\\\{NOUN,PROPN,VERB,ADJ\\\}then

6:returnNULL

7:endif

8:returnlowercase lemma of

t​o​k​e​ntoken
9:endfunction

10:functionGroundSubject\(

t​o​k​e​n,s​e​n​t​e​n​c​etoken,sentence\)

11:if

t​o​k​e​ntokennot in generic subject listthen

12:returnNormalize\(

t​o​k​e​ntoken\)

13:endif

14:foreach named entity in

s​e​n​t​e​n​c​esentencedo

15:if

e​n​t​i​t​y\.l​a​b​e​l∈\{P​E​R​S​O​N,O​R​G,G​P​E,N​O​R​P\}entity\.label\\in\\\{PERSON,ORG,GPE,NORP\\\}then

16:returnlowercase entity text

17:endif

18:endfor

19:returnNormalize\(

t​o​k​e​ntoken\)

20:endfunction

### IV\-DEvaluation Metrics

To comprehensively assess factual and relational faithfulness, we employ lexical, semantic, and relation\-aware evaluation metrics\. Conventional measures includingROUGE\[[4](https://arxiv.org/html/2608.08180#bib.bib4)\]andSummaC\[[21](https://arxiv.org/html/2608.08180#bib.bib21)\]are used as comparative baselines, while relation\-oriented metrics such asEF1and the proposedRelation Hallucination Index \(RHI\)quantify structured factual consistency\. The Entity Hallucination Index \(EHI\)\[[5](https://arxiv.org/html/2608.08180#bib.bib5)\]is additionally considered to contrast entity\-level behavior with relation\-level evaluation\.

#### IV\-D1Lexical and Consistency Metrics

ROUGE evaluates lexical overlap between reference \(RR\) and generated summary \(SS\):

ROUGE\-N=∑g​r​a​mn∈RCountm​a​t​c​h​\(g​r​a​mn\)∑g​r​a​mn∈RCount​\(g​r​a​mn\)\\text\{ROUGE\-N\}=\\frac\{\\sum\_\{gram\_\{n\}\\in R\}\\text\{Count\}\_\{match\}\(gram\_\{n\}\)\}\{\\sum\_\{gram\_\{n\}\\in R\}\\text\{Count\}\(gram\_\{n\}\)\}\(1\)
Relational alignment is summarized using Extractive\-Faithfulness F1 \(EF1\)\[[22](https://arxiv.org/html/2608.08180#bib.bib22)\]:

E​F​1=2×P​r​e​c​i​s​i​o​n×R​e​c​a​l​lP​r​e​c​i​s​i​o​n\+R​e​c​a​l​lEF1=\\frac\{2\\times Precision\\times Recall\}\{Precision\+Recall\}\(2\)
P​r​e​c​i​s​i​o​n=\|I∩G\|\|G\|,R​e​c​a​l​l=\|R∩G\|\|R\|Precision=\\frac\{\|I\\cap G\|\}\{\|G\|\},\\quad Recall=\\frac\{\|R\\cap G\|\}\{\|R\|\}
SummaC measures document–summary consistency using entailment aggregation\[[21](https://arxiv.org/html/2608.08180#bib.bib21)\]:

SummaC​\(D,S\)=1\|S\|​∑si∈Smaxdj∈D⁡Entail​\(dj,si\)\\text\{SummaC\}\(D,S\)=\\frac\{1\}\{\|S\|\}\\sum\_\{s\_\{i\}\\in S\}\\max\_\{d\_\{j\}\\in D\}\\text\{Entail\}\(d\_\{j\},s\_\{i\}\)\(3\)
Coherence is computed as the average semantic similarity between adjacent sentences using sentence\-level embeddings:

Coherence​\(S\)=1N−1​∑i=1N−1cos⁡\(𝐞i,𝐞i\+1\)\\text\{Coherence\}\(S\)=\\frac\{1\}\{N\-1\}\\sum\_\{i=1\}^\{N\-1\}\\cos\\left\(\\mathbf\{e\}\_\{i\},\\mathbf\{e\}\_\{i\+1\}\\right\)\(4\)
whereS=\{s1,s2,…,sN\}S=\\\{s\_\{1\},s\_\{2\},\.\.\.,s\_\{N\}\\\}denotes the generated summary consisting ofNNsentences, and𝐞i\\mathbf\{e\}\_\{i\}represents the sentence embedding of sentencesis\_\{i\}\. Higher values indicate stronger semantic continuity between consecutive sentences\[[20](https://arxiv.org/html/2608.08180#bib.bib20)\]\.

#### IV\-D2Entity Hallucination Index

Entity\-level factual reliability is measured using EHI\[[5](https://arxiv.org/html/2608.08180#bib.bib5)\]:

E​H​I=eP​H\+eE​FeP​H\+eE​F\+eN​H\+eO​F\+eL​FEHI=\\frac\{e^\{PH\}\+e^\{EF\}\}\{e^\{PH\}\+e^\{EF\}\+e^\{NH\}\+e^\{OF\}\+e^\{LF\}\}\(5\)

#### IV\-D3Relation Hallucination Decomposition

LetII,RR, andGGdenote relation triples extracted from input, reference, and generated summaries\.

E​F=3​\|I∩R∩G\|\|I\|\+\|R\|\+\|G\|EF=\\frac\{3\|I\\cap R\\cap G\|\}\{\|I\|\+\|R\|\+\|G\|\}\(6\)
P​H=2​\|R∩G\|\|R\|\+\|G\|PH=\\frac\{2\|R\\cap G\|\}\{\|R\|\+\|G\|\}\(7\)
O​F=2​\(\|I∩G\|−\|I∩R∩G\|\)\|I\|\+\|G\|OF=\\frac\{2\(\|I\\cap G\|\-\|I\\cap R\\cap G\|\)\}\{\|I\|\+\|G\|\}\(8\)
N​H=\|\|G\|−\(\|R∩G\|\+\|I∩G\|−\|I∩R∩G\|\)\|\|G\|NH=\\frac\{\\left\|\|G\|\-\(\|R\\cap G\|\+\|I\\cap G\|\-\|I\\cap R\\cap G\|\)\\right\|\}\{\|G\|\}\(9\)
L​F=\|R\|−\(\|I∩R\|−\|I∩R∩G\|\)\|R\|\+\|G\|LF=\\frac\{\|R\|\-\(\|I\\cap R\|\-\|I\\cap R\\cap G\|\)\}\{\|R\|\+\|G\|\}\(10\)
L​H=\|I\|−\|I∩G\|\|I\|\+\|G\|LH=\\frac\{\|I\|\-\|I\\cap G\|\}\{\|I\|\+\|G\|\}\(11\)

#### IV\-D4Relation Hallucination Index

The proposed RHI aggregates positive and hallucinated relational behaviors:

R​H​I=1\+E​F\+P​H2−O​F\+N​H\+L​H\+L​F4RHI=1\+\\frac\{EF\+PH\}\{2\}\-\\frac\{OF\+NH\+LH\+LF\}\{4\}\(12\)
R​H​In​o​r​m=R​H​I−R​H​Im​i​nR​H​Im​a​x−R​H​Im​i​nRHI\_\{norm\}=\\frac\{RHI\-RHI\_\{min\}\}\{RHI\_\{max\}\-RHI\_\{min\}\}\(13\)

## VResults

We evaluate relational faithfulness using EF1, EHI, ROUGE, SummaC, coherence, and the proposed Relation Hallucination Index \(RHI\)\. Results are reported separately for abstractive news datasets \(XSUM\+XLSUM\) and the CNN/DailyMail SumEval benchmark to analyze cross\-domain consistency\.

### V\-AAverage Performance on XSUM \+ XLSUM

TABLE I:Average Metrics on XSUM \+ XLSUMTable[I](https://arxiv.org/html/2608.08180#S5.T1)shows that lexical and entailment\-based metrics favor PEGASUS due to stronger surface alignment and discourse coherence\. However, RHI assigns the highest score to BART, indicating improved preservation of source relations despite moderate ROUGE values\. This divergence highlights RHI’s ability to detect relational correctness beyond token overlap or semantic similarity\. GPT\-3\.5 achieves higher EF1 but lower RHI, suggesting increased relation insertion variability typical of generative models under highly abstractive settings\.

### V\-BPerformance on CNN/DailyMail SumEval

TABLE II:Average Metrics on CNN/DailyMail SumEvalAcross the SumEval benchmark \(Table[II](https://arxiv.org/html/2608.08180#S5.T2)\), BART consistently achieves the highest RHI and EF1, demonstrating stable relational grounding on longer documents\. DistilBART records near\-zero ROUGE and EF1 scores, indicating failure in effective summary generation rather than metric bias\. Despite moderate SummaC values, its reduced RHI confirms substantial loss of relational structure, validating the sensitivity of RHI to generation collapse scenarios\.

### V\-CStatistical Validation

Statistical analysis across both the XSUM\+XLSUM and CNN/DailyMail SumEval datasets confirms the discriminative strength of the proposed RHI metric\. Paired t\-test results show statistically significant performance differences among summarization models \(p<0\.01p<0\.01\), indicating that RHI consistently distinguishes summaries with well\-preserved relations from those containing higher levels of hallucination\. In contrast to ROUGE, EF1, EHI, and SummaC, which often yield comparable scores across models, RHI provides clearer separation and stable ranking across datasets, demonstrating its effectiveness for relation\-level factual evaluation\.

### V\-DBaseline vs Updated RHI Framework

TABLE III:Baseline vs Updated RHI \(XSUM\+XLSUM\)Tables[III](https://arxiv.org/html/2608.08180#S5.T3)and[IV](https://arxiv.org/html/2608.08180#S5.T4)show consistent RHI improvements across all models, where refinement contributes the major performance gain \(Δr​e​f\\Delta\_\{ref\}\) and normalization provides additional stabilization \(Δn​o​r​m\\Delta\_\{norm\}\)\. The cumulative improvements match the overallΔ\\Deltaobserved in Table[III](https://arxiv.org/html/2608.08180#S5.T3), confirming the effectiveness of the proposed framework design\.

TABLE IV:Ablation Study of RHI Components \(XSUM\+XLSUM\)
### V\-ECross\-Dataset Hallucination Behaviour

TABLE V:Cross\-Dataset Hallucination Factors and Normalized RHIAcross both datasets, hallucination decomposition reveals consistent relational trends supporting RHI evaluation\. Models with balanced Extractiveness \(EF\) and Positive Hallucination \(PH\) while maintaining reduced Over\-Focus \(OF\) achieve higher normalized RHI scores, indicating stable relation preservation\. Improvements observed on the CNN/DailyMail benchmark demonstrate that the proposed framework remains robust under longer and less abstractive summaries\.

Notably, generation failure in DistilBART results in near\-zero relational factors, leading to reduced RHInormdespite high coherence, confirming that RHI evaluates factual grounding rather than fluency\. The consistent separation of models across datasets validates that aggregated hallucination factors collectively strengthen RHI’s sensitivity and enable reliable cross\-domain hallucination assessment\.

### V\-FDistributional Analysis of RHI

![Refer to caption](https://arxiv.org/html/2608.08180v1/Dataset_xlsum_xsum__CDF_rhi_line_plot_400_records.png)Figure 3:Cumulative Distribution of RHI across models for XLSum/XSum Dataset![Refer to caption](https://arxiv.org/html/2608.08180v1/CDF_400_records_SUM_EVAL_CNN_Dataset_Line_Plot.png)Figure 4:Cumulative Distribution of RHI across models for CNN/Daily Mail DatasetFigures[3](https://arxiv.org/html/2608.08180#S5.F3)and[4](https://arxiv.org/html/2608.08180#S5.F4)present the cumulative distribution of RHI scores across the abstractive \(XSUM\+XLSUM\) and long\-document \(CNN/DailyMail\) evaluation settings\. In both datasets, BART and PEGASUS exhibit smoother right\-shifted distributions, indicating consistent preservation of source relations across a large proportion of summaries\. T5 shows moderate dispersion, reflecting variable relational grounding under different contextual conditions\. In contrast, DistilBART demonstrates an earlier saturation and compressed distribution in Fig\.[4](https://arxiv.org/html/2608.08180#S5.F4), revealing reduced relational coverage and confirming performance degradation observed in quantitative evaluation\. The consistent ordering of model curves across Fig\.[3](https://arxiv.org/html/2608.08180#S5.F3)and Fig\.[4](https://arxiv.org/html/2608.08180#S5.F4)further indicates that RHI maintains stable discriminative behaviour independent of dataset abstraction level or document length\.

### V\-GDiscussion

The experimental evaluation demonstrates that analysing summaries through relational structure provides insights that are not reflected by overlap\- or entailment\-oriented metrics\. Across both evaluation settings, models exhibiting comparable lexical quality show noticeable differences when assessed using relation consistency, indicating that factual reliability depends strongly on preserved entity–event interactions rather than surface similarity\. The grounded extraction strategy enables stable identification of meaningful relations, reducing sensitivity to stylistic variation across datasets\.

Furthermore, normalized RHI scores maintain consistent model ordering under both highly abstractive and long\-document summarization scenarios\. The observed distributional separation confirms that relation decomposition effectively exposes hallucination patterns arising from omission, distortion, or unsupported relation generation\. These findings suggest that relation\-aware evaluation offers a practical mechanism for examining factual behaviour at structural granularity while remaining adaptable across model architectures and dataset characteristics\.

## VIConclusion and Future Work

This paper introduced a grounded framework for assessing relation\-level hallucination in abstractive summarization through the proposed Relation Hallucination Index \(RHI\)\. Experimental results across diverse datasets demonstrate that the proposed formulation provides stable and discriminative evaluation of factual consistency, enabling clearer differentiation between relation\-preserving and hallucination\-prone summarization systems\. The study shows that modelling factuality at the level of entity–event interactions offers improved diagnostic capability compared to traditional evaluation measures focused primarily on lexical or semantic similarity\.

Future research will investigate extending the framework toward document\-level reasoning by incorporating cross\-sentence relation modeling and richer contextual representations\. Another promising direction involves leveraging RHI\-driven feedback during model optimization to encourage generation mechanisms that prioritize factual grounding alongside linguistic fluency\.

## References

- \[1\]P\. Katwe, R\. C\. Balabantaray, and K\. P\. Vittala, “Evaluating Relation Hallucination in Text Summarization: An Introduction to the Relation Hallucination Index,” in*Proc\. 16th Annu\. Meeting Forum Inf\. Retrieval Eval\. \(FIRE\)*, 2024, pp\. 88–94, doi: 10\.1145/3734947\.3734958\.
- \[2\]Z\. Ji*et al\.*, “Survey of Hallucination in Natural Language Generation,”*ACM Comput\. Surveys*, vol\. 55, no\. 12, pp\. 1–38, 2023, doi: 10\.1145/3571730\.
- \[3\]S\. Qi, L\. Gui, Y\. He, and Z\. Yuan, “A Survey of Automatic Hallucination Evaluation on Natural Language Generation,”*arXiv preprint arXiv:2404\.12041*, 2025\.
- \[4\]C\.\-Y\. Lin, “ROUGE: A Package for Automatic Evaluation of Summaries,” in*Text Summarization Branches Out*, Barcelona, Spain, 2004, pp\. 74–81\.
- \[5\]K\. Praveenkumar, R\. C\. Balabantaray, and K\. P\. Vittala, “Entity Hallucination Index in Abstractive Summarization—A Metric,” in*Proc\. 2023 Int\. Conf\. Commun\., Circuits, and Systems \(IC3S\)*, 2023, pp\. 1–5\.
- \[6\]P\. Qi*et al\.*, “Stanza: A Python Natural Language Processing Toolkit for Many Human Languages,”*arXiv preprint arXiv:2003\.07082*, 2020\.
- \[7\]W\. Wu*et al\.*, “Precisely the Point: Adversarial Augmentations for Faithful and Informative Text Generation,”*arXiv preprint arXiv:2210\.12367*, 2022\.
- \[8\]M\. Lewis*et al\.*, “BART: Denoising Sequence\-to\-Sequence Pre\-training for Natural Language Generation, Translation, and Comprehension,” in*Proc\. ACL*, 2020\.
- \[9\]J\. Zhang*et al\.*, “PEGASUS: Pre\-training with Extracted Gap\-Sentences for Abstractive Summarization,” in*Proc\. ICML*, 2020\.
- \[10\]C\. Raffel*et al\.*, “Exploring the Limits of Transfer Learning with a Unified Text\-to\-Text Transformer,”*J\. Mach\. Learn\. Res\.*, vol\. 21, no\. 140, pp\. 1–67, 2020\.
- \[11\]T\. B\. Brown*et al\.*, “Language Models are Few\-Shot Learners,” in*Proc\. NeurIPS*, 2020\.
- \[12\]M\.\-C\. de Marneffe and C\. D\. Manning, “The Stanford Typed Dependencies Representation,” in*Proc\. Cross\-Framework and Cross\-Domain Parser Evaluation*, 2008, pp\. 1–8\.
- \[13\]D\. Zeng, K\. Liu, S\. Lai, G\. Zhou, and J\. Zhao, “Relation Classification via Convolutional Deep Neural Network,” in*Proc\. COLING*, 2014\.
- \[14\]L\. He, K\. Lee, M\. Lewis, and L\. Zettlemoyer, “Deep Semantic Role Labeling: What Works and What’s Next,” in*Proc\. ACL*, 2017\.
- \[15\]H\. Zhou, W\. Ren, G\. Liu, B\. Su, and W\. Lu, “Entity\-Aware Abstractive Multi\-Document Summarization,” in*Findings of the Association for Computational Linguistics: ACL*, 2021, pp\. 351–362, doi: 10\.18653/v1/2021\.findings\-acl\.30\.
- \[16\]W\. Chen, Y\. Su, X\. Yan, and W\. Y\. Wang, “KGPT: Knowledge\-Grounded Pre\-Training for Data\-to\-Text Generation,”*arXiv preprint arXiv:2010\.02307*, 2020\.
- \[17\]G\. Lample, M\. Ballesteros, S\. Subramanian, K\. Kawakami, and C\. Dyer, “Neural Architectures for Named Entity Recognition,” in*Proc\. NAACL\-HLT*, San Diego, CA, USA, 2016, pp\. 260–270, doi: 10\.18653/v1/N16\-1030\.
- \[18\]N\. Mihindukulasooriya, S\. Tiwari, C\. F\. Enguix, and K\. Lata, “Text2KGBench: A Benchmark for Ontology\-Driven Knowledge Graph Generation from Text,”*arXiv preprint arXiv:2308\.02357*, 2023\.
- \[19\]J\. Maynez, S\. Narayan, B\. Bohnet, and R\. McDonald, “On Faithfulness and Factuality in Abstractive Summarization,” in*Proc\. 58th Annu\. Meeting Assoc\. Comput\. Linguistics \(ACL\)*, 2020, pp\. 1906–1919\.
- \[20\]N\. Reimers and I\. Gurevych, “Sentence\-BERT: Sentence Embeddings using Siamese BERT\-Networks,” in*Proc\. EMNLP*, 2019\.
- \[21\]P\. Laban, T\. Schnabel, P\. N\. Bennett, and M\. A\. Hearst, “SummaC: Re\-Visiting NLI\-based Models for Inconsistency Detection in Summarization,”arXiv preprint arXiv:2111\.09525, 2021\.
- \[22\]M\. Olek, “About Evaluation of F1 Score for RECENT Relation Extraction System,”arXiv preprint arXiv:2305\.09410, 2023\.