KGCQual: An Interpretable Framework for Evaluating the Knowledge Graph Construction Quality from Text

# KGCQual: An Interpretable Framework for Evaluating the Knowledge Graph Construction Quality from Text
Source: [https://arxiv.org/html/2607.10212](https://arxiv.org/html/2607.10212)
11institutetext:1VIT Vellore, Vellore, India
11email:nipun\.misra2022@vitstudent\.ac\.in
2IIIT\-Delhi, Delhi, India
11email:\{aanchal21224, yogender21505, vikranth22570, manuj\}@iiitd\.ac\.in
3Indian Institute of Technology Palakkad, Kerala, India\.
11email:raghava@iitpkd\.ac\.inVikranth UdandaraoAanchal GuptaYogender KumarManuj MukherjeeRaghava Mutharaju

###### Abstract

Knowledge Graphs \(KGs\) are increasingly constructed through automated extraction pipelines; however, such systems often introduce spurious or incomplete triples, which degrade downstream performance\. Existing evaluation practices rely heavily on task\-specific metrics or small\-scale manual verification, offering limited insight into the structural and semantic fidelity of extracted graphs\. We propose a novel, interpretable metric for intrinsic KG quality assessment that measures how closely an automatically extracted graph approximates an “ideal” graph capturing the key noun phrases, predicate relations, and basic linguistic phenomena such as negation expressed in the source text\. Our framework integrates two complementary components: \(1\) an entity\-level assessment that evaluates completeness, resolution quality, and connectivity, and \(2\) a relation\-level assessment that judges predicate preservation and multiplicity using lexical similarity, dependency\-parse alignment, and light\-weight negation handling to ensure semantic faithfulness\. We evaluate our metric across multiple state\-of\-the\-art triple extraction systems and datasets, including WebNLG, TinyButMighty, and BenchIE, demonstrating that it reliably identifies omissions, redundancy, and structural deviations that existing metrics overlook\. Our work offers a scalable, model\-agnostic, and interpretable framework for comparing automated KG construction methods and provides a foundation for standardised evaluation\. We further validate the metric through an ablation study isolating noun and verb components, and a downstream evaluation showing that KGCQual scores correlate significantly with link prediction performance \(ρ=−0\.900\\rho=\-0\.900,p=0\.037p=0\.037\) on the same extracted KGs\. The code repository is available at[https://github\.com/kracr/kg\-quality\-metric](https://github.com/kracr/kg-quality-metric)\.

## 1Introduction

Knowledge Graphs \(KGs\) have become a foundational abstraction enabling structured representation of entities, relations, and facts in a form amenable to reasoning, querying, and integration across heterogeneous data sources\[[11](https://arxiv.org/html/2607.10212#bib.bib11),[18](https://arxiv.org/html/2607.10212#bib.bib14)\]\. A growing fraction of modern KGs spanning domains such as scientific literature, news, and enterprise documents are constructed through automated Information Extraction \(IE\) pipelines\. These systems transform natural language text into subject–predicate–object triples that are later serialized into RDF, linked to ontologies, or validated through SHACL constraints\.

However, despite their widespread adoption, the quality of triples produced by IE systems remains highly variable\. Existing KG quality frameworks \(e\.g\.,\[[18](https://arxiv.org/html/2607.10212#bib.bib14),[6](https://arxiv.org/html/2607.10212#bib.bib12)\]\) provide rich dimensions such as accuracy, completeness, and consistency, but*presuppose*that the underlying triples are already correct or human\-curated\. In contrast, for automatically extracted KGs, the fundamental bottleneck lies one level earlier: the lack of an*intrinsic, sentence\-level method*for assessing whether extracted triples faithfully preserve the semantics expressed in the source text\. Current evaluations rely predominantly on \(i\) downstream task performance, which is indirect and application\-specific, or \(ii\) manual annotation, which is costly and non\-scalable\. This creates a persistent gap between Semantic Web quality frameworks \(which expect high\-integrity RDF graphs\) and the practical realities of pipeline\-driven KG construction\. In particular, no existing metric jointly captures*structural*,*lexical*, and*semantic*fidelity of extracted triples with respect to their textual origin\. Yet such intrinsic valuation is critical: errors such as missing entities, merged noun phrases, incorrect relation multiplicity, or undetected negation propagate into downstream KGs and ultimately compromise reasoning, linking, and query results\.

To address this gap, we proposeKGCQual, a principled and interpretable framework for evaluating the quality of automatically constructed triples prior to their integration into a KG\. Our approach is grounded in the notion of an*ideal reference graph*for a sentence \- an abstract representation of all its salient entities and predicates\. While this ideal graph is not observed directly, we approximate it by combining part\-of\-speech cues, dependency relations, and lightweight semantic similarity methods\. KGCQual evaluates triple quality along two complementary dimensions:

1. 1\.Entity Quality \(Noun\-Level\): measuring completeness, granularity \(resolution\), and connectivity of entity references derived from the text\. This captures whether the extracted triples maintain the structure expected of RDF nodes corresponding to noun phrases\.
2. 2\.Relation Quality \(Verb\-Level\): assessing predicate accuracy, multiplicity, and semantic similarity, including cases where paraphrasing or morphological variation occurs\. This reflects whether the extracted predicates preserve the relational semantics intended in the source sentence\.

Together, these form a normalized, interpretable score indicating how well an IE tool preserves sentence\-level semantics before the triples are serialized into RDF or integrated into a broader KG\. To illustrate the challenge, consider the following sentence:

> “The Urchin Software Configuration Management system is a free and open source initiative to provide an enterprise\-scale solution to managing the configuration of many software applications running on many servers across an enterprise\.”

State\-of\-the\-art IE systems such as MinIE\[[9](https://arxiv.org/html/2607.10212#bib.bib20)\], OllIE\[[13](https://arxiv.org/html/2607.10212#bib.bib22)\], and Large Language Model \(LLM\)–based extractors often omit essential entities \(e\.g\.,“free, open\-source initiative”\), merge distinct noun phrases \(e\.g\.,“configuration of many software applications”\), or reduce verb multiplicity \(e\.g\., collapsing multiple“is”relations\)\. Figures[2](https://arxiv.org/html/2607.10212#S1.F2)–[4](https://arxiv.org/html/2607.10212#S1.F4)visualize these discrepancies\. Such errors degrade KG completeness and semantic fidelity if not detected early\.

KGCQual directly targets these shortcomings by providing a systematic way to quantify deviations from an ideal graph, enabling fair comparison of IE tools, LLMs, and extraction pipelines\. By grounding evaluation in both linguistic structures and Semantic Web quality principles, our framework bridges the gap between NLP\-driven extraction and KG\-centric validation\.

![Refer to caption](https://arxiv.org/html/2607.10212v1/minie.png)Figure 1:Triple graph extracted by MinIE
![Refer to caption](https://arxiv.org/html/2607.10212v1/ollieie.png)Figure 2:Triple graph extracted by OllIE

![Refer to caption](https://arxiv.org/html/2607.10212v1/gpt.png)Figure 3:Triple graph extracted by GPT based model
![Refer to caption](https://arxiv.org/html/2607.10212v1/ideal.png)Figure 4:Ideal triple graph derived using POS tagging

## 2Related Work

### 2\.1Knowledge Graph Quality Assessment

Ensuring the quality of Knowledge Graphs \(KGs\) is very important for downstream applications\. Foundational surveys\[[18](https://arxiv.org/html/2607.10212#bib.bib14),[11](https://arxiv.org/html/2607.10212#bib.bib11)\]outline core dimensions such as accuracy, completeness, consistency, and relevancy, and provide lifecycle\-oriented frameworks for evaluating KGs from creation to maintenance\. These approaches primarily operate*post hoc*on already constructed graphs, identifying conflicting triples, schema violations, outdated assertions, or structural anomalies through rule\-based checks, constraint validation \(e\.g\., SHACL\), or statistical consistency measures\[[6](https://arxiv.org/html/2607.10212#bib.bib12)\]\. Such methods presuppose that triples ingested into a KG are semantically reliable\. However, when triples originate from automatic extraction pipelines, the primary source of error lies*prior*to graph construction: the extraction stage itself\.

### 2\.2Triple\-Level Intrinsic Evaluation

Despite the centrality of triples as the atomic units of RDF graphs, intrinsic triple\-level evaluation remains underexplored\. Existing benchmarks such as WebNLG, ReFinED, and BenchIE\[[11](https://arxiv.org/html/2607.10212#bib.bib11),[6](https://arxiv.org/html/2607.10212#bib.bib12),[18](https://arxiv.org/html/2607.10212#bib.bib14)\]evaluate IE systems through manually curated gold triples or downstream tasks, but they do not offer a*sentence\-grounded, interpretable metric*that quantifies how well extracted triples preserve the fine\-grained semantic structure of the source text\. Sampling\-based accuracy estimation\[[17](https://arxiv.org/html/2607.10212#bib.bib7)\]helps assess large KGs, but is not tailored to decomposing extraction errors in terms of entity completeness, predicate fidelity, or structural correspondence to linguistic cues\.

In contrast, our work focuses on the earliest stage of KG construction by providing asentence\-level intrinsic metricthat characterizes how faithfully an IE system captures the entities and relations expressed in text\. This complements existing KG quality frameworks by addressing a stage they do not directly evaluate:*the correctness of triples before they become part of an RDF graph*\.

### 2\.3Information Extraction Quality and Semantic Fidelity

Research in Information Extraction has examined extraction errors such as missing entities, incorrect argument boundaries, and predicate omissions\. Systems such as MinIE and OllIE\[[9](https://arxiv.org/html/2607.10212#bib.bib20),[13](https://arxiv.org/html/2607.10212#bib.bib22)\]reduce syntactic complexity but often at the expense of semantic completeness\. LLM\-based extractors\[[15](https://arxiv.org/html/2607.10212#bib.bib24)\]improve fluency but may alter predicate forms or merge multi\-word noun phrases, reducing the semantic resolution required for high\-quality KG\.

Several works scrutinize semantic fidelity in greater depth\. Negation handling has been shown to be a major failure point: recent studies report that state\-of\-the\-art open IE systems incorrectly preserve negation in up to 40% of cases\[[7](https://arxiv.org/html/2607.10212#bib.bib27)\], while parse\-driven approaches achieve improved but still imperfect performance\[[4](https://arxiv.org/html/2607.10212#bib.bib28)\]\. However, these efforts treat negation as an isolated linguistic phenomenon rather than integrating it into a broader KG quality model\. Similarly, semantic similarity measures and paraphrase detection are often used heuristically in open IE, yet they are rarely incorporated into principled KG\-oriented evaluation metrics\.

### 2\.4Positioning of Our Approach

Existing work offers rich frameworks for*graph\-level*KG quality and localized techniques for*linguistic error detection*, but no unified metric that connects the two\. Our contribution fills this gap by integrating:

- •Structural fidelity: completeness, granularity, and connectivity of entity nodes,
- •Predicate fidelity: relation multiplicity, lexical/semantic preservation,
- •Negation fidelity: correctness of polarity propagation in triples,

Unlike prior approaches that evaluate IE systems indirectly through task performance or manually annotated datasets, our framework provides a*direct, intrinsic, and interpretable*measure of triple correctness with respect to the source sentence\. This positions our work as a bridge between linguistically grounded IE evaluation and Semantic Web quality assurance, addressing a critical but largely unexamined stage of the KG lifecycle\.

## 3Proposed Methodology

### 3\.1Motivation

Information Extraction \(IE\) systems—rule\-based, statistical, or LLM\-driven—serve as the first step in automatically constructing Knowledge Graphs \(KGs\) from natural language text\. However, different extractors often produce triples of varying granularity, completeness, and relational accuracy, leading to inconsistencies in the resulting KG\. Current KG quality frameworks assume that triples are already correct when ingested; yet in practice, a substantial proportion of KG errors originate*prior*to integration, during extraction\.

We therefore introduce a principled metric for*intrinsic*evaluation of triple extraction quality\. The metric compares an extractor’s output against an*ideal graph*derived directly from linguistic structure\. It decomposes triple quality into two fundamental dimensions: \(1\)entity fidelity\(noun phrases\), and \(2\)relation fidelity\(verbs, predicate multiplicity\)\. This reflects how well the extracted KG preserves the sentence’s informational structure before RDF serialization or ontology grounding\.

### 3\.2Ideal Graph Construction

For a given sentence, we construct an*ideal reference graph*GiG\_\{i\}whose nodes correspond to distinct noun phrases and whose edges correspond to verb phrases derived from dependency parsing\. The ideal graph encodes:

- •All noun phrases as distinct nodes,
- •All verb phrases \(including multiplicities\),
- •Polarity \(negation\) when expressed in the sentence,
- •A single connected component for each sentence\.

This graph serves as the target structure for quality comparison\. The IE system’s output induces a graphG1G\_\{1\}, which we evaluate againstGiG\_\{i\}using the metrics defined below\.

### 3\.3Noun Metric

The noun metricMNM\_\{N\}assesses whether the extracted KG preserves the expected set of entities with respect to:

1. 1\.Completeness– all noun phrases appear inG1G\_\{1\},
2. 2\.Resolution– each noun phrase is represented as a distinct node,
3. 3\.Connectivity– the extracted graph forms a single connected component\.

LetN​\(Gx\)N\(G\_\{x\}\)be the number of noun phrases,V​\(Gx\)V\(G\_\{x\}\)be the number of nodes, andC​\(Gx\)C\(G\_\{x\}\)be the number of connected components\.

#### 3\.3\.1Resolution

The ideal graph satisfiesV​\(Gi\)=N​\(Gi\)V\(G\_\{i\}\)=N\(G\_\{i\}\)\. Deviation from this equality reflects the merging of distinct noun phrases:

1−V​\(G1\)N​\(G1\)\.1\-\\frac\{V\(G\_\{1\}\)\}\{N\(G\_\{1\}\)\}\.

#### 3\.3\.2Negation\-Aware Matching

To handle polarity\-sensitive nouns, we incorporate negation markers into the string matching process\. We detect negation using theUniversal Dependencies\(UD\) syntactic framework, which annotates grammatical relations such as thenegedge linking a negation word \(e\.g\., “not”, “no”\) to its head noun or verb\. This allows us to determine whether an extracted noun is negated or not\.

Letλ=0\.3\\lambda=0\.3be a penalty constant andJWdenote the Jaro–Winkler similarity\. For each ideal nounnjn\_\{j\}:

τnj¬=\{JW​\(njext,njideal\),if their UD\-based negation status matches,λ⋅JW​\(njext,njideal\),otherwise\.\\tau\_\{n\_\{j\}\}^\{\\neg\}=\\begin\{cases\}\\text\{JW\}\(n\_\{j\}^\{\\text\{ext\}\},n\_\{j\}^\{\\text\{ideal\}\}\),&\\text\{if their UD\-based negation status matches\},\\\\ \\lambda\\cdot\\text\{JW\}\(n\_\{j\}^\{\\text\{ext\}\},n\_\{j\}^\{\\text\{ideal\}\}\),&\\text\{otherwise\}\.\\end\{cases\}λ\\lambdais the weight of the penalty for negations, determining how much emphasis is placed on polarity mismatch penalties\. For this study, we fixλ=0\.3\\lambda=0\.3to assign a reasonable penalty for negation errors without allowing them to overshadow the overall score\. This replaces strict matching with a graded penalty, ensuring that polarity violations \(e\.g\.,*“cat”*vs\.*“not cat”*\) are treated as entity\-level errors\.

#### 3\.3\.3Completeness

Missing noun phrases incur additive penalties:

N​\(Gi\)−N​\(G1\)N\(G\_\{i\}\)\-N\(G\_\{1\}\)

#### 3\.3\.4Connectivity

The ideal graph has a single component, so:

#### 3\.3\.5Combined Noun Score

Aggregating all deviations yields:

dN​\(G1,Gi\)=\(1−V​\(G1\)N​\(G1\)\)\+∑j=1N​\(Gi\)\(1−τnj\)\+\(C​\(G1\)−1\)d\_\{N\}\(G\_\{1\},G\_\{i\}\)=\\left\(1\-\\frac\{V\(G\_\{1\}\)\}\{N\(G\_\{1\}\)\}\\right\)\+\\sum\_\{j=1\}^\{N\(G\_\{i\}\)\}\(1\-\\tau\_\{n\_\{j\}\}\)\+\(C\(G\_\{1\}\)\-1\)
The range of this quantity is\[0,2​N​\(Gi\)\]\[0,2N\(G\_\{i\}\)\]; therefore, we normalize:

MN=dN​\(G1,Gi\)2​N​\(Gi\)M\_\{N\}=\\frac\{d\_\{N\}\(G\_\{1\},G\_\{i\}\)\}\{2N\(G\_\{i\}\)\}A value of0indicates perfect noun\-level fidelity to the ideal graph\.

### 3\.4Verb Metric

Relations \(verbs\) are central to KG semantics, encoding the predicate structure linking entities\. Errors typically arise in two forms:*loss of multiplicity*and*lexical drift*\. The verb metric captures both\.

Let:

V​b​\(Gx\)=number of predicate instances \(including multiplicities\)Vb\(G\_\{x\}\)=\\text\{number of predicate instances \(including multiplicities\)\}
#### 3\.4\.1Predicate Multiplicity

Dependency parsing provides reliable subject–verb–object structures and reveals when a verb governs multiple conjunctive objects \(e\.g\., likes cricket, football, and basketball\)\. Missing predicate instances are penalized as:

V​b​\(Gi\)−V​b​\(G1\)Vb\(G\_\{i\}\)\-Vb\(G\_\{1\}\)

#### 3\.4\.2Lexical and Semantic Similarity

We compute a similarity scoreτvj\\tau\_\{v\_\{j\}\}for each extracted predicate using:

τvj=max⁡\(JW​\(vext,videal\),cos⁡\(𝐞​\(vext\),𝐞​\(videal\)\)\),\\tau\_\{v\_\{j\}\}=\\max\\Big\(\\text\{JW\}\(v^\{\\text\{ext\}\},v^\{\\text\{ideal\}\}\),\\cos\(\\mathbf\{e\}\(v^\{\\text\{ext\}\}\),\\mathbf\{e\}\(v^\{\\text\{ideal\}\}\)\)\\Big\),where𝐞​\(⋅\)\\mathbf\{e\}\(\\cdot\)denotes contextual or static word embeddings\. Lemmatization is applied prior to similarity computation\.

#### 3\.4\.3Combined Verb Score

The unnormalized verb deviation is:

dV​\(G1,Gi\)=\(V​b​\(Gi\)−V​b​\(G1\)\)\+∑j=1V​b​\(Gi\)\(1−τvj\)d\_\{V\}\(G\_\{1\},G\_\{i\}\)=\\left\(Vb\(G\_\{i\}\)\-Vb\(G\_\{1\}\)\\right\)\+\\sum\_\{j=1\}^\{Vb\(G\_\{i\}\)\}\(1\-\\tau\_\{v\_\{j\}\}\)
The normalized verb metric is:

MV=dV​\(G1,Gi\)2​V​b​\(Gi\)M\_\{V\}=\\frac\{d\_\{V\}\(G\_\{1\},G\_\{i\}\)\}\{2Vb\(G\_\{i\}\)\}

### 3\.5Final Metric

The complete triple\-quality metric, KGCQual, combines entity and relation fidelity:

M=α​MN\+\(1−α\)​MVM=\\alpha M\_\{N\}\+\(1\-\\alpha\)M\_\{V\}whereα∈\[0,1\]\\alpha\\in\[0,1\]controls the relative weight of entity vs\. relation quality\. We setα\\alphato 0\.5 to indicate that both the components are equally important\. This produces a score in\[0,1\]\[0,1\], where0represents perfect fidelity to the ideal graph\. The metric is extractor\-agnostic and can be applied to any pipeline producing subject–predicate–object triples, enabling principled comparison of heterogeneous IE systems prior to KG integration\.

## 4Evaluation

This section evaluates the proposed metric across eight extraction systems – five classical OpenIE tools \(MinIE\[[9](https://arxiv.org/html/2607.10212#bib.bib20)\], OllIE\[[13](https://arxiv.org/html/2607.10212#bib.bib22)\], Stanford OpenIE 4\.5\.3/4\.5\.6\[[1](https://arxiv.org/html/2607.10212#bib.bib23)\], ClausIE\[[5](https://arxiv.org/html/2607.10212#bib.bib21)\]\) and three LLM\-based extractors \(Claude 3\.7\[[12](https://arxiv.org/html/2607.10212#bib.bib25)\], Gemini 2\.5 Pro\[[14](https://arxiv.org/html/2607.10212#bib.bib26)\], GPT\-4\.0\-mini\[[15](https://arxiv.org/html/2607.10212#bib.bib24)\]\)\. For all experiments, noun phrases are identified using the NLTK POS tagger\[[3](https://arxiv.org/html/2607.10212#bib.bib17)\]\. The evaluation assesses each system’s ability to preserve \(i\) noun\-phrase granularity and coverage, and \(ii\) predicate multiplicity and semantic accuracy, as captured by our noun and verb metrics\.

### 4\.1Datasets

We evaluate on three widely used benchmarks for triple extraction, chosen for their linguistic diversity and annotation quality:

- •WebNLG\[[8](https://arxiv.org/html/2607.10212#bib.bib9)\]: 643 English sentences paired with 5408 gold triples \(avg\. 8\.41 triples/sentence\), covering diverse DBpedia domains\.
- •TinyButMighty\[[16](https://arxiv.org/html/2607.10212#bib.bib10)\]: 720 complex and compound sentences yielding 3764 annotated triples \(avg\. 5\.28 triples/sentence\); designed to stress\-test OpenIE systems\.
- •BenchIE\[[10](https://arxiv.org/html/2607.10212#bib.bib16)\]: 300 naturally occurring sentences with 3307 gold triples \(avg\. 11\.02 triples/sentence\); a challenging testbed for fine\-grained relational extraction\.

These datasets jointly capture a broad range of syntactic constructions, entity configurations, and predicate forms—conditions necessary to evaluate the behaviour of extraction systems under realistic linguistic variability\.

### 4\.2Experimental Protocol

For each sentence, we \(i\) extract triples using each system, \(ii\) construct the corresponding KG, \(iii\) compute our noun and verb metrics against the ideal graph obtained via POS tagging and dependency parsing, and \(iv\) average scores over all sentences of each dataset\. Lower scores indicate higher extraction fidelity\.

To illustrate metric behaviour, we present two representative sentences, showing how the metric penalises missing noun\-phrase resolution, incomplete predicate multiplicity, and semantic deviation\.

#### 4\.2\.1Example 1

Input: *“The asteroid 1147 Stavropolis has an apoapsis of 418476000000\.0, a rotation period of 20378\.5, a periapsis of 260855000000\.0, and an orbital period of 1249\.6 days\.”*

ClausIE produces a single aggregated triple, collapsing four verb relations\. Since the POS tagger detects four predicate instances, the metric heavily penalises loss of multiplicity while rewarding perfect noun coverage\. Using the provided counts:

MN=0\.06,MV=0\.375,M=0\.2175\.M\_\{N\}=0\.06,\\qquad M\_\{V\}=0\.375,\\qquad M=0\.2175\.This illustrates the intended behaviour: noun extraction is faithful, but predicate structure is under\-specified\.

#### 4\.2\.2Example 2

Input: *“The Mason School of Business are the current tenants of Alan B Miller Hall, located at 101 Ukrop Way is in the United States and owned by the College of William and Mary\.”*

Stanford OpenIE \(v4\.5\.3\) misses one noun phrase and splits the graph into two components\. Verb extraction is partially correct but shows lexical mismatch\. Computing the metric gives:

MN=0\.0914,MV=0\.1744,M=0\.1329\.M\_\{N\}=0\.0914,\\qquad M\_\{V\}=0\.1744,\\qquad M=0\.1329\.The example demonstrates how the metric simultaneously captures noun loss, graph disconnectivity, and predicate variation\.

### 4\.3Quantitative Results

Tables[1](https://arxiv.org/html/2607.10212#S4.T1)–[3](https://arxiv.org/html/2607.10212#S4.T3)report the average noun metric, verb metric, and final combined score for all systems across all datasets\. A lower value denotes higher triple\-extraction fidelity\.

Table 1:Metric values for IE Tools and LLMs on WebNLG datasetTable 2:Metric values for IE Tools and LLMs on TinyButMighty datasetTable 3:Metric values for IE Tools and LLMs on BenchIE dataset
### 4\.4Gold\-Standard Triple Evaluation

To contextualize system performance, we run our metric on the gold\-standard triples in each dataset\. As shown in Table[4](https://arxiv.org/html/2607.10212#S4.T4), all gold sets achieve substantially lower scores than automated systems:

WebNLG:​0\.1585,TinyButMighty:​0\.2486,BenchIE:​0\.1424\.\\text\{WebNLG: \}0\.1585,\\quad\\text\{TinyButMighty: \}0\.2486,\\quad\\text\{BenchIE: \}0\.1424\.These values establish empirical lower bounds for our metric and confirm alignment with human\-annotated extractions\.

Table 4:Metric values for handwritten triples on the datasets
### 4\.5Interpretation via Likert\-Scale Categorization

For qualitative interpretability, we map final metric scores to a 5\-point Likert scale, anchored by the empirical thresholds observed in gold\-standard triples \(≤0\.25\\leq 0\.25\)\. This mapping provides a practical diagnostic for extraction quality\.

- •Grade 1 \(Excellent\):M≤0\.25M\\leq 0\.25
- •Grade 2–4 \(Intermediate\):0\.25<M<0\.450\.25<M<0\.45
- •Grade 5 \(Poor\):M≥0\.45M\\geq 0\.45

Across all datasets, ClausIE consistently falls into Grade 1, while MinIE occupies Grade 5\. LLM\-based extractors outperform traditional OpenIE systems on average but do not match gold\-standard quality\.

This analysis highlights that the metric discriminates reliably across systems with varying extraction behaviour and correlates well with qualitative judgments\.

## 5Discussion

This section examines the diagnostic and discriminative capabilities of the proposed metric by analyzing error patterns exhibited by two representative extraction systems: ClausIE and Stanford CoreNLP 4\.5\.6\. We select sentences yielding extremely low and high metric scores and conduct detailed case studies\. The goal is to demonstrate that the metric not only quantifies extraction quality but also functions as an interpretable tool for identifying systematic weaknesses in IE pipelines\.

### 5\.1Tool\-Level Behaviour

Across all datasets, ClausIE exhibits substantial variance in extraction quality\. For instance, the following sentence receives a near\-optimal score \(≈0\.006\\approx 0\.006\):

> *Cornell University is in Ithaca, New York and their nickname is Cornell Big Red\. They are the publisher of Administrative Science Quarterly and are affiliated with the Association of American Universities\.*

Here ClausIE successfully preserves both noun\-phrase structure and predicate multiplicity\. In contrast, it assigns a much higher score \(0\.3920\.392\) to:

> *Angola International Airport is located in Ícolo e Bengo, Luanda Province, Angola\. The runway is 4000ft long and is 159m a\.s\.l\.*

reflecting noun\-phrase fusion, predicate under\-specification, and incomplete coverage of relational structure\. Stanford CoreNLP 4\.5\.6 shows a similar pattern\. A low score \(≈0\.037\\approx 0\.037\) is obtained for:

> *Alan Shepard has died in California\. He was born in New Hampshire and graduated from NWC MA in 1957\. He served as a test pilot\.*

while a more linguistically complex example:

> *Above the Veil is an Australian novel and the sequel to Aenir and Castle\. It was followed by Into Battle and The Violet Keystone\.*

yields a substantially higher score \(0\.4460\.446\)\. This variation demonstrates that the metric is sensitive to structural, lexical, and contextual extraction failures\.

### 5\.2Case Study 1: ClausIE

Consider:

> *The inaugural Principal was Mr Reidal who managed the school from 1953 to 1956, the next principal was Bill Walker who served from 1957 to 1959\.*

ClausIE outputs:

- •\[the next principal, was, Bill\]
- •\[The inaugural Principal, was, Mr Reidal who managed school from 1953\]

The score \(\>0\.3\>0\.3\) reflects several extraction defects:

- •Noun incompleteness: spans such as1957 to 1959and the full temporal phraseschool from 1953 to 1956are truncated or omitted\.
- •Low resolution: compound noun phrases are merged, preventing correct KG structure from emerging\.
- •Predicate omission:managedandservedare missing, underspecifying the event structure\.

The ideal noun and verb sets, derived via POS tagging and dependency parsing, are:

- •Nouns:inaugural Principal, Mr Reidal, school from 1953 to 1956, next principal, Bill Walker, 1957 to 1959
- •Verbs:was, managed, was, served

Corrected triples:

- •\[The inaugural Principal, was, Mr Reidal\]
- •\[Mr Reidal, managed, the school from 1953 to 1956\]
- •\[The next principal, was, Bill Walker\]
- •\[Bill Walker, served, from 1957 to 1959\]

Evaluated with our metric, these revised triples yield a reduced score \(0\.0450\.045\), demonstrating that the metric rewards improvements aligned with KG quality principles such as granularity, coverage, and relation fidelity\.

### 5\.3Case Study 2: Negation Handling

Negation represents an important form of semantic polarity that is often mishandled by extraction tools\. For example:

> *“The pharmaceutical company did not receive FDA approval, nor did it secure investor funding\.”*

ClausIE extracts:

- •\(pharmaceutical company, received, FDA approval\)
- •\(pharmaceutical company, secured, investor funding\)

The baseline metric \(without negation\) assigns a low score \(0\.190\.19\), despite both the triples reversing the meaning of the sentence\. Using negation\-aware matching, the score becomes0\.580\.58, a205%205\\%increase\. This confirms that the negation extension substantially improves semantic robustness by penalizing polarity\-inverting extractions\.

### 5\.4Insights and Implications

Across both the case studies, the metric demonstrates three important properties:

1. 1\.Interpretability:Each metric component maps directly to identifiable extraction behaviours \(e\.g\., noun fusion, predicate loss, disconnectivity, negation omission\), offering actionable diagnostics\.
2. 2\.Sensitivity:Small structural or lexical refinements in triples consistently produce measurable metric improvements, supporting its use in iterative model development\.
3. 3\.Semantic grounding:The negation\-aware variant captures distinctions missed by surface\-level comparison metrics, improving alignment with KG correctness criteria\.

These results show that the metric is not merely a scoring mechanism but a principled evaluative tool that exposes fine\-grained extraction failures and guides targeted system improvements\.

### 5\.5Ablation Study: Noun vs\. Verb Metric Contribution

To understand the relative contribution of each metric component, we conduct an ablation study on the TinyButMighty dataset, evaluating three weighting scenarios: noun\-only \(wN=1\.0,wV=0\.0w\_\{N\}=1\.0,\\,w\_\{V\}=0\.0\), verb\-only \(wN=0\.0,wV=1\.0w\_\{N\}=0\.0,\\,w\_\{V\}=1\.0\), and equal weight for nouns and verbs\. Table[5](https://arxiv.org/html/2607.10212#S5.T5)reports the average scores per tool under each scenario\. Lower scores indicate higher fidelity\.

Table 5:Ablation study: average scores under different noun/verb weightings on TinyButMighty \(lower is better\)Three findings emerge\. First,predicate extraction is the primary weaknessacross all automated tools: every system scores worse on the verb\-only scenario than on the noun\-only scenario, confirming that relation extraction is harder than entity extraction\. Second, thegold\-standard ideal is the sole exception—its verb score \(0\.276\) is lower than its noun score \(0\.348\), indicating that human\-annotated triples capture predicates more faithfully than entities\. Third,ClausIE achieves the best overall score\(0\.1919 under equal weights\), while GPT\-4o mini performs the worst \(0\.5467\), suggesting that LLM\-based extractors over\-generate or paraphrase predicates in ways that hurt verb fidelity despite their fluent outputs\.

### 5\.6Downstream Validation: Correlation with Link Prediction

To validate that KGCQual scores reflect*downstream utility*, we test whether systems with lower \(better\) scores also produce KGs that support better link prediction\. For each IE system, we use its extracted triples, train a Tucker\[[2](https://arxiv.org/html/2607.10212#bib.bib31)\]embedding model on it, and measure the Mean Reciprocal Rank \(MRR\) on a test split\. Across all the three datasets, Table[6](https://arxiv.org/html/2607.10212#S5.T6)shows a near\-monotonic correspondence: as KGCQual worsens, MRR falls consistently\. The Spearman rank correlation isρ=−0\.900\\rho=\-0\.900\(p=0\.037p=0\.037\), confirming statistical significance\.

Table 6:KGCQual scores vs\. TuckER link\-prediction MRR \(aggregated across WebNLG, TinyButMighty, BenchIE; lower KGCQual = better; higher MRR = better\)\.This result provides direct evidence that KGCQual is not merely a structural proxy but correlates with real downstream task performance\. IE systems that better preserve entity and predicate structure produce KGs on which embedding models learn more effectively\.

## 6Limitations

Although the proposed noun–verb metric provides a principled, structurally grounded measure of extraction quality, it exhibits important limitations with respect to deeper semantic validation\. The metric is intentionally aligned with surface\-form properties, such as completeness, resolution, lexical correspondence, and predicate multiplicity, but does not yet assess whether the extracted triples preserve the*intended meaning*or*global semantics*of the sentence\. A key failure case emerges when triples are structurally plausible but semantically incoherent\. The metric is sensitive to*structural fidelity*but not to*semantic coherence*\.

Future work will focus on augmenting the metric with semantic components that go beyond lexical alignment\. Planned extensions include:

- •Semantic plausibility checks: assessing whether a triple aligns with the sentence’s propositional content rather than simply containing the correct lexical elements\.
- •Contextual semantic validation: exploring embeddings, entailment models, or sentence–triple alignment methods to evaluate whether extracted relations are semantically faithful to the original text\.

These additions aim to close the gap between structural correctness and semantic correctness, enabling more comprehensive evaluation of IE systems within Knowledge Graph construction pipelines\.

## 7Conclusion

We introduce KGCQual, a sentence\-level intrinsic metric for evaluating the quality of triples extracted by Information Extraction \(IE\) systems, including LLMs\. Our formulation captures two fundamental dimensions of extraction quality relevant to Knowledge Graph \(KG\) construction: \(i\) noun\-phrase fidelity, reflecting completeness, granularity, and graph connectivity, and \(ii\) predicate fidelity, capturing verb multiplicity and lexical alignment\. Grounded in POS tagging and dependency parsing, the metric provides a transparent, interpretable measure of how closely a system’s extracted triples approximate an ideal sentence\-level graph\. Our evaluation results show that systems merging noun phrases, omitting key entities, or under\-generating predicates score higher \(worse\), while systems aligning with human\-annotated gold triples score lower \(better\)\. This confirms the metric behaves as intended and signals extraction fidelity\.

An ablation study confirms that predicate extraction is the primary weakness across all systems, with verb\-metric scores consistently higher than noun\-metric scores\. A downstream validation further demonstrates that KGCQual scores correlate significantly with link prediction MRR, providing direct evidence that the metric reflects downstream KG quality and not merely surface\-level structural alignment\.

By offering a principled, interpretable, and computationally efficient evaluation method, this work provides a practical tool for enhancing the reliability of triple extraction pipelines in Knowledge Graph applications\.

## Declaration of use of Generative AI

We used OpenAI’s ChatGPT to help polish some sentences for clarity and to assist in formatting parts of this paper in LaTeX\. All technical content, implementation, validation, and analysis were developed and verified by the authors, who take full responsibility for the work\.

## References

- \[1\]G\. Angeli, M\. J\. Premkumar, and C\. Manning\(2015\)Stanford openie \(version 4\.x\) system description\.Technical reportStanford NLP\.Note:Technical Report / System DemoCited by:[§4](https://arxiv.org/html/2607.10212#S4.p1.1)\.
- \[2\]I\. Balažević, C\. Allen, and T\. M\. Hospedales\(2019\)TuckER: tensor factorization for knowledge graph completion\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing \(EMNLP\-IJCNLP\),pp\. 5185–5194\.Cited by:[§5\.6](https://arxiv.org/html/2607.10212#S5.SS6.p1.2)\.
- \[3\]E\. Brill\(1995\)Transformation\-based error\-driven learning and natural language processing: a case study in part\-of\-speech tagging\.Computational Linguistics21\(4\)\.Cited by:[§4](https://arxiv.org/html/2607.10212#S4.p1.1)\.
- \[4\]I\. Dagan, B\. Dolan, and B\. Magnini\(2009\)Recognizing textual entailment\.ACL\.Cited by:[§2\.3](https://arxiv.org/html/2607.10212#S2.SS3.p2.1)\.
- \[5\]L\. Del Corro and R\. Gemulla\(2013\)ClausIE: clause\-based open information extraction\.InProceedings of WWW 2013,pp\. 355–366\.Cited by:[§4](https://arxiv.org/html/2607.10212#S4.p1.1)\.
- \[6\]M\. Färber, F\. Bartscherer, C\. Menne, and A\. Rettinger\(2018\)Linked data quality of dbpedia, freebase, opencyc, wikidata, and yago\.Semantic Web9\(1\),pp\. 77–129\.Cited by:[§1](https://arxiv.org/html/2607.10212#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.10212#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2607.10212#S2.SS2.p1.1)\.
- \[7\]R\. Gabriel, S\. Soderland, and O\. Etzioni\(2011\)Open information extraction from the web\.InIJCAI,Cited by:[§2\.3](https://arxiv.org/html/2607.10212#S2.SS3.p2.1)\.
- \[8\]C\. Gardent, A\. Shimorina, S\. Narayan, and L\. Perez\-Beltrachini\(2017\)The webnlg challenge: generating text from rdf data\.InProceedings of the WebNLG Challenge,Cited by:[1st item](https://arxiv.org/html/2607.10212#S4.I1.i1.p1.1)\.
- \[9\]K\. Gashteovski, R\. Gemulla, and L\. Del Corro\(2017\)MinIE: minimizing facts in open information extraction\.InProceedings of EMNLP,pp\. 2630–2640\.Cited by:[§1](https://arxiv.org/html/2607.10212#S1.p7.1),[§2\.3](https://arxiv.org/html/2607.10212#S2.SS3.p1.1),[§4](https://arxiv.org/html/2607.10212#S4.p1.1)\.
- \[10\]K\. Gashteovski, M\. Yu, B\. Kotnis, C\. Lawrence, M\. Niepert, and G\. Glavaš\(2022\)BenchIE: a framework for multi\-faceted fact\-based open information extraction evaluation\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[3rd item](https://arxiv.org/html/2607.10212#S4.I1.i3.p1.1)\.
- \[11\]A\. Hogan, E\. Blomqvist, M\. Cochez, C\. D’Amato, G\. de Melo, C\. Gutiérrez, S\. Kirrane, S\. Neumaier, A\. Polleres, V\. Prince, P\. Škoda, C\. Lange, and A\. Zimmermann\(2021\)Knowledge graphs\.ACM Computing Surveys54\(4\),pp\. 1–37\.Cited by:[§1](https://arxiv.org/html/2607.10212#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.10212#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2607.10212#S2.SS2.p1.1)\.
- \[12\]J\. Lindsey, W\. Gurnee, E\. Ameisen, B\. Chen, A\. Pearce, and L\. Nichols\(2024\)On the biology of a large language model\.InProceedings of the Transformer Circuits Interpretability Workshop \(TCIW\) 2024,Cited by:[§4](https://arxiv.org/html/2607.10212#S4.p1.1)\.
- \[13\]Mausam, H\.\-P\. Schmitz, and O\. Etzioni\(2012\)Ollie: learning open information extraction from question\-answer pairs\.InProceedings of ACL 2012,Cited by:[§1](https://arxiv.org/html/2607.10212#S1.p7.1),[§2\.3](https://arxiv.org/html/2607.10212#S2.SS3.p1.1),[§4](https://arxiv.org/html/2607.10212#S4.p1.1)\.
- \[14\]S\. Patel and L\. Gomez\(2024\)Comparison of llms in extracting synthesis conditions: gemini excels in completeness\.InProceedings of ACL,Cited by:[§4](https://arxiv.org/html/2607.10212#S4.p1.1)\.
- \[15\]S\. Roy Choudhury and Z\. Dey\(2024\)GPT\-3 powered information extraction for building robust knowledge bases\.Cited by:[§2\.3](https://arxiv.org/html/2607.10212#S2.SS3.p1.1),[§4](https://arxiv.org/html/2607.10212#S4.p1.1)\.
- \[16\]M\. Salman, A\. Haller, S\. J\. Rodriguez Mendez, and U\. Naseem\(2024\)Tiny but mighty: a crowdsourced benchmark dataset for triple extraction from unstructured text\.InProceedings of the 20th Joint ACL–ISO Workshop on Interoperable Semantic Annotation @ LREC\-COLING,Torino, Italy,pp\. 71–81\.Cited by:[2nd item](https://arxiv.org/html/2607.10212#S4.I1.i2.p1.1)\.
- \[17\]A\. B\. Smith, P\. R\. Miller, and H\. Wilson\(2020\)Efficient knowledge graph accuracy evaluation\.InProceedings of the IEEE International Conference on Data Engineering \(ICDE\),Cited by:[§2\.2](https://arxiv.org/html/2607.10212#S2.SS2.p1.1)\.
- \[18\]A\. Zaveri, A\. Rula, A\. Maurino, R\. Pietrobon, J\. Lehmann, and S\. Auer\(2016\)Quality assessment for linked data: a survey\.Semantic Web7\(1\),pp\. 63–93\.Cited by:[§1](https://arxiv.org/html/2607.10212#S1.p1.1),[§1](https://arxiv.org/html/2607.10212#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.10212#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2607.10212#S2.SS2.p1.1)\.

## Appendix 0\.AAblation Study: Full Results

Table[7](https://arxiv.org/html/2607.10212#Pt0.A1.T7)extends Table[5](https://arxiv.org/html/2607.10212#S5.T5)with all five weighting scenarios\. Lower scores indicate higher extraction fidelity\.

Table 7:Ablation study on TinyButMighty: average scores under all five noun/verb weighting scenarios \(lower is better\)\.The ranking of tools is stable across all scenarios\. ClausIE consistently achieves the lowest scores and GPT\-4o mini the highest among automated tools\. The Ideal set is the only case where the verb score \(0\.276\) is lower than the noun score \(0\.348\), confirming that human\-annotated triples capture predicates more faithfully than any automated system\.

## Appendix 0\.BDownstream Validation: Full Results

Tables[8](https://arxiv.org/html/2607.10212#Pt0.A2.T8)–[10](https://arxiv.org/html/2607.10212#Pt0.A2.T10)report MRR and Hits@10 for every \(IE system, embedding model\) combination, split by dataset\. TuckER results are used for the correlation analysis in Section[5\.6](https://arxiv.org/html/2607.10212#S5.SS6); ComplEx and NodePiece are included for completeness\. ClausIE on BenchIE produced no valid triples and is omitted from Table[10](https://arxiv.org/html/2607.10212#Pt0.A2.T10)\.

Table 8:Downstream link prediction results onWebNLG\.Table 9:Downstream link prediction results onTinyButMighty\.Table 10:Downstream link prediction results onBenchIE\. ClausIE produced no valid triples on this dataset and is excluded\.### 0\.B\.1Note on ComplEx and NodePiece

ComplEx and NodePiece produced near\-zero MRR across most runs\. Both models require substantially larger KGs to learn reliable embeddings\. The extracted KGs in this study range from 258 to 5 356 triples—an order of magnitude smaller than standard benchmarks such as FB15k\-237, which provides 272 115 training triples\. TuckER is empirically more robust to small graph sizes and is therefore used as the primary model for the correlation analysis in Section[5\.6](https://arxiv.org/html/2607.10212#S5.SS6)\.

### 0\.B\.2Per\-Dataset Spearman Correlation

Table[11](https://arxiv.org/html/2607.10212#Pt0.A2.T11)reports exact Spearman rank correlations between KGCQual scores and TuckER MRR computed per\-dataset\.

Table 11:Per\-dataset Spearman rank correlation \(ρ\\rho\) between KGCQual score and TuckER MRR\.nn= number of IE systems with valid results\.Significance is achieved on TinyButMighty—the dataset with the most linguistically complex sentences—but not on WebNLG or BenchIE in isolation\. Two factors explain this\. First, per\-dataset sample sizes of 4–5 IE systems reduce statistical power substantially\. Second, the exclusion of ClausIE from BenchIE due to an extraction error removes the best\-performing system from that dataset, distorting its ranking\. These limitations motivate the aggregated analysis in Section[5\.6](https://arxiv.org/html/2607.10212#S5.SS6), which pools all 14 data points and yieldsρ=−0\.900\\rho=\-0\.900,p=0\.037p=0\.037\.