MissDiag: Diagnostic Evaluation of Incomplete-Knowledge Robustness in KGQA and KG-RAG

# Diagnostic Evaluation of Incomplete-Knowledge Robustnessin KGQA and KG-RAG
Source: [https://arxiv.org/html/2608.18489](https://arxiv.org/html/2608.18489)
## MissDiag: Diagnostic Evaluation of Incomplete\-Knowledge Robustness in KGQA and KG\-RAG

###### Abstract

Knowledge graph question answering \(KGQA\) and knowledge\-graph\-based retrieval\-augmented generation \(KG\-RAG\) aim to ground answers in explicit graph evidence, but real\-world knowledge graphs are often sparse, outdated, and incomplete\. Existing robustness evaluations usually report aggregate changes in answer quality after evidence is removed or perturbed, which measures sensitivity to incomplete support but leaves the source of degradation under\-specified: the same score change can conflate the type of missing evidence, the response of the evaluated system, and the sensitivity of the answer\-matching protocol\. To address this gap, we proposeMissDiag, a diagnostic evaluation framework for incomplete\-knowledge robustness in KGQA and KG\-RAG\. MissDiag keeps the question and gold answer fixed while applying structurally typed missingness interventions to benchmark\-provided support graphs, enabling paired comparisons that decompose robustness changes by evidence type, system response, and evaluation protocol rather than reducing them to a single aggregate score drop\. Experiments across multiple system families show that incomplete\-knowledge robustness is better understood as a typed degradation phenomenon than as a uniform property: answer\-adjacent evidence loss produces the largest observed degradation, source\-context removal is often neutral and can be beneficial, and semantic answer matching changes absolute scores while preserving the main typed degradation patterns\. By transforming aggregate robustness measurement into typed diagnostic attribution, MissDiag provides a more interpretable basis for comparing, diagnosing, and stress\-testing KGQA and KG\-RAG systems under incomplete knowledge\.

> Code will be released after the review process\.

## Introduction

Knowledge graph question answering \(KGQA\) and knowledge\-graph\-based retrieval\-augmented generation \(KG\-RAG\) aim to ground answers in explicit graph evidence rather than relying only on parametric memory\. This makes them important testbeds for evaluating structured reasoning, factual grounding, and answer trustworthiness\. Prior work has advanced this goal through semantic parsing and query\-graph construction\([1](https://arxiv.org/html/2608.18489#bib.bib28);[43](https://arxiv.org/html/2608.18489#bib.bib29)\), embedding\-based and graph\-based QA\([14](https://arxiv.org/html/2608.18489#bib.bib13);[31](https://arxiv.org/html/2608.18489#bib.bib10);[28](https://arxiv.org/html/2608.18489#bib.bib30);[41](https://arxiv.org/html/2608.18489#bib.bib7);[8](https://arxiv.org/html/2608.18489#bib.bib8);[22](https://arxiv.org/html/2608.18489#bib.bib9)\), retrieve\-and\-reason systems over knowledge bases and text\([10](https://arxiv.org/html/2608.18489#bib.bib33);[18](https://arxiv.org/html/2608.18489#bib.bib32)\), LLM\-based KBQA\([38](https://arxiv.org/html/2608.18489#bib.bib35)\), and increasingly challenging benchmarks\([5](https://arxiv.org/html/2608.18489#bib.bib31);[7](https://arxiv.org/html/2608.18489#bib.bib3);[2](https://arxiv.org/html/2608.18489#bib.bib6);[46](https://arxiv.org/html/2608.18489#bib.bib1)\)\. More recently, graph\-grounded LLM studies have examined whether KG augmentation improves factuality, reasoning quality, retrieval efficiency, and trustworthiness in open\-ended generation settings\([36](https://arxiv.org/html/2608.18489#bib.bib36);[32](https://arxiv.org/html/2608.18489#bib.bib34);[30](https://arxiv.org/html/2608.18489#bib.bib26);[49](https://arxiv.org/html/2608.18489#bib.bib2)\)\. Across these lines of work, however, a persistent difficulty remains: real\-world knowledge graphs are rarely complete\. They are often sparse, outdated, and unevenly populated across entities and relations, making robustness under incomplete knowledge a central requirement for reliable KGQA and KG\-RAG systems\.

![Refer to caption](https://arxiv.org/html/2608.18489v1/figures/eg.png)Figure 1:Motivation for typed degradation analysis\. The same QA instance can show similar aggregate degradation under structurally different missingness conditions\. Aggregate score drops alone do not reveal which evidence type was removed, whether answer\-local evidence was affected, or whether the effect is systematic\.Existing research has studied incomplete knowledge from several complementary perspectives, including contextual reasoning over partial graph structures\([21](https://arxiv.org/html/2608.18489#bib.bib14)\), integration of textual evidence\([39](https://arxiv.org/html/2608.18489#bib.bib11);[11](https://arxiv.org/html/2608.18489#bib.bib12);[33](https://arxiv.org/html/2608.18489#bib.bib20)\), KG embeddings and relation prediction\([35](https://arxiv.org/html/2608.18489#bib.bib37);[27](https://arxiv.org/html/2608.18489#bib.bib38);[34](https://arxiv.org/html/2608.18489#bib.bib39);[14](https://arxiv.org/html/2608.18489#bib.bib13);[26](https://arxiv.org/html/2608.18489#bib.bib15);[48](https://arxiv.org/html/2608.18489#bib.bib18);[45](https://arxiv.org/html/2608.18489#bib.bib21);[25](https://arxiv.org/html/2608.18489#bib.bib16);[9](https://arxiv.org/html/2608.18489#bib.bib19)\), knowledge graph completion\([48](https://arxiv.org/html/2608.18489#bib.bib18);[9](https://arxiv.org/html/2608.18489#bib.bib19)\), completion\-aware reasoning pipelines\([20](https://arxiv.org/html/2608.18489#bib.bib17);[42](https://arxiv.org/html/2608.18489#bib.bib23);[12](https://arxiv.org/html/2608.18489#bib.bib22)\), and LLM\-based or multi\-agent reasoning over incomplete graphs\([40](https://arxiv.org/html/2608.18489#bib.bib48);[19](https://arxiv.org/html/2608.18489#bib.bib25)\)\. In parallel, evaluation studies have asked whether completion methods improve downstream QA and whether current benchmarks and metrics support reliable conclusions under incomplete or imperfect knowledge\([44](https://arxiv.org/html/2608.18489#bib.bib24);[23](https://arxiv.org/html/2608.18489#bib.bib4);[29](https://arxiv.org/html/2608.18489#bib.bib5);[46](https://arxiv.org/html/2608.18489#bib.bib1);[47](https://arxiv.org/html/2608.18489#bib.bib27);[49](https://arxiv.org/html/2608.18489#bib.bib2);[30](https://arxiv.org/html/2608.18489#bib.bib26)\)\. These studies have substantially improved our understanding of how systems recover missing facts, exploit auxiliary evidence, or reason over partial structures\. Nevertheless, current evaluation practice still has a basic attribution limitation: most evaluations remove, corrupt, or recover evidence and then report the resulting change in answer quality\. Such degradation\-based evaluation can measure whether performance changes, but it does not explain why the change occurs\.

This ambiguity is illustrated in Figure[1](https://arxiv.org/html/2608.18489#Sx1.F1): the same question–answer instance may exhibit similar aggregate score drops under structurally different evidence\-removal conditions, even though the underlying failure mechanisms are different\. A lower score may indicate that essential supporting evidence has become unavailable; it may also indicate that the system fails to exploit retained evidence, that graph conversion or candidate construction changes the effective search space, or that the evaluation protocol fails to recognize a semantically acceptable answer\. Conversely, apparent robustness or even improvement under incomplete evidence may reflect support pruning or metric behavior rather than stronger reasoning\. This matters because robustness under incomplete knowledge is often used as evidence of reasoning capability and system reliability\. If similar score changes can arise from different causes, then aggregate degradation alone is insufficient for interpreting robustness claims\. What is needed is not only a performance measurement protocol, but a diagnostic evaluation framework that can attribute degradation to different forms of missing evidence, compare how system families respond to the same intervention, and expose how much the conclusion depends on the evaluation metric\.

To address this gap, we proposeMissDiag, a diagnostic evaluation framework for incomplete\-knowledge robustness in KGQA and KG\-RAG\. As shown in Figure[2](https://arxiv.org/html/2608.18489#Sx1.F2), MissDiag starts from a benchmark\-provided evaluation instance consisting of a question, a gold answer set, and a local support graph\. It keeps the question and gold answer fixed while transforming the support graph into structurally distinct incomplete\-evidence conditions\. The primary missingness operators include random support loss, source\-context loss, relation\-level removal, and answer\-adjacent removal\. By evaluating the same system on the same question–answer pair before and after each typed intervention, MissDiag separates three factors that are usually entangled in incomplete\-knowledge evaluation: missingness type, system response, and evaluation sensitivity\.

![Refer to caption](https://arxiv.org/html/2608.18489v1/figures/FW.png)Figure 2:Overview of MissDiag\. \(1\)Input: an instance contains a question, gold answer set, and local support graph\. \(2\)Typed Missingness Operators: typed operators select removable support edges for random, source\-context, relation\-level, and answer\-adjacent missingness\. \(3\)Severity\-Controlled Missingness: a shared severity budget produces an incomplete support graph\. \(4\)Paired Evaluation: the same system is evaluated under complete and incomplete support to compute paired degradation\. \(5\)Typed Degradation Profile: degradation values are summarized across missingness type, severity, system, and metric\.Rather than treating incomplete knowledge as a single robustness condition, MissDiag represents it as a typed degradation phenomenon\. The framework reports robustness through degradation profiles indexed by missingness type, severity, system, and evaluation metric, with structural slices used for further analysis\. This design enables paired and interpretable comparison across trained KGQA models, graph\-structured prompting methods, iterative KG agents, and direct LLM baselines\. It therefore allows robustness claims to be examined in terms of where degradation comes from, when it reflects genuine evidence loss, and when it should instead be attributed to system behavior or evaluation sensitivity\.

Our contributions are as follows:

- •We formulate incomplete\-knowledge robustness evaluation as a diagnostic attribution problem, showing that aggregate score changes conflate evidence availability, system behavior, and evaluation protocol, and therefore cannot by themselves support reliable conclusions about reasoning robustness\.
- •We introduce MissDiag, a controlled diagnostic framework that applies structurally typed missingness interventions to benchmark support graphs while preserving paired question–answer comparisons, enabling degradation to be analyzed by missingness source rather than only by overall performance loss\.
- •We instantiate MissDiag across multiple system families, severity levels, structural slices, and answer\-matching metrics, demonstrating that the same missing\-evidence intervention can lead to degradation, near invariance, or improvement depending on system design and evaluation protocol\. This reveals robustness patterns that aggregate scores obscure\.

## Related Work

### KGQA and Graph\-Grounded QA

KGQA aims to answer natural\-language questions using structured graph evidence\. Early work often relied on semantic parsing or query\-graph construction to map questions into executable logical forms or graph queries\([1](https://arxiv.org/html/2608.18489#bib.bib28);[43](https://arxiv.org/html/2608.18489#bib.bib29)\)\. Later graph\-based models retrieve and reason over local evidence subgraphs, making the support structure itself part of the answering process\([31](https://arxiv.org/html/2608.18489#bib.bib10);[13](https://arxiv.org/html/2608.18489#bib.bib43);[22](https://arxiv.org/html/2608.18489#bib.bib9)\)\. In parallel, retrieval\-centered QA and graph\-grounded LLM methods use retrieved knowledge to support generation and reasoning beyond purely parametric memory\([10](https://arxiv.org/html/2608.18489#bib.bib33);[18](https://arxiv.org/html/2608.18489#bib.bib32);[36](https://arxiv.org/html/2608.18489#bib.bib36);[38](https://arxiv.org/html/2608.18489#bib.bib35);[32](https://arxiv.org/html/2608.18489#bib.bib34)\)\. Benchmarks such as LC\-QuAD 2\.0, GrailQA, KQA Pro, and KGQAGen broaden evaluation beyond simple fact lookup by emphasizing compositionality, generalization, and dataset reliability\([5](https://arxiv.org/html/2608.18489#bib.bib31);[7](https://arxiv.org/html/2608.18489#bib.bib3);[2](https://arxiv.org/html/2608.18489#bib.bib6);[46](https://arxiv.org/html/2608.18489#bib.bib1)\)\. These studies provide the system and benchmark context for our work\. MissDiag differs by treating existing systems as diagnostic subjects under controlled evidence manipulation rather than proposing another KGQA architecture\.

### Incomplete\-Knowledge Question Answering

Incomplete knowledge is a persistent challenge for KGQA because real\-world graphs are sparse, unevenly populated, and often missing facts needed for multi\-hop reasoning\. Prior work has addressed this issue by reasoning over partial graph structures\([21](https://arxiv.org/html/2608.18489#bib.bib14)\), incorporating auxiliary textual evidence\([39](https://arxiv.org/html/2608.18489#bib.bib11);[11](https://arxiv.org/html/2608.18489#bib.bib12)\), using graph completion or completion\-aware QA pipelines\([44](https://arxiv.org/html/2608.18489#bib.bib24);[42](https://arxiv.org/html/2608.18489#bib.bib23)\), and applying LLM\-centered reasoning to incomplete graph evidence\([40](https://arxiv.org/html/2608.18489#bib.bib48);[49](https://arxiv.org/html/2608.18489#bib.bib2)\)\. These approaches mainly ask how to recover or maintain answer quality when knowledge is missing\. Our work asks a complementary evaluation question: when answer quality changes under incomplete knowledge, how should the change be attributed? Instead of treating missingness as a single condition, MissDiag separates different structural forms of evidence loss and compares their effects under a paired protocol\.

### Benchmark Reliability and Evaluation Sensitivity

Evaluation conclusions in KGQA and graph\-grounded QA are sensitive to dataset construction, evidence availability, and answer\-matching protocols\. Dataset audits and leaderboard analyses show that KGQA benchmarks can contain annotation issues, heterogeneous difficulty, and inconsistent reporting practices\([29](https://arxiv.org/html/2608.18489#bib.bib5);[23](https://arxiv.org/html/2608.18489#bib.bib4);[46](https://arxiv.org/html/2608.18489#bib.bib1)\)\. More broadly, adversarial evaluation and behavioral testing show that aggregate metrics can hide distinct failure mechanisms behind similar score changes\([15](https://arxiv.org/html/2608.18489#bib.bib42);[24](https://arxiv.org/html/2608.18489#bib.bib40);[6](https://arxiv.org/html/2608.18489#bib.bib41)\)\. Recent KG\-RAG and trustworthiness studies further suggest that graph augmentation and incomplete evidence require careful evaluation design\([30](https://arxiv.org/html/2608.18489#bib.bib26);[47](https://arxiv.org/html/2608.18489#bib.bib27);[49](https://arxiv.org/html/2608.18489#bib.bib2)\)\. This literature motivates our view that evaluation is not a neutral reporting layer\. MissDiag extends this line of work by decomposing incomplete\-knowledge evaluation into missingness type, system response, severity, and metric sensitivity, so that robustness claims can be interpreted beyond a single aggregate degradation score\.

## Method

This section introduces MissDiag, a diagnostic evaluation framework for incomplete\-knowledge robustness in KGQA and KG\-RAG\. As shown in Figure[2](https://arxiv.org/html/2608.18489#Sx1.F2), the framework keeps the question, gold answer, and evaluated system fixed, transforms the support graph with severity\-controlled typed missingness operators, and compares the resulting outputs against the complete\-support condition\. This design turns aggregate robustness changes into paired degradation profiles indexed by missingness type, severity, system, and metric\. The section first formulates the diagnostic evaluation problem, then describes support graph construction, defines the missingness operators, and presents the paired degradation profile used as the main diagnostic output\.

### Diagnostic Formulation

Each evaluation instance is represented as

xi=\(qi,Ai∗,Gi\),x\_\{i\}=\(q\_\{i\},A\_\{i\}^\{\\ast\},G\_\{i\}\),\(1\)whereqiq\_\{i\}is the question,Ai∗A\_\{i\}^\{\\ast\}is the gold answer set, andGi=\(Vi,Ei\)G\_\{i\}=\(V\_\{i\},E\_\{i\}\)is the local support graph\. The source entities linked to the question are denoted bySi⊆ViS\_\{i\}\\subseteq V\_\{i\}, and the gold\-answer entities aligned to graph nodes are denoted byYi⊆ViY\_\{i\}\\subseteq V\_\{i\}\. Operators requiring unavailable alignments mark the instance infeasible for the corresponding operator\. Given an evaluated systemff, MissDiag compares the complete\-support predictionf⁡\(qi,Gi\)f\(q\_\{i\},G\_\{i\}\)with predictions obtained after transforming only the support edges\. Across conditions, the question, gold answer, source entities, and node inventory remain fixed; only the retained edge evidence changes\.

### Support Graph Construction

The local support graphGiG\_\{i\}is constructed from the evidence associated with instancexix\_\{i\}\. When the evidence is given as triples, proof paths, or a retrieved subgraph, it is converted into a labeled directed graph\. Graph distances used by the missingness operators are computed on the undirected projection of this graph\. For a nodeuuand node setBB,dist⁡\(u,B\)\\mathrm\{dist\}\(u,B\)denotes the shortest\-path distance fromuuto any node inBBon the undirected projection\.

The edge set is written as

Ei=Eisup∪Eictx,E\_\{i\}=E\_\{i\}^\{\\mathrm\{sup\}\}\\cup E\_\{i\}^\{\\mathrm\{ctx\}\},\(2\)whereEisupE\_\{i\}^\{\\mathrm\{sup\}\}denotes the original support evidence andEictxE\_\{i\}^\{\\mathrm\{ctx\}\}denotes a possibly empty set of optional source\-anchored context edges\. The context edges are constructed before any missingness intervention, follow a deterministic selection rule, and remain fixed across all conditions\. This ensures that all missingness operators start from the same support graph\.

After an intervention, the evaluated system receives only the retained support evidence\. The node inventory may be kept fixed internally for alignment and paired comparison, but isolated nodes are not exposed as additional answer hints in generative KG\-RAG prompts\. Additional construction details are provided in the supplementary material\.

### Severity\-Controlled Missingness

Letα∈\[0,1\]\\alpha\\in\[0,1\]denote the missingness severity andni=\|Ei\|n\_\{i\}=\|E\_\{i\}\|the number of support edges\. For each instance, the nominal removal budget is defined as

κi​\(α\)=\{0,ni≤1​or​α=0,min⁡\(ni−1,ηi​\(α\)\),otherwise,\\kappa\_\{i\}\(\\alpha\)=\\begin\{cases\}0,&n\_\{i\}\\leq 1\\text\{ or \}\\alpha=0,\\\\ \\min\\bigl\(n\_\{i\}\-1,\\,\\eta\_\{i\}\(\\alpha\)\\bigr\),&\\text\{otherwise,\}\\end\{cases\}\(3\)whereηi​\(α\)=max⁡\(1,round⁡\(α​ni\)\)\\eta\_\{i\}\(\\alpha\)=\\max\(1,\\mathrm\{round\}\(\\alpha n\_\{i\}\)\)\. This budget preserves complete support atα=0\\alpha=0and keeps at least one support edge when removal is applied\.

Each missingness operatormmdefines an ordered candidate edge listLi\(m\)L\_\{i\}^\{\(m\)\}\. Given the shared budget, the removed edge setRi\(m,α\)R\_\{i\}^\{\(m,\\alpha\)\}consists of the firstmin⁡\(κi​\(α\),\|Li\(m\)\|\)\\min\(\\kappa\_\{i\}\(\\alpha\),\|L\_\{i\}^\{\(m\)\}\|\)edges inLi\(m\)L\_\{i\}^\{\(m\)\}\. The retained support graph is

G~i\(m,α\)=\(Vi,Ei∖Ri\(m,α\)\),\\widetilde\{G\}\_\{i\}^\{\(m,\\alpha\)\}=\(V\_\{i\},E\_\{i\}\\setminus R\_\{i\}^\{\(m,\\alpha\)\}\),\(4\)whereRi\(m,α\)R\_\{i\}^\{\(m,\\alpha\)\}is the removed edge set\. Ifκi​\(α\)\>0\\kappa\_\{i\}\(\\alpha\)\>0butLi\(m\)L\_\{i\}^\{\(m\)\}is empty, the instance is marked infeasible for operatormm\.

### Typed Missingness Operators

MissDiag uses four typed missingness operators: random, source\-context, relation\-level, and answer\-adjacent missingness\. Each operator first defines a candidate edge set and then orders it intoLi\(m\)L\_\{i\}^\{\(m\)\}, which is used by the shared removal budget\. The operators are designed to probe different structural forms of evidence loss under the same removal budget, rather than to define mutually exclusive edge categories\.

For an edgee=\(u,r,v\)e=\(u,r,v\), two structural scores are used:

τi​\(e\)=max⁡\{dist⁡\(u,Si\),dist⁡\(v,Si\)\},\\tau\_\{i\}\(e\)=\\max\\\{\\mathrm\{dist\}\(u,S\_\{i\}\),\\mathrm\{dist\}\(v,S\_\{i\}\)\\\},\(5\)ρi​\(e\)=min⁡\{dist⁡\(u,Yi\),dist⁡\(v,Yi\)\}\.\\rho\_\{i\}\(e\)=\\min\\\{\\mathrm\{dist\}\(u,Y\_\{i\}\),\\mathrm\{dist\}\(v,Y\_\{i\}\)\\\}\.\(6\)Here,τi​\(e\)\\tau\_\{i\}\(e\)measures source depth andρi​\(e\)\\rho\_\{i\}\(e\)measures answer distance\.

#### Random missingness\.

Random missingness uses all support edges as candidates,Ci\(rand\)=EiC\_\{i\}^\{\(\\mathrm\{rand\}\)\}=E\_\{i\}\. The ordered listLi\(rand\)L\_\{i\}^\{\(\\mathrm\{rand\}\)\}is a uniform random permutation ofCi\(rand\)C\_\{i\}^\{\(\\mathrm\{rand\}\)\}\. This operator serves as a quantity\-matched baseline for generic support loss\.

#### Source\-context missingness\.

Source\-context missingness targets edges incident to source entities but not incident to aligned answer entities:

Ci\(src\)=\{e=\(u,r,v\)∈Ei:\\displaystyle C\_\{i\}^\{\(\\mathrm\{src\}\)\}=\\\{e=\(u,r,v\)\\in E\_\{i\}:\{u,v\}∩Si≠∅,\\displaystyle\\\{u,v\\\}\\cap S\_\{i\}\\neq\\emptyset,\(7\)\{u,v\}∩Yi=∅\}\.\\displaystyle\\\{u,v\\\}\\cap Y\_\{i\}=\\emptyset\\\}\.The ordered listLi\(src\)L\_\{i\}^\{\(\\mathrm\{src\}\)\}is obtained by sorting candidates by increasingτi​\(e\)\\tau\_\{i\}\(e\), with deterministic tie\-breaking\. This operator probes sensitivity to source\-side context while excluding answer\-incident evidence from its candidate set\.

#### Relation\-level missingness\.

Relation\-level missingness uses all support edges as candidates,Ci\(rel\)=EiC\_\{i\}^\{\(\\mathrm\{rel\}\)\}=E\_\{i\}, and groups them by relation type\. Relation blocks are ordered by decreasing frequency inEiE\_\{i\}, and edges are removed block by block under the shared budget\. This operator probes sensitivity to relation\-family evidence\.

#### Answer\-adjacent missingness\.

Answer\-adjacent missingness targets edges incident to, or one hop from, aligned answer entities:

Ci\(ans\)=\{e∈Ei:ρi​\(e\)≤1\}\.C\_\{i\}^\{\(\\mathrm\{ans\}\)\}=\\\{e\\in E\_\{i\}:\\rho\_\{i\}\(e\)\\leq 1\\\}\.\(8\)The ordered listLi\(ans\)L\_\{i\}^\{\(\\mathrm\{ans\}\)\}is obtained by sorting candidates by increasingρi​\(e\)\\rho\_\{i\}\(e\), with deterministic tie\-breaking\. This operator probes sensitivity to answer\-local grounding evidence\.

### Paired Degradation Profiles

Letμ\\mube the answer\-set evaluation metric used for a given comparison\. MissDiag treats the metric as an explicit axis of the diagnostic profile\. For systemff, the complete\-support score of instanceiiisμ⁡\(f⁡\(qi,Gi\),Ai∗\)\\mu\(f\(q\_\{i\},G\_\{i\}\),A\_\{i\}^\{\\ast\}\), and the score under missingness typemmand severityα\\alphaisμ⁡\(f⁡\(qi,G~i\(m,α\)\),Ai∗\)\\mu\(f\(q\_\{i\},\\widetilde\{G\}\_\{i\}^\{\(m,\\alpha\)\}\),A\_\{i\}^\{\\ast\}\)\. The paired degradation is defined as

δi\(m,α,μ\)=μ⁡\(f⁡\(qi,Gi\),Ai∗\)−μ⁡\(f⁡\(qi,G~i\(m,α\)\),Ai∗\)\.\\delta\_\{i\}^\{\(m,\\alpha,\\mu\)\}=\\mu\(f\(q\_\{i\},G\_\{i\}\),A\_\{i\}^\{\\ast\}\)\-\\mu\(f\(q\_\{i\},\\widetilde\{G\}\_\{i\}^\{\(m,\\alpha\)\}\),A\_\{i\}^\{\\ast\}\)\.\(9\)Positive values indicate performance loss after evidence removal, while negative values indicate improvement\.

Letℐ\(m,α\)\\mathcal\{I\}^\{\(m,\\alpha\)\}be the feasible instance set for operatormmat severityα\\alpha\. The dataset\-level degradation of systemffis

Δf\(m,α,μ\)=1\|ℐ\(m,α\)\|​∑i∈ℐ\(m,α\)δi\(m,α,μ\)\.\\Delta\_\{f\}^\{\(m,\\alpha,\\mu\)\}=\\frac\{1\}\{\|\\mathcal\{I\}^\{\(m,\\alpha\)\}\|\}\\sum\_\{i\\in\\mathcal\{I\}^\{\(m,\\alpha\)\}\}\\delta\_\{i\}^\{\(m,\\alpha,\\mu\)\}\.\(10\)
The main diagnostic output is the typed degradation profile

𝐃f\(α,μ\)=\[Δf\(rand,α,μ\)Δf\(src,α,μ\)Δf\(rel,α,μ\)Δf\(ans,α,μ\)\]\.\\mathbf\{D\}\_\{f\}^\{\(\\alpha,\\mu\)\}=\\begin\{bmatrix\}\\Delta\_\{f\}^\{\(\\mathrm\{rand\},\\alpha,\\mu\)\}\\\\ \\Delta\_\{f\}^\{\(\\mathrm\{src\},\\alpha,\\mu\)\}\\\\ \\Delta\_\{f\}^\{\(\\mathrm\{rel\},\\alpha,\\mu\)\}\\\\ \\Delta\_\{f\}^\{\(\\mathrm\{ans\},\\alpha,\\mu\)\}\\end\{bmatrix\}\.\(11\)This profile summarizes how a fixed system degrades under different missingness types at a given severity and metric\. Instances infeasible for an operator are excluded from that operator’s aggregation, with feasibility details provided in the supplementary material\.

## Experiments

We evaluate whether MissDiag provides a more informative view of incomplete\-knowledge robustness than a single aggregate score\. Following the method design, the evaluation first compares typed degradation profiles across system families, then examines severity effects, metric sensitivity, and structural slices\. The experiments are organized around four research questions:

- •RQ1:Do missingness types produce distinct degradation profiles?
- •RQ2:How do profiles change as severity increases?
- •RQ3:Do profiles depend on the evaluation metric?
- •RQ4:When is answer\-adjacent degradation strongest?

### Experimental Setup

#### Data\.

We use KGQAGen\-10k\([46](https://arxiv.org/html/2608.18489#bib.bib1)\)as the main evaluation benchmark\. Its instances provide question–answer pairs and local evidence structures, which are converted into the support\-graph format used by MissDiag\. We link source entities from the question and align gold\-answer entities to graph nodes when available\. The main experiments use the development split\. Dataset\-specific support construction details are provided in the supplementary material\.

#### Evaluation protocol\.

We use a paired fixed\-input protocol\. For each instance, the question, gold answer set, source entities, and node inventory are kept fixed across complete and incomplete\-support conditions; only the retained support edges change\. For a fixed instance, missingness type, and severity, all systems are evaluated on the same transformed support graph\. The main cross\-system comparison uses 1,050 examples for which the complete\-support condition and all four missingness conditions are feasible\. LLM\-focused analyses use the corresponding feasible subset, with sample sizes reported in the relevant tables\.

#### Systems\.

We evaluate four system families: \(1\)Trained KGQA models, including ReaRev\([22](https://arxiv.org/html/2608.18489#bib.bib9)\), NuTrea\([4](https://arxiv.org/html/2608.18489#bib.bib49)\), and NSM\([13](https://arxiv.org/html/2608.18489#bib.bib43)\); \(2\)Graph prompting methods, including MindMap\([37](https://arxiv.org/html/2608.18489#bib.bib46)\), StructGPT\([16](https://arxiv.org/html/2608.18489#bib.bib44)\), and KG\-GPT\([17](https://arxiv.org/html/2608.18489#bib.bib45)\); \(3\)KG agents, including ToG\([32](https://arxiv.org/html/2608.18489#bib.bib34)\), PoG\([3](https://arxiv.org/html/2608.18489#bib.bib47)\), and GoG\([40](https://arxiv.org/html/2608.18489#bib.bib48)\); and \(4\)Direct LLM baselines, including Qwen2\.5\-7B\-Instruct111https://huggingface\.co/Qwen/Qwen2\.5\-7B\-Instruct, Qwen2\.5\-14B\-Instruct222https://huggingface\.co/Qwen/Qwen2\.5\-14B\-Instruct, Llama\-3\.1\-8B\-Instruct333https://huggingface\.co/meta\-llama/Llama\-3\.1\-8B\-Instruct, and Mistral\-7B\-Instruct\-v0\.3444https://huggingface\.co/mistralai/Mistral\-7B\-Instruct\-v0\.3\. For graph prompting methods and KG agents, all methods use Qwen2\.5\-7B\-Instruct as the shared LLM backbone across complete\-support and missingness conditions\. This controls for backbone capability and focuses the comparison on prompting, planning, and agent workflow\. For all LLM\-based systems, graph access is restricted to the transformed local support graph\.

#### Missingness conditions\.

We compare complete support with four typed missingness conditions: random, source\-context, relation\-level, and answer\-adjacent missingness\. Unless otherwise stated, the main comparison uses severityα=0\.3\\alpha=0\.3\. The severity analysis evaluatesα∈\{0\.1,0\.3,0\.5\}\\alpha\\in\\\{0\.1,0\.3,0\.5\\\}\.

#### Metrics and reporting\.

The primary metric is exact macro set\-level F1\. Results are reported as complete\-support F1 and paired degradationΔ\\DeltaF1, where positive values indicate performance loss after evidence removal and negative values indicate improvement\. Metric sensitivity is evaluated by comparing exact F1 with semantic F1 for direct LLM baselines\. Semantic F1 replaces exact string matching with semantic equivalence matching before computing set\-level precision and recall\. Additional low\-level details, including prompt templates and feasibility bookkeeping, are provided in the supplementary material\.

Table 1:Main typed degradation profiles across system families atα=0\.3\\alpha=0\.3\. Scores are reported as complete\-support macro F1 and pairedΔ\\DeltaF1 under each missingness type\. PositiveΔ\\DeltaF1 indicates degradation; negative values indicate improvement\.

### Implementation Details

Random missingness uses a fixed global seed, with per\-example seeds derived from the sample identifier, severity, and operator\. Non\-random operators use deterministic tie\-breaking when ordering candidate edges\. ReaRev, NuTrea, and NSM are trained once on complete\-support data using pinned official implementations, and their best\-F1 checkpoints are reused across all missingness conditions without condition\-specific retraining\. For LLM\-based systems, retained graph edges are serialized as textual triples and provided as the only graph evidence in the prompt\. Inference uses deterministic greedy decoding without sampling, with fixed generation and reasoning budgets across evidence conditions\. Experiments are conducted using NVIDIA GH200 GPUs; LLM\-based experiments use PyTorch 2\.9\.1, CUDA 12\.8, Transformers 4\.46\.2, and bfloat16 inference\. Additional implementation details are provided in the supplementary material\.

### RQ1: Typed Degradation Profiles Across Systems

Table[1](https://arxiv.org/html/2608.18489#Sx4.T1)reports the main cross\-system comparison on 1,050 paired development examples atα=0\.3\\alpha=0\.3\. Each row gives the complete\-support F1 of a system and its pairedΔ\\DeltaF1 under the four missingness conditions, directly instantiating the typed degradation profile defined in the method\. Three patterns are clear: First, answer\-adjacent removal is the dominant degradation condition for every system, with drops ranging from 10\.3 to 21\.3 F1 points\. This shows that answer\-local evidence loss is consistently more damaging than generic support removal across trained KGQA models, graph prompting methods, KG agents, and direct LLM baselines\. Second, source\-context removal behaves differently: it is small for several prompting and direct LLM systems, and negative for several trained KGQA models\. This indicates that removing source\-side context can sometimes reduce distracting evidence rather than harm prediction\. Third, random and relation\-level removal usually produce intermediate degradation, suggesting that support quantity and relation\-family loss affect performance but do not explain the full degradation pattern\.

Together, these results provide the main empirical support for typed degradation profiles: under the same paired protocol, incomplete\-support degradation depends on what kind of evidence is missing and how each system uses the remaining graph\. A single aggregate degradation score would collapse these distinct effects and obscure the difference between harmful answer\-local loss, neutral or beneficial source\-context removal, and intermediate random or relation\-level loss\.

### RQ2: Severity Effects

Figure[3](https://arxiv.org/html/2608.18489#Sx4.F3)reports severity\-dependent degradation for representative direct LLM baselines\. The severity parameter varies overα∈\{0\.1,0\.3,0\.5\}\\alpha\\in\\\{0\.1,0\.3,0\.5\\\}, while the question, gold answer, metric, and missingness operators remain fixed\. The curves show that increasing severity amplifies degradation, but not uniformly across missingness types\. Answer\-adjacent removal remains the strongest degradation condition at every severity and grows most sharply asα\\alphaincreases\. Random and relation\-level removal also increase with severity, but their degradation remains consistently below answer\-adjacent removal\. Source\-context removal stays comparatively small and changes little across severity levels\. These results show that the typed degradation pattern is not an artifact of the mainα=0\.3\\alpha=0\.3setting\. Severity controls the scale of evidence loss, while the relative behavior of missingness types remains structurally distinct\.

Figure 3:Severity effects on paired degradation\. Curves show pairedΔ\\DeltaF1 across missingness types for Qwen2\.5\-7B\-Instruct and Llama\-3\.1\-8B\-Instruct asα\\alphaincreases\.
### RQ3: Metric Sensitivity

Table[2](https://arxiv.org/html/2608.18489#Sx4.T2)compares paired degradation under exact F1 and semantic F1 for direct LLM baselines atα=0\.3\\alpha=0\.3\. This analysis tests whether the typed degradation profile changes when answer matching allows semantic equivalence rather than exact surface matching\. The results show that metric choice changes degradation magnitudes but not the main typed pattern\. For all four direct LLM baselines, answer\-adjacent removal remains the largest degradation condition under both exact F1 and semantic F1\. Semantic matching slightly changes individualΔ\\DeltaF1 values, but it does not reverse the ordering of missingness effects\. This indicates that the main diagnostic conclusion depends primarily on the type of missing evidence rather than on the particular answer\-matching rule\.

Table 2:Metric sensitivity of typed degradation profiles for direct LLM baselines atα=0\.3\\alpha=0\.3\. Rand, Src, Rel, and Ans\-adj denote random, source\-context, relation\-level, and answer\-adjacent missingness\.
### RQ4: Structural Slices

Table[3](https://arxiv.org/html/2608.18489#Sx4.T3)examines when answer\-adjacent degradation is strongest\. The analysis compares random and answer\-adjacent degradation across answer cardinality and support\-graph size, averaged over the four direct LLM baselines\. The gap is defined as answer\-adjacentΔ\\DeltaF1 minus randomΔ\\DeltaF1\.

Table 3:Structural slices for direct LLM baselines atα=0\.3\\alpha=0\.3\. Rand and Ans\-adj report pairedΔ\\DeltaF1 under random and answer\-adjacent missingness\. Gap is the difference between Ans\-adj and Rand degradation\.Answer\-adjacent degradation is most pronounced for multi\-answer questions and small\-support graphs\. Multi\-answer examples show a 19\.9\-point gap over random removal, compared with 7\.6 points for single\-answer examples\. The gap also decreases as support size grows, from 14\.2 points on small\-support graphs to 4\.5 points on large\-support graphs\. These results show that answer\-local evidence loss is especially harmful when answers are structurally harder to recover or when alternative support paths are limited\.

## Discussion

Taken together, the experiments show that incomplete\-knowledge robustness is better characterized by typed degradation profiles than by a single aggregate score\. The dominant effect comes from answer\-adjacent evidence loss, while severity, metric, and structural analyses clarify when this effect is amplified or preserved\. A system can appear robust under one missingness type while being highly sensitive to another\. This is most visible in the contrast between source\-context and answer\-adjacent removal\. Source\-context removal is often small or even beneficial, suggesting that some source\-side evidence may introduce distraction or increase the burden of evidence selection\. In contrast, answer\-adjacent removal consistently produces the largest degradation, indicating that systems depend strongly on evidence near the aligned answer entities\. These two effects would be collapsed by an aggregate missing\-evidence score, even though they imply different failure mechanisms\.

Typed profiles also make cross\-system comparison more interpretable\. Higher complete\-support F1 does not necessarily imply stronger robustness under all forms of missingness\. For example, direct LLM baselines achieve strong complete\-support performance but can still show large answer\-adjacent degradation\. Conversely, some trained KGQA models show negative degradation under source\-context removal, indicating that their behavior under incomplete evidence is not captured by complete\-support accuracy alone\. This highlights MissDiag’s diagnostic role in revealing how systems respond to different evidence losses, not only how much their average score changes\.

## Conclusion

This paper studies incomplete\-knowledge KGQA and KG\-RAG evaluation as a diagnostic problem\. A score drop under incomplete knowledge is not directly interpretable because it can conflate missing evidence, system response, and evaluation protocol\. MissDiag addresses this ambiguity by keeping the question, gold answer, source entities, and node inventory fixed while applying severity\-controlled typed missingness operators to support edges\. Across trained KGQA models, graph prompting methods, KG agents, and direct LLM baselines, answer\-adjacent evidence loss produces the most consistent degradation, whereas source\-context removal can be neutral or beneficial\. Severity, metric, and structural analyses further show when this typed pattern is preserved or amplified\. The central implication is methodological: incomplete\-knowledge robustness should be reported as typed degradation profiles, rather than as one aggregate score drop\.

## References

- Berantet al\.\(2013\)J\. Berant, A\. Chou, R\. Frostig, and P\. LiangSemantic parsing on freebase from question\-answer pairs\.InProceedings of the 2013 Conference on Empirical Methods in Natural Language Processing,pp\. 1533–1544\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p1.1),[KGQA and Graph\-Grounded QA](https://arxiv.org/html/2608.18489#Sx2.SSx1.p1.1)\.
- Caoet al\.\(2022\)S\. Cao, J\. Shi, L\. Pan, L\. Nie, Y\. Xiang, L\. Hou, J\. Li, B\. He, and H\. ZhangKQA pro: a dataset with explicit compositional programs for complex question answering over knowledge base\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 6101–6119\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p1.1),[KGQA and Graph\-Grounded QA](https://arxiv.org/html/2608.18489#Sx2.SSx1.p1.1)\.
- Chenet al\.\(2024\)L\. Chen, P\. Tong, Z\. Jin, Y\. Sun, J\. Ye, and H\. XiongPlan\-on\-graph: self\-correcting adaptive planning of large language model on knowledge graphs\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[Systems\.](https://arxiv.org/html/2608.18489#Sx4.SSx1.SSS0.Px3.p1.1)\.
- Choiet al\.\(2023\)H\. K\. Choi, S\. Lee, J\. Chu, and H\. J\. KimNutrea: neural tree search for context\-guided multi\-hop kgqa\.Advances in Neural Information Processing Systems36,pp\. 35954–35965\.Cited by:[Systems\.](https://arxiv.org/html/2608.18489#Sx4.SSx1.SSS0.Px3.p1.1)\.
- Dubeyet al\.\(2019\)M\. Dubey, D\. Banerjee, A\. Abdelkawi, and J\. LehmannLC\-quad 2\.0: a large dataset for complex question answering over wikidata and dbpedia\.InThe Semantic Web – ISWC 2019,pp\. 69–78\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p1.1),[KGQA and Graph\-Grounded QA](https://arxiv.org/html/2608.18489#Sx2.SSx1.p1.1)\.
- Gardneret al\.\(2020\)M\. Gardner, Y\. Artzi, V\. Basmov, J\. Berant, B\. Bogin, S\. Chen, P\. Dasigi, D\. Dua, Y\. Elazar, A\. Gottumukkala,et al\.Evaluating models’ local decision boundaries via contrast sets\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 1307–1323\.Cited by:[Benchmark Reliability and Evaluation Sensitivity](https://arxiv.org/html/2608.18489#Sx2.SSx3.p1.1)\.
- Guet al\.\(2021\)Y\. Gu, S\. Kase, M\. Vanni, B\. Sadler, P\. Liang, X\. Yan, and Y\. SuBeyond i\.i\.d\.: three levels of generalization for question answering on knowledge bases\.InProceedings of The Web Conference 2021,pp\. 3477–3488\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p1.1),[KGQA and Graph\-Grounded QA](https://arxiv.org/html/2608.18489#Sx2.SSx1.p1.1)\.
- Gu and Su \(2022\)Y\. Gu and Y\. SuArcaneQA: dynamic program induction and contextualized encoding for knowledge base question answering\.InProceedings of the 29th International Conference on Computational Linguistics,pp\. 1718–1731\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p1.1)\.
- Guoet al\.\(2023\)Q\. Guo, X\. Wang, Z\. Zhu, P\. Liu, and L\. XuA knowledge inference model for question answering on an incomplete knowledge graph\.Applied Intelligence53\(7\),pp\. 7634–7646\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1)\.
- Guuet al\.\(2020\)K\. Guu, K\. Lee, Z\. Tung, P\. Pasupat, and M\. ChangRetrieval augmented language model pre\-training\.InProceedings of the 37th International Conference on Machine Learning,pp\. 3929–3938\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p1.1),[KGQA and Graph\-Grounded QA](https://arxiv.org/html/2608.18489#Sx2.SSx1.p1.1)\.
- Hanet al\.\(2020\)J\. Han, B\. Cheng, and X\. WangOpen domain question answering based on text enhanced knowledge graph with hyperedge infusion\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 1475–1481\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1),[Incomplete\-Knowledge Question Answering](https://arxiv.org/html/2608.18489#Sx2.SSx2.p1.1)\.
- Hanet al\.\(2025\)R\. Han, J\. Liu, H\. Bi, T\. Peng, and L\. LiuSCR: a completion\-then\-reasoning framework for multi\-hop question answering over incomplete knowledge graph\.Neurocomputing,pp\. 131027\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1)\.
- Heet al\.\(2021\)G\. He, Y\. Lan, J\. Jiang, W\. X\. Zhao, and J\. WenImproving multi\-hop knowledge base question answering by learning intermediate supervision signals\.InProceedings of the Fourteenth ACM International Conference on Web Search and Data Mining,pp\. 553–561\.External Links:[Document](https://dx.doi.org/10.1145/3437963.3441753)Cited by:[KGQA and Graph\-Grounded QA](https://arxiv.org/html/2608.18489#Sx2.SSx1.p1.1),[Systems\.](https://arxiv.org/html/2608.18489#Sx4.SSx1.SSS0.Px3.p1.1)\.
- Huanget al\.\(2019\)X\. Huang, J\. Zhang, D\. Li, and P\. LiKnowledge graph embedding based question answering\.InProceedings of the Twelfth ACM International Conference on Web Search and Data Mining,pp\. 105–113\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p1.1),[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1)\.
- Jia and Liang \(2017\)R\. Jia and P\. LiangAdversarial examples for evaluating reading comprehension systems\.InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,pp\. 2021–2031\.Cited by:[Benchmark Reliability and Evaluation Sensitivity](https://arxiv.org/html/2608.18489#Sx2.SSx3.p1.1)\.
- Jianget al\.\(2023\)J\. Jiang, K\. Zhou, Z\. Dong, K\. Ye, W\. X\. Zhao, and J\. WenStructGPT: a general framework for large language model to reason over structured data\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 9237–9251\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.574)Cited by:[Systems\.](https://arxiv.org/html/2608.18489#Sx4.SSx1.SSS0.Px3.p1.1)\.
- Kimet al\.\(2023\)J\. Kim, Y\. Kwon, Y\. Jo, and E\. ChoiKG\-GPT: a general framework for reasoning on knowledge graphs using large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 9410–9421\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.631)Cited by:[Systems\.](https://arxiv.org/html/2608.18489#Sx4.SSx1.SSS0.Px3.p1.1)\.
- Lewiset al\.\(2020\)P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in Neural Information Processing Systems33,pp\. 9459–9474\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p1.1),[KGQA and Graph\-Grounded QA](https://arxiv.org/html/2608.18489#Sx2.SSx1.p1.1)\.
- Liuet al\.\(2026\)J\. Liu, P\. Shao, W\. Qin, F\. Liu, Y\. Yang, and R\. HongDebate over mixed\-knowledge: a robust multi\-agent reasoning framework for incomplete knowledge graph question answering\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 15333–15341\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1)\.
- Liuet al\.\(2022\)L\. Liu, B\. Du, J\. Xu, Y\. Xia, and H\. TongJoint knowledge graph completion and question answering\.InProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 1098–1108\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1)\.
- Maiet al\.\(2019\)G\. Mai, K\. Janowicz, B\. Yan, R\. Zhu, L\. Cai, and N\. LaoContextual graph attention for answering logical queries over incomplete knowledge graphs\.InProceedings of the 10th International Conference on Knowledge Capture,pp\. 171–178\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1),[Incomplete\-Knowledge Question Answering](https://arxiv.org/html/2608.18489#Sx2.SSx2.p1.1)\.
- Mavromatis and Karypis \(2022\)C\. Mavromatis and G\. KarypisReaRev: adaptive reasoning for question answering over knowledge graphs\.InFindings of the Association for Computational Linguistics: EMNLP 2022,pp\. 2447–2458\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p1.1),[KGQA and Graph\-Grounded QA](https://arxiv.org/html/2608.18489#Sx2.SSx1.p1.1),[Systems\.](https://arxiv.org/html/2608.18489#Sx4.SSx1.SSS0.Px3.p1.1)\.
- Perevalovet al\.\(2022\)A\. Perevalov, X\. Yan, L\. Kovriguina, L\. Jiang, A\. Both, and R\. UsbeckKnowledge graph question answering leaderboard: a community resource to prevent a replication crisis\.InProceedings of the Thirteenth Language Resources and Evaluation Conference,pp\. 2998–3007\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1),[Benchmark Reliability and Evaluation Sensitivity](https://arxiv.org/html/2608.18489#Sx2.SSx3.p1.1)\.
- Ribeiroet al\.\(2020\)M\. T\. Ribeiro, T\. Wu, C\. Guestrin, and S\. SinghBeyond accuracy: behavioral testing of nlp models with checklist\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 4902–4912\.Cited by:[Benchmark Reliability and Evaluation Sensitivity](https://arxiv.org/html/2608.18489#Sx2.SSx3.p1.1)\.
- Saxenaet al\.\(2022\)A\. Saxena, A\. Kochsiek, and R\. GemullaSequence\-to\-sequence knowledge graph completion and question answering\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2814–2828\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1)\.
- Saxenaet al\.\(2020\)A\. Saxena, A\. Tripathi, and P\. TalukdarImproving multi\-hop question answering over knowledge graphs using knowledge base embeddings\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 4498–4507\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1)\.
- Schlichtkrullet al\.\(2018\)M\. Schlichtkrull, T\. N\. Kipf, P\. Bloem, R\. van den Berg, I\. Titov, and M\. WellingModeling relational data with graph convolutional networks\.InThe Semantic Web,pp\. 593–607\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1)\.
- Shiet al\.\(2021\)J\. Shi, S\. Cao, L\. Hou, J\. Li, and H\. ZhangTransferNet: an effective and transparent framework for multi\-hop question answering over relation graph\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 4149–4158\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p1.1)\.
- Steinmetz and Sattler \(2021\)N\. Steinmetz and K\. SattlerWhat is in the kgqa benchmark datasets? survey on challenges in datasets for question answering on knowledge graphs\.Journal on Data Semantics10\(3\),pp\. 241–265\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1),[Benchmark Reliability and Evaluation Sensitivity](https://arxiv.org/html/2608.18489#Sx2.SSx3.p1.1)\.
- Suiet al\.\(2025\)Y\. Sui, Y\. He, Z\. Ding, and B\. HooiCan knowledge graphs make large language models more trustworthy? an empirical study over open\-ended question answering\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 12685–12701\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p1.1),[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1),[Benchmark Reliability and Evaluation Sensitivity](https://arxiv.org/html/2608.18489#Sx2.SSx3.p1.1)\.
- Sunet al\.\(2019a\)H\. Sun, T\. Bedrax\-Weiss, and W\. CohenPullNet: open domain question answering with iterative retrieval on knowledge bases and text\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),pp\. 2380–2390\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p1.1),[KGQA and Graph\-Grounded QA](https://arxiv.org/html/2608.18489#Sx2.SSx1.p1.1)\.
- Sunet al\.\(2024\)J\. Sun, C\. Xu, L\. Tang, S\. Wang, C\. Lin, Y\. Gong, L\. Ni, H\. Shum, and J\. GuoThink\-on\-graph: deep and responsible reasoning of large language model on knowledge graph\.InThe Twelfth International Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p1.1),[KGQA and Graph\-Grounded QA](https://arxiv.org/html/2608.18489#Sx2.SSx1.p1.1),[Systems\.](https://arxiv.org/html/2608.18489#Sx4.SSx1.SSS0.Px3.p1.1)\.
- Sunet al\.\(2023\)Q\. Sun, C\. Zhang, Z\. Hu, Z\. Jin, J\. Yu, and L\. LiuMulti\-hop question answering over incomplete knowledge graph with abstract conceptual evidence\.Applied Intelligence53\(21\),pp\. 25731–25751\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1)\.
- Sunet al\.\(2019b\)Z\. Sun, Z\. Deng, J\. Nie, and J\. TangRotatE: knowledge graph embedding by relational rotation in complex space\.arXiv preprint arXiv:1902\.10197\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1)\.
- Trouillonet al\.\(2016\)T\. Trouillon, J\. Welbl, S\. Riedel, É\. Gaussier, and G\. BouchardComplex embeddings for simple link prediction\.InProceedings of the 33rd International Conference on Machine Learning,pp\. 2071–2080\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1)\.
- Wanget al\.\(2024\)Y\. Wang, N\. Lipka, R\. A\. Rossi, A\. Siu, R\. Zhang, and T\. DerrKnowledge graph prompting for multi\-document question answering\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 19206–19214\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p1.1),[KGQA and Graph\-Grounded QA](https://arxiv.org/html/2608.18489#Sx2.SSx1.p1.1)\.
- Wenet al\.\(2024\)Y\. Wen, Z\. Wang, and J\. SunMindMap: knowledge graph prompting sparks graph of thoughts in large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 10370–10388\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.558),[Link](https://aclanthology.org/2024.acl-long.558/)Cited by:[Systems\.](https://arxiv.org/html/2608.18489#Sx4.SSx1.SSS0.Px3.p1.1)\.
- Xionget al\.\(2024\)G\. Xiong, J\. Bao, and W\. ZhaoInteractive\-kbqa: multi\-turn interactions for knowledge base question answering with large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 10561–10582\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p1.1),[KGQA and Graph\-Grounded QA](https://arxiv.org/html/2608.18489#Sx2.SSx1.p1.1)\.
- Xionget al\.\(2019\)W\. Xiong, M\. Yu, S\. Chang, X\. Guo, and W\. Y\. WangImproving question answering over incomplete kbs with knowledge\-aware reader\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 4258–4264\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1),[Incomplete\-Knowledge Question Answering](https://arxiv.org/html/2608.18489#Sx2.SSx2.p1.1)\.
- Xuet al\.\(2024\)Y\. Xu, S\. He, J\. Chen, Z\. Wang, Y\. Song, H\. Tong, G\. Liu, J\. Zhao, and K\. LiuGenerate\-on\-graph: treat LLM as both agent and KG for incomplete knowledge graph question answering\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 18410–18430\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1023)Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1),[Incomplete\-Knowledge Question Answering](https://arxiv.org/html/2608.18489#Sx2.SSx2.p1.1),[Systems\.](https://arxiv.org/html/2608.18489#Sx4.SSx1.SSS0.Px3.p1.1)\.
- Yeet al\.\(2022\)X\. Ye, S\. Yavuz, K\. Hashimoto, Y\. Zhou, and C\. XiongRNG\-kbqa: generation augmented iterative ranking for knowledge base question answering\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 6032–6043\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p1.1)\.
- Yeet al\.\(2024\)X\. Ye, L\. Xiao, C\. Zhang, and T\. YamasakiE\-rearev: adaptive reasoning for question answering over incomplete knowledge graphs by edge and meaning extensions\.InNatural Language Processing and Information Systems,pp\. 85–95\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1),[Incomplete\-Knowledge Question Answering](https://arxiv.org/html/2608.18489#Sx2.SSx2.p1.1)\.
- Yihet al\.\(2015\)W\. Yih, M\. Chang, X\. He, and J\. GaoSemantic parsing via staged query graph generation: question answering with knowledge base\.InProceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 1321–1331\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p1.1),[KGQA and Graph\-Grounded QA](https://arxiv.org/html/2608.18489#Sx2.SSx1.p1.1)\.
- Yuet al\.\(2023\)D\. Yu, Y\. Gu, C\. Xiong, and Y\. YangCompleQA: benchmarking the impacts of knowledge graph completion methods on question answering\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 12748–12755\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1),[Incomplete\-Knowledge Question Answering](https://arxiv.org/html/2608.18489#Sx2.SSx2.p1.1)\.
- Zanet al\.\(2022\)D\. Zan, S\. Wang, H\. Zhang, K\. Zhou, W\. Wu, W\. X\. Zhao, B\. Wu, B\. Guan, and Y\. WangComplex question answering over incomplete knowledge graph as n\-ary link prediction\.In2022 International Joint Conference on Neural Networks \(IJCNN\),pp\. 1–8\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1)\.
- Zhanget al\.\(2025\)L\. Zhang, Z\. Jiang, H\. Chi, H\. Chen, M\. ElKoumy, F\. Wang, Q\. Wu, Z\. Zhou, S\. Pan, S\. Wang, and Y\. MaDiagnosing and addressing pitfalls in kg\-rag datasets: toward more reliable benchmarking\.InNeurIPS 2025 Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=Vd5JXiX073)Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p1.1),[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1),[KGQA and Graph\-Grounded QA](https://arxiv.org/html/2608.18489#Sx2.SSx1.p1.1),[Benchmark Reliability and Evaluation Sensitivity](https://arxiv.org/html/2608.18489#Sx2.SSx3.p1.1),[Data\.](https://arxiv.org/html/2608.18489#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026\)L\. Zhang, Z\. Jiang, H\. Chi, H\. Chen, M\. Elkoumy, F\. Wang, Q\. Wu, Z\. Zhou, S\. Pan, S\. Wang,et al\.Diagnosing and addressing pitfalls in kg\-rag datasets: toward more reliable benchmarking\.Advances in Neural Information Processing Systems38\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1),[Benchmark Reliability and Evaluation Sensitivity](https://arxiv.org/html/2608.18489#Sx2.SSx3.p1.1)\.
- Zhaoet al\.\(2022\)F\. Zhao, Y\. Li, J\. Hou, and L\. BaiImproving question answering over incomplete knowledge graphs with relation prediction\.Neural Computing and Applications34\(8\),pp\. 6331–6348\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1)\.
- Zhouet al\.\(2026\)D\. Zhou, Y\. Zhu, X\. Wang, H\. Zhou, Y\. He, J\. Chen, S\. Staab, and E\. KharlamovWhat breaks knowledge graph based rag? benchmarking and empirical insights into reasoning under incomplete knowledge\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2522–2538\.Cited by:[Introduction](https://arxiv.org/html/2608.18489#Sx1.p1.1),[Introduction](https://arxiv.org/html/2608.18489#Sx1.p2.1),[Incomplete\-Knowledge Question Answering](https://arxiv.org/html/2608.18489#Sx2.SSx2.p1.1),[Benchmark Reliability and Evaluation Sensitivity](https://arxiv.org/html/2608.18489#Sx2.SSx3.p1.1)\.