Universal Pathologies, Conditional Consequences: A Triple-Robustness Analysis of RAG for Multi-Hop Traceability

# A Triple-Robustness Analysis of RAG for Multi-Hop Traceability
Source: [https://arxiv.org/html/2608.05153](https://arxiv.org/html/2608.05153)
## Universal Pathologies, Conditional Consequences: A Triple\-Robustness Analysis of RAG for Multi\-Hop Traceability

andBurak ÖzdemirIndependent ResearcherAnkaraTurkey[bozdemir@gmail\.com](https://arxiv.org/html/2608.05153v1/mailto:[email protected])

###### Abstract\.

GraphRAG underperforms vector RAG on citation precision in many reports, but*where*and*why*have remained corpus\-bound\. We present a triple\-robustness analysis that holds the retrieval architecture fixed and varies three orthogonal axes —embedder\(local e5\-small→\\toAzure text\-embedding\-3\-small\),corpus\(DO\-178C typed\-edge requirements→\\toWikipedia paragraph chains via MuSiQue\), andjudge\(paired GPT\-5\.4×\\timesGPT\-4\.1\) — across 4,440 main\-matrix runs, 600 cross\-corpus runs, and 1,200 paired faithfulness judgments\.\(C2a\)Over\-citation is architecturally universal: GraphRAG emits 11–15 IDs per answer at citation precision 0\.12–0\.23 and retrieval recall 0\.68–0\.87 across all three settings\.\(C2b\)Its faithfulness consequence is corpus\-conditional: in typed\-edge DO\-178C, GraphRAG faithfulness collapses 74%→\\to40% across hops; on Wikipedia chains the same pipeline rises 42%→\\to58% because over\-cited paragraphs remain topically supporting\.\(C1\)Stratum\-conditional winners are corpus\-conditional but embedder\-robust: vanilla wins 2\-hop on DO\-178C, GraphRAG wins 2\-hop on MuSiQue, identical under either embedder\.\(C3\)Single\-judge LLM faithfulness is fragile to retrieval state: same\-judge self\-κ\\kappaacross embedders is0\.1370\.137for GPT\-5\.4 \(verdict change on 41% of items\)\. A learned router on dense embeddings alone reaches macro\-F1F\_\{1\}0\.860\.86on hop classification\(C4\)\. We argue triple\-robustness is the minimum bar for trustworthy RAG architecture claims\.

retrieval\-augmented generation, GraphRAG, requirements traceability, DO\-178C, MuSiQue, LLM\-as\-judge, kappa paradox, triple\-robustness, embedder fragility

††copyright:none††conference:Proceedings of the 35th ACM International Conference on Information and Knowledge Management; November 7–11, 2026; Rome, Italy††journalyear:2026††ccs:Information systems Retrieval models and ranking††ccs:Computing methodologies Question answering††ccs:Software and its engineering Requirements analysis## 1\.Introduction

DO\-178C\-style aerospace requirements are authored as typed link graphs: each requirementderives\_fromparents,satisfiessystem\-level intent, andtraces\_toverification artifacts\. Certification authorities increasingly require auditable chains spanning two or more hops\(RTCA, Inc\.,[2011](https://arxiv.org/html/2608.05153#bib.bib24); SAE International,[2010](https://arxiv.org/html/2608.05153#bib.bib25); European Union Aviation Safety Agency \(EASA\),[2025](https://arxiv.org/html/2608.05153#bib.bib26); Federal Aviation Administration,[2024](https://arxiv.org/html/2608.05153#bib.bib27)\)\. RAG architectures for this task have produced contradictory verdicts: Microsoft GraphRAG\(Edgeet al\.,[2024](https://arxiv.org/html/2608.05153#bib.bib6)\)exploits structure but under\-performs vector RAG on detailed queries\(Hanet al\.,[2025](https://arxiv.org/html/2608.05153#bib.bib11); Xianget al\.,[2026](https://arxiv.org/html/2608.05153#bib.bib10)\); Adaptive\-RAG\(Jeonget al\.,[2024](https://arxiv.org/html/2608.05153#bib.bib3)\)routes by complexity yet is hop\-blind; LiSSA\(Fuchßet al\.,[2025](https://arxiv.org/html/2608.05153#bib.bib12)\)and TVR\(Niuet al\.,[2025](https://arxiv.org/html/2608.05153#bib.bib13)\)omit typed\-graph reasoning\. The literature describes*what*sometimes works in*which*setting; what is missing is a mechanism\-level account of*why*the verdicts shift\.

We address this with atriple\-robustnessanalysis\. Holding a five\-pipeline architecture matrix fixed \(vanilla, agentic, agentic\+graph, GraphRAG, learned adaptive\), we vary three orthogonal axes: the retrieval embedder \(e5\-small 384d→\\toAzure 3\-small 1536d\), the corpus \(DO\-178C→\\toWikipedia paragraph chains via MuSiQue mapped to 1/2/3\+\-hop strata\), and the faithfulness judge \(paired GPT\-5\.4\+\+GPT\-4\.1; single GPT\-5\.4 on MuSiQue\)\. The design produces 4,440 main\-matrix runs, 600 cross\-corpus runs, and 1,200 paired judgments\.

We make four contributions\.\(C2a\)Over\-citation is an*architectural universal*: GraphRAG cites 11–15 IDs at citation precision 0\.12–0\.23 with retrieval recall 0\.68–0\.87 across all three settings \(Table[2](https://arxiv.org/html/2608.05153#S5.T2)\)\.\(C2b\)Its faithfulness consequence is*corpus\-conditional*: DO\-178C collapses 74%→\\to40%, MuSiQue rises 42%→\\to58%\.\(C1\)Stratum\-conditional winners are*corpus\-conditional but embedder\-robust*\(Table[1](https://arxiv.org/html/2608.05153#S5.T1)\): vanilla wins 2\-hop on DO\-178C; GraphRAG wins 2\-hop on MuSiQue\.\(C3\)*Single\-judge faithfulness is embedder\-fragile*: GPT\-5\.4 self\-κ\\kappaacross embedders is just0\.1370\.137on the paired 300\-tuple subset \(Table[3](https://arxiv.org/html/2608.05153#S5.T3)\)\.\(C4, supporting\)a logistic\-regression router on dense embeddings alone reaches macro\-F1F\_\{1\}0\.860\.86on hop classification\.

§[2](https://arxiv.org/html/2608.05153#S2)surveys related work; §[3](https://arxiv.org/html/2608.05153#S3)the method; §[4](https://arxiv.org/html/2608.05153#S4)the protocol; §[5](https://arxiv.org/html/2608.05153#S5)results; §[6](https://arxiv.org/html/2608.05153#S6)threats\.

## 2\.Related Work

Agentic and Adaptive RAG\.Self\-RAG\(Asaiet al\.,[2024](https://arxiv.org/html/2608.05153#bib.bib1)\)and CRAG\(Yanet al\.,[2024](https://arxiv.org/html/2608.05153#bib.bib2)\)introduced reflective and corrective retrieval\. Adaptive\-RAG\(Jeonget al\.,[2024](https://arxiv.org/html/2608.05153#bib.bib3)\)routes by predicted query complexity; Probing\-RAG\(Baeket al\.,[2025](https://arxiv.org/html/2608.05153#bib.bib30)\)extends with internal\-state probes; Search\-R1\(Jinet al\.,[2025](https://arxiv.org/html/2608.05153#bib.bib5)\)trains agentic retrieval via RL; RAG\-Critic\(Donget al\.,[2025](https://arxiv.org/html/2608.05153#bib.bib4)\)adds an iterative critic\. None condition routing on typed\-graph hop distance or expose a typed graph\-lookup tool\.

GraphRAG and multi\-hop QA\.Microsoft GraphRAG\(Edgeet al\.,[2024](https://arxiv.org/html/2608.05153#bib.bib6)\)and follow\-ups\(Guoet al\.,[2024](https://arxiv.org/html/2608.05153#bib.bib7); Gutiérrezet al\.,[2025](https://arxiv.org/html/2608.05153#bib.bib8); Luoet al\.,[2025](https://arxiv.org/html/2608.05153#bib.bib9)\)construct entity\-relation graphs at indexing time\. Han et al\.\(Hanet al\.,[2025](https://arxiv.org/html/2608.05153#bib.bib11)\)and GraphRAG\-Bench\(Xianget al\.,[2026](https://arxiv.org/html/2608.05153#bib.bib10)\)report that GraphRAG*does not*dominate vector RAG across all query types — our C2a confirms this with quantified over\-citation metrics in safety\-critical RE, and C2b localises*why*\(mechanism vs\. consequence\) across corpora\. MuSiQue\(Trivediet al\.,[2022](https://arxiv.org/html/2608.05153#bib.bib20)\)and MultiHop\-RAG\(Tang and Yang,[2024](https://arxiv.org/html/2608.05153#bib.bib33)\)are adjacent multi\-hop benchmarks; we use MuSiQue for cross\-corpus replication of C2a only\.

RAG for requirements traceability\.LiSSA\(Fuchßet al\.,[2025](https://arxiv.org/html/2608.05153#bib.bib12)\), TVR\(Niuet al\.,[2025](https://arxiv.org/html/2608.05153#bib.bib13)\), and Graph\-RAG for compliance\(Masoudifardet al\.,[2024](https://arxiv.org/html/2608.05153#bib.bib14)\)evaluate single regulated domains without hop stratification or dual\-judge protocols\. Our triple\-robustness design is the natural completion of this thread\.

Citation evaluation, judge bias, and theκ\\kappaparadox\.ALCE\(Gaoet al\.,[2023](https://arxiv.org/html/2608.05153#bib.bib31)\)introduced citationP/R/F1P/R/F\_\{1\}for grounded generation; Wallat et al\.\(Wallatet al\.,[2024](https://arxiv.org/html/2608.05153#bib.bib32)\)show that ALCE\-style correctness is not faithfulness, motivating LLM judges\. RAGChecker\(Ruet al\.,[2024](https://arxiv.org/html/2608.05153#bib.bib17)\)provides a single\-judge framework\. Self\-preference bias\(Wataokaet al\.,[2024](https://arxiv.org/html/2608.05153#bib.bib19)\)motivates judge ensembles, and the Feinstein–Cicchettiκ\\kappaparadox\(Feinstein and Cicchetti,[1990](https://arxiv.org/html/2608.05153#bib.bib34)\)with Gwet’s AC1\(Gwet,[2008](https://arxiv.org/html/2608.05153#bib.bib35)\)establishes the prevalence correction we apply\. Our C3 extends this with same\-judge self\-κ\\kappaacross embedders as a sharper fragility statistic\.

Statistical protocol\.Berg\-Kirkpatrick et al\.\(Berg\-Kirkpatricket al\.,[2012](https://arxiv.org/html/2608.05153#bib.bib22)\)and Koehn\(Koehn,[2004](https://arxiv.org/html/2608.05153#bib.bib23)\)popularised paired permutation tests; we use BCa bootstrap CIs with Holm\-corrected Wilcoxon and Cliff’sδ\\delta\.

## 3\.Method

QueryqqHop\-Adaptive RouterpredictsH^​\(q\)∈\{1,2,3\+\}\\widehat\{H\}\(q\)\\in\\\{1,2,3\{\+\}\\\}Vanillaembed→\\totop\-kk→\\toLLMAgenticLangGraph loopGraphRAGseeds \+ 2\-hop walkAgentic\+GraphLangGraph \+ graphRouter/CriticRetrieverToolssearch, graphSynthesizeriter≤3\\leq 3Agentic LoopusesusesAnswera​\(q\)a\(q\)\+ cited IDsC​\(q\)C\(q\)Vector Store\(Chroma\)Graph Store\(Neo4j Aura\)→\\toP1–P4→\\toP3, P4Figure 1\.Five\-pipeline architecture with shared embedder, vector store, and typed\-edge graph\. Vanilla and GraphRAG bypass the agentic loop; Agentic and Agentic\+Graph route through router/critic–retriever–tools \(iter≤3\{\\leq\}3\) before synthesis\. The adaptive pipeline \(V1 rule\-based or V2 learned\) selects one of the four base pipelines per query\.Block diagram of the five\-pipeline RAG architecture: a query enters a hop\-adaptive router that selects one of four base pipelines \(Vanilla, Agentic, GraphRAG, Agentic\+Graph\)\. The Agentic and Agentic\+Graph pipelines invoke an inner Agentic Loop comprising a router/critic, retriever, tools \(search and graph\), and a synthesizer that iterates up to three times\. Outputs combine into an answer with cited IDs\. A shared vector store \(Chroma\) feeds P1–P4 and a shared graph store \(Neo4j Aura\) feeds P3 and P4\.### 3\.1\.Pipelines

All pipelines share embedder, ChromaDB vector store, Neo4j typed\-edge graph, Azure GPT\-5\.4 generator, and grounded synthesis prompt; they differ only in retrieval\.\(i\) Vanilla: dense top\-kk\+ one synthesis call\.\(ii\) Agentic: LangGraph router\-retriever\-critic loop,search\_documentstool only, iter cap 3\.\(iii\) GraphRAG: vector seeds \+ up\-to\-2\-hop typed\-graph walk via Cypher, single synthesis\.\(iv\) Agentic\+Graph: \(ii\) plus a typed\-edgegraph\_lookuptool\.\(v\) Adaptive: chooses among \(i\)–\(iv\) per query; we evaluate a rule\-based router \(V1\) and a learned out\-of\-fold logistic regression \(V2\)\.

### 3\.2\.Triple\-Robustness Axes

Embedder:local intfloat/multilingual\-e5\-small \(384d\) vs\. Azuretext\-embedding\-3\-small\(1536d\); ChromaDB collections rebuilt per embedder, Neo4j shared\.Corpus:a 1,132\-requirement DO\-178C\-style aerospace certification corpus across 32 modules, LLM\-anonymized from proprietary source artifacts for release \(§[GenAI Usage Disclosure](https://arxiv.org/html/2608.05153#Sx1)\); and a 200\-query MuSiQue\(Trivediet al\.,[2022](https://arxiv.org/html/2608.05153#bib.bib20)\)subset \(67/67/66 across 2/3/4\-hop, mapped to our 1/2/3\+\-hop strata,REFERENCESedges only between consecutive supporting paragraphs\)\.Judge:GPT\-5\.4\+\+GPT\-4\.1 on the DO\-178C main matrix; single\-judge GPT\-5\.4 on MuSiQue\.

### 3\.3\.Metrics and Judging

We report ALCE\-style citationP/R/F1P/R/F\_\{1\}\(Gaoet al\.,[2023](https://arxiv.org/html/2608.05153#bib.bib31)\)against gold requirement\-ID sets and retrieval recall on retrieved contexts\. Each faithfulness judgment is a strict\-JSON binary verdict over the retrieved\-context block\. We report Cohen’sκ\\kappa, Gwet’s AC1\(Gwet,[2008](https://arxiv.org/html/2608.05153#bib.bib35); Feinstein and Cicchetti,[1990](https://arxiv.org/html/2608.05153#bib.bib34)\), raw agreement, McNemar exact\-binomialpp, and Spearmanρ\\rhoon per\-pipeline ranking; we further report same\-judge self\-κ\\kappaacross embedders on a paired 300\-tuple subset \(§[5\.3](https://arxiv.org/html/2608.05153#S5.SS3)\)\.

### 3\.4\.V2 Learned Router

14 features per query: 11 hand text features \(ID regex, keyword flags, log token length\) and 3 PCs of the query embedding \(fit per fold\)\. Multinomial logistic regression,L2L\_\{2\}\(C=1C\{=\}1\), Platt\-calibrated, under stratified 10\-fold×\\times5\-repeat CV \(50 fits\)\. The per\-stratum routing target is the empirical mean\-F1F\_\{1\}winner from the locked main matrix; we report out\-of\-fold predictions for all 296 queries\.

## 4\.Experimental Setup

Main matrix and cross\-corpus sample\.DO\-178C: 5 pipelines×\\times296 hop\-stratified queries×\\times3 seeds=4,440=4\{,\}440runs, executed under both embedders \(*v2*,*v3*\)\. MuSiQue: 3 pipelines \(vanilla, GraphRAG, agentic\+graph\)×\\times200 queries×\\times1 seed=600=600runs, Azure embedder only\. Agentic and adaptive are not re\-run on MuSiQue \(C2a needs only the three\-pipeline trio\)\.

Faithfulness judging\.On DO\-178C, multi\-judge protocol \(GPT\-5\.4\+\+GPT\-4\.1\) is applied to a 300\-row stratified subset \(60 per pipeline, seed 42\), yielding 600 paired binary judgments per embedder\. The v3 judged subset is explicitly*pinned*to the same 300 \(query, pipeline, repeat\) tuples as v2 so C3 deltas are computed on identical items\. On MuSiQue, we single\-judge all 600 with GPT\-5\.4\.

Statistical protocol\.95% BCa bootstrap intervals\(Du,[2025](https://arxiv.org/html/2608.05153#bib.bib21)\)\(B=1000B\{=\}1000, paired at the query level\), Wilcoxon signed\-rank with Holm correction across the 30 pipeline\-pair×\\timesstratum contrasts, and Cliff’sδ\\delta\(negligible<0\.147\{<\}0\.147; Macbeth 2011\)\. A pipeline pair is reported as significantly different only when the BCa interval excludes zero, Holmp<0\.05p\{<\}0\.05, and\|δ\|≥0\.147\|\\delta\|\\\!\\geq\\\!0\.147jointly hold\.

Reproducibility\.The full pipeline regenerates from oneMakefiletarget on locked CSVs; seeds, sampling protocols, and Azure model snapshots are in supplementary materials\. The MuSiQue subgraph \(3,996 paragraph chunks, 399REFERENCESedges\) is built deterministically fromdgslibisey/MuSiQue\(validation split, seed2026051120260511\)\.

## 5\.Results

Table 1\.Per\-stratum citationF1F\_\{1\}across three settings\. v2 main = DO\-178C corpus with local e5\-small embeddings \(4,440 runs\); v3 main = same corpus with Azure text\-embedding\-3\-small \(4,440 runs\); MuSiQue = 200\-query Wikipedia stratified subset with Azure embedder \(600 runs, vanilla / agentic\-graph / graphrag only\)\. Bold marks the per\-\(setting, stratum\) maximum\.![Refer to caption](https://arxiv.org/html/2608.05153v1/x1.png)Figure 2\.Per\-stratum citationF1F\_\{1\}on the v2 main matrix \(DO\-178C, local e5\-small embedder\)\. Bars: the five pipelines across the 1\-/2\-/3\+\-hop strata with BCa 95% CIs;∗\*marks pipelines significantly different from the per\-stratum winner \(Holm\-Wilcoxonp<0\.05p\{<\}0\.05,\|δ\|≥0\.147\|\\delta\|\{\\geq\}0\.147\)\. Cross\-corpus and cross\-embedder comparisons are in Table[1](https://arxiv.org/html/2608.05153#S5.T1)\.Grouped bar chart of per\-stratum citation F1 for five RAG pipelines \(vanilla, agentic, agentic\-graph, graphrag, adaptive\) on the DO\-178C v2 main matrix, broken down by 1\-hop, 2\-hop, and 3\+\-hop strata, with error bars and asterisks marking pipelines significantly different from the per\-stratum winner\.### 5\.1\.C1: Winners are Corpus\-Conditional, Embedder\-Robust

Table[1](https://arxiv.org/html/2608.05153#S5.T1)reports per\-stratumF1F\_\{1\}across three settings\. On DO\-178C the dominance pattern is identical under both embedders: agentic\-graph wins 1\-hop and 3\+\-hop, vanilla wins 2\-hop\. The 1\-hop and 3\+\-hop wins do not separate agentic\-graph from agentic \(Cliff’sδ<0\.147\\delta\{<\}0\.147in every stratum\) and are read as an \{agentic\-graph, agentic\} tie\. On MuSiQue the winners shift: agentic\-graph retains 1\-hop, but GraphRAG wins both 2\-hop \(F1=0\.375F\_\{1\}\{=\}0\.375\) and 3\+\-hop \(F1=0\.342F\_\{1\}\{=\}0\.342\), reversing vanilla’s 2\-hop dominance\. The pattern is corpus\-conditional, not universal\.

Table 2\.GraphRAG triple\-robustness: \(top, C2a\) over\-citation mechanism is universal across embedder and corpus swaps; \(bottom, C2b\) faithfulness consequence is corpus\-conditional — collapses on DO\-178C but stays flat/rises on MuSiQue\.C2a: over\-citation mechanismv2 mainv3 mainMuSiQueMean IDs cited14\.914\.911\.2Citation precision0\.1200\.1290\.227Retrieval recall0\.6810\.7240\.873CitationF1F\_\{1\}overall0\.2020\.2170\.358C2b: faithfulness consequence1\-hop2\-hop3\+\-hopv2 main \(GPT\-5\.4\)0\.740\.640\.40v3 main pinned \(GPT\-5\.4\)0\.520\.550\.40MuSiQue \(GPT\-5\.4\)0\.420\.540\.58![Refer to caption](https://arxiv.org/html/2608.05153v1/x2.png)Figure 3\.Multi\-judge faithfulness on the v2 main matrix\.Left: faithfulness fraction \(5 pipelines×\\times3 hop strata×\\times2 judges\); the GraphRAG row exhibits the monotonic collapse \(74%→64%→40%74\\%\{\\to\}64\\%\{\\to\}40\\%under GPT\-5\.4\) that drives C2b on DO\-178C\.Right: per\-stratum Gwet AC1, surfacing theκ\\kappaparadox at the 3\+\-hop stratum \(AC1=0\.57\{=\}0\.57at high prevalence; cf\. Table[3](https://arxiv.org/html/2608.05153#S5.T3)\)\. v3 and MuSiQue panels are reported in Table[2](https://arxiv.org/html/2608.05153#S5.T2)\.Heatmap of faithfulness percentages for five RAG pipelines \(rows: vanilla, agentic, agentic\-graph, graphrag, adaptive\) across three hop strata times two judges \(columns: 1\-hop GPT\-5\.4, 1\-hop GPT\-4\.1, 2\-hop GPT\-5\.4, 2\-hop GPT\-4\.1, 3\+\-hop GPT\-5\.4, 3\+\-hop GPT\-4\.1\) on the v2 main matrix\. The graphrag row darkens monotonically left\-to\-right under both judges\. A side panel shows per\-stratum Gwet AC1 bars\.
### 5\.2\.C2: GraphRAG’s Universal Pathology, Conditional Consequence

\(C2a\) Over\-citation is architectural\.The top block of Table[2](https://arxiv.org/html/2608.05153#S5.T2)establishes universality\. Across embedder*and*corpus swaps GraphRAG cites a mean of 11–15 IDs per answer at citation precision 0\.12–0\.23 with retrieval recall in the 0\.68–0\.87 range\. This is not a corpus\-specific artifact — the same mechanism manifests on Wikipedia paragraph chains with a different gold structure — and converges with prior cross\-domain observations\(Hanet al\.,[2025](https://arxiv.org/html/2608.05153#bib.bib11); Xianget al\.,[2026](https://arxiv.org/html/2608.05153#bib.bib10); Wallatet al\.,[2024](https://arxiv.org/html/2608.05153#bib.bib32)\)\.

\(C2b\) The faithfulness consequence is corpus\-conditional\.The bottom block reveals the consequence split\. On DO\-178C with local embedder, GraphRAG faithfulness collapses monotonically across hops \(74%/64%/40%\)\. Under embedder swap it flattens to≈50%\{\\approx\}50\\%\. On MuSiQue it*rises*monotonically \(42%/54%/58%\)\. The mechanistic reading: on typed\-edge regulated corpora, over\-cited IDs are adversarial distractors whose claims contradict the question; on Wikipedia paragraph chains over\-cited paragraphs remain topically related to the question and typically satisfy the judge’s support criterion\. Over\-citation is therefore architecturally universal but consequentially domain\-bound — a constraint on inferring faithfulness from citation precision alone\.

Table 3\.Multi\-judge protocol fragility\. Top: inter\-judge agreement \(GPT\-5\.4 vs GPT\-4\.1\) drops by half overall under embedder swap \(v2 local→\\tov3 Azure\) on the same 300 paired tuples\. Bottom: same\-judge self\-consistency across embedders — GPT\-5\.4 changes verdict on 41% of items when only the retrieval embedding changes; GPT\-4\.1 is more stable\.
### 5\.3\.C3: Single\-Judge Faithfulness is Embedder\-Fragile

Table[3](https://arxiv.org/html/2608.05153#S5.T3)reports inter\-judge agreement on the paired 300\-tuple subset and same\-judge self\-κ\\kappaacross embedders\. Inter\-judgeκ\\kappabetween GPT\-5\.4 and GPT\-4\.1 halves under embedder swap on identical items \(0\.30→\\to0\.17 overall\)\. Per\-stratum Gwet AC1 surfaces theκ\\kappaparadox at the 3\+\-hop stratum \(AC1=0\.57\{=\}0\.57in v2,0\.350\.35in v3, withκ<0\.10\\kappa\{<\}0\.10in both\)\. The headline is the same\-judge row:*the same judge reading the same question and pipeline output changes verdict on 41% of items when only the retrieval embedding changes*\(κGPT\-5\.4=0\.14\\kappa\_\{\\text\{GPT\-5\.4\}\}\{=\}0\.14\)\. GPT\-4\.1 is materially more stable \(κ=0\.48\\kappa\{=\}0\.48\)\. To our knowledge this is the first quantification of single\-judge fragility to upstream retrieval state in a controlled paired design\.

Table 4\.V2 router replication and Azure\-embedder boost\. Same 14\-feature schema \(11 hand text features, 3 PCs of the query embedding\); only the embedder differs\. Text features remain mostly inert \(4/11 non\-zero\), so the \+0\.08 macro\-F1F\_\{1\}gain is attributable to dense\-embedding hop\-decodability\.#### C4 \(supporting\): Dense embeddings alone classify hop\.

Table[4](https://arxiv.org/html/2608.05153#S5.T4)shows the V2 router replication\. The 50\-fold macro\-F1F\_\{1\}improves0\.78±0\.08→0\.86±0\.050\.78\\pm 0\.08\\to 0\.86\\pm 0\.05under embedder swap, and Adaptive\-V2 closes 56% of the V1→\\toOracle gap with Holm\-significant gains in all three strata under both embedders\. Of 11 hand text features only four \(kw\_which,kw\_satisfies,kw\_references,kw\_verifies\) are ever non\-zero on the 296 queries; the rest are inert\. The router therefore operates almost exclusively on the 3 query\-embedding PCs, suggesting hop distance is largely an embedding\-decodable property of the query text in regulated\-domain corpora\.

## 6\.Discussion

#### Why does over\-citation’s consequence depend on corpus?

GraphRAG’s graph traversal expands candidates beyond what dense retrieval surfaces; the synthesizer then dutifully cites every ID it touches\. On DO\-178C, typed edges connect requirements with adversarial specificity:derives\_fromexpansion drags in unrelated parent requirements;referencesexpansion pulls in cross\-module artifacts whose claims contradict the question — the judge marks the resulting citations unsupported\. On Wikipedia paragraph chains the over\-cited content is by construction topically adjacent; the judge typically finds support in the expanded context even when many cites are extraneous\. The same retrieval behaviour therefore has different faithfulness consequences\. We predict the same coupling in any regulated\-domain RAG that exposes typed\-edge expansion over an adversarial gold structure\.

#### Implications for LLM\-judge practice\.

Same\-judge self\-κ=0\.137\\kappa\{=\}0\.137across embedders means: comparing faithfulness across retrieval modules with different embedders, using a single judge, reports embedder\-conditional verdicts, not invariant faithfulness\. Multi\-judge ensembles mitigate only partially: inter\-judgeκ\\kappaitself halves under embedder swap\. Triple\-robustness, or at minimum embedder\-controlled paired\-tuple judging, is the minimum defensible bar for LLM\-judge faithfulness\.

#### Threats to validity\.

Construct\.ALCE\-styleF1F\_\{1\}does not measure rationale quality; we pair it with multi\-judge faithfulness, while acknowledging LLM\-judge bias\(Wataokaet al\.,[2024](https://arxiv.org/html/2608.05153#bib.bib19); Wallatet al\.,[2024](https://arxiv.org/html/2608.05153#bib.bib32)\)\. C3 is the explicit reckoning\.External\.The DO\-178C corpus is a single proprietary aerospace certification dataset that has been LLM\-anonymized for release because the raw source artifacts are subject to IP and export\-control restrictions \(§[GenAI Usage Disclosure](https://arxiv.org/html/2608.05153#Sx1)\); the absoluteF1F\_\{1\}levels reported here therefore characterize this single regulated\-domain setting and may not transfer to other proprietary or open\-domain projects\. The MuSiQue cross\-corpus replication \(C2a\) is the load\-bearing external\-validity check for the architectural\-universal claim; cross\-corpus replication of C1 and C2b is left for the journal extension\.Statistical\.Per\-stratumn∈\[95,111\]n\\\!\\in\\\!\[95,111\]is at the lower edge of BCa stability under heavy skew; we report Cliff’sδ\\deltaalongside intervals and observed no qualitative reversal under a percentile fallback\.Reranker / router engineering\.No cross\-encoder reranker is included in the main pipelines \(orthogonal axis, future work\); C4 is framed as a diagnostic instrument revealing hop\-class decodability, not a recommended production routing system\.

## 7\.Conclusion

We presented a triple\-robustness analysis \(embedder×\\timescorpus×\\timesjudge\) of five RAG architectures for multi\-hop requirements traceability\. The empirical picture is sharper than either uniform endorsement or uniform critique of GraphRAG\.*Over\-citation is architecturally universal*;*its faithfulness consequence is corpus\-conditional*;*stratum\-conditional winners are corpus\-conditional but embedder\-robust*; and*single\-judge faithfulness verdicts are embedder\-fragile in a way multi\-judge protocols only partially mitigate*\. We argue that triple\-robustness is the minimum defensible bar for RAG architecture claims and that a learned hop router on dense embeddings alone is a useful diagnostic instrument for characterising the structure of the problem\.

## GenAI Usage Disclosure

We disclose all uses of generative AI in this work in accordance with the ACM Policy on Authorship and the CIKM 2026 disclosure requirement\.\(a\) Corpus anonymization\.The DO\-178C\-style corpus used in this paper is derived from proprietary aerospace certification artifacts that cannot be released directly due to intellectual\-property and export\-control restrictions\. Azure OpenAI GPT\-5\.4 was used under a structured reformatting prompt \(provided in the supplementary material\) to anonymize and rewrite the source material into 1,132 standalone requirement records while preserving the original typed\-edge structure \(derives\_from,satisfies,references,traces\_to,verifies\) and module taxonomy\. All resulting records were programmatically validated for format and uniqueness; no semantic validation or fact synthesis was performed by GenAI\.\(b\) Pipeline components\.GPT\-5\.4 is the generation backbone of all five RAG pipelines under study; it is theobjectof evaluation, not an authorial tool\.\(c\) LLM\-as\-judge\.GPT\-5\.4 and GPT\-4\.1 are used as the two faithfulness judges in the locked dual\-judge protocol; this is part of the paper’s experimental analysis and is fully documented in §[4](https://arxiv.org/html/2608.05153#S4)\.\(d\) Writing\.GenAI tools \(Claude Sonnet 4\.5, GPT\-5\.4\) were used for prose polishing, LaTeX\-table assembly, and code\-comment generation\. No section of the paper was authoredde novoby an LLM\. All claims, numbers, citations, and conclusions were verified by the author\(s\)\. The author\(s\) take full responsibility for the content\.

## References

- A\. Asai, Z\. Wu, Y\. Wang, A\. Sil, and H\. Hajishirzi \(2024\)Self\-RAG: learning to retrieve, generate, and critique through self\-reflection\.InProc\. International Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2608.05153#S2.p1.1)\.
- I\. Baek, H\. Chang, B\. Kim, J\. Lee, and H\. Lee \(2025\)Probing\-RAG: self\-probing to guide language models in selective document retrieval\.InFindings of NAACL,Note:arXiv:2410\.13339Cited by:[§2](https://arxiv.org/html/2608.05153#S2.p1.1)\.
- T\. Berg\-Kirkpatrick, D\. Burkett, and D\. Klein \(2012\)An empirical investigation of statistical significance in NLP\.InProc\. EMNLP\-CoNLL,Cited by:[§2](https://arxiv.org/html/2608.05153#S2.p5.1)\.
- G\. Dong, J\. Jin, X\. Li, Y\. Zhu, Z\. Dou, and J\. Wen \(2025\)RAG\-Critic: leveraging automated critic\-guided agentic workflow for retrieval augmented generation\.InProc\. 63rd Annual Meeting of the Association for Computational Linguistics \(ACL\): Long Papers,pp\. 3551–3578\.Note:Anthology ID: 2025\.acl\-long\.179Cited by:[§2](https://arxiv.org/html/2608.05153#S2.p1.1)\.
- W\. Du \(2025\)When\+1%\{\+\}1\\%is not enough: a paired bootstrap protocol for evaluating small improvements\.arXiv preprint arXiv:2511\.19794\.Cited by:[§4](https://arxiv.org/html/2608.05153#S4.p3.6)\.
- D\. Edge, H\. Trinh, N\. Cheng, J\. Bradley, A\. Chao, A\. Mody, S\. Truitt, and J\. Larson \(2024\)From local to global: a graph RAG approach to query\-focused summarization\.arXiv preprint arXiv:2404\.16130\.Cited by:[§1](https://arxiv.org/html/2608.05153#S1.p1.1),[§2](https://arxiv.org/html/2608.05153#S2.p2.1)\.
- European Union Aviation Safety Agency \(EASA\) \(2025\)Notice of proposed amendment NPA 2025\-07: artificial intelligence concept paper\.Note:EASACited by:[§1](https://arxiv.org/html/2608.05153#S1.p1.1)\.
- Federal Aviation Administration \(2024\)AI safety roadmap, version 1\.Note:FAACited by:[§1](https://arxiv.org/html/2608.05153#S1.p1.1)\.
- A\. R\. Feinstein and D\. V\. Cicchetti \(1990\)High agreement but low kappa: i\. the problems of two paradoxes\.Journal of Clinical Epidemiology43\(6\),pp\. 543–549\.Cited by:[§2](https://arxiv.org/html/2608.05153#S2.p4.4),[§3\.3](https://arxiv.org/html/2608.05153#S3.SS3.p1.5)\.
- D\. Fuchß, T\. Hey, J\. Keim, H\. Liu, N\. Ewald, T\. Thirolf, and A\. Koziolek \(2025\)LiSSA: toward generic traceability link recovery through retrieval\-augmented generation\.InProc\. IEEE/ACM 47th International Conference on Software Engineering \(ICSE\),Cited by:[§1](https://arxiv.org/html/2608.05153#S1.p1.1),[§2](https://arxiv.org/html/2608.05153#S2.p3.1)\.
- T\. Gao, H\. Yen, J\. Yu, and D\. Chen \(2023\)Enabling large language models to generate text with citations\.InProc\. EMNLP,Note:arXiv:2305\.14627Cited by:[§2](https://arxiv.org/html/2608.05153#S2.p4.4),[§3\.3](https://arxiv.org/html/2608.05153#S3.SS3.p1.5)\.
- Z\. Guo, L\. Xia, Y\. Yu, T\. Ao, and C\. Huang \(2024\)LightRAG: simple and fast retrieval\-augmented generation\.arXiv preprint arXiv:2410\.05779\.Cited by:[§2](https://arxiv.org/html/2608.05153#S2.p2.1)\.
- B\. J\. Gutiérrez, Y\. Shu, W\. Qi, S\. Zhou, and Y\. Su \(2025\)HippoRAG 2: from RAG to memory: non\-parametric continual learning for large language models\.arXiv preprint arXiv:2502\.14802\.Cited by:[§2](https://arxiv.org/html/2608.05153#S2.p2.1)\.
- K\. L\. Gwet \(2008\)Computing inter\-rater reliability and its variance in the presence of high agreement\.British Journal of Mathematical and Statistical Psychology61\(1\),pp\. 29–48\.Cited by:[§2](https://arxiv.org/html/2608.05153#S2.p4.4),[§3\.3](https://arxiv.org/html/2608.05153#S3.SS3.p1.5)\.
- H\. Han, L\. Ma, Y\. Wang, H\. Shomer, Y\. Lei, Z\. Qi, K\. Guo, Z\. Hua, B\. Long, H\. Liu, C\. C\. Aggarwal, and J\. Tang \(2025\)RAG vs\. GraphRAG: a systematic evaluation and key insights\.arXiv preprint arXiv:2502\.11371\.Cited by:[§1](https://arxiv.org/html/2608.05153#S1.p1.1),[§2](https://arxiv.org/html/2608.05153#S2.p2.1),[§5\.2](https://arxiv.org/html/2608.05153#S5.SS2.p1.1)\.
- S\. Jeong, J\. Baek, S\. Cho, S\. J\. Hwang, and J\. C\. Park \(2024\)Adaptive\-RAG: learning to adapt retrieval\-augmented large language models through question complexity\.InProc\. NAACL,Cited by:[§1](https://arxiv.org/html/2608.05153#S1.p1.1),[§2](https://arxiv.org/html/2608.05153#S2.p1.1)\.
- B\. Jin, H\. Zeng, Z\. Yue, J\. Yoon, S\. Arik, D\. Wang, H\. Zamani, and J\. Han \(2025\)Search\-R1: training LLMs to reason and leverage search engines with reinforcement learning\.arXiv preprint arXiv:2503\.09516\.Cited by:[§2](https://arxiv.org/html/2608.05153#S2.p1.1)\.
- P\. Koehn \(2004\)Statistical significance tests for machine translation evaluation\.InProc\. EMNLP,Cited by:[§2](https://arxiv.org/html/2608.05153#S2.p5.1)\.
- H\. Luo, H\. E, G\. Chen, Q\. Lin, Y\. Guo, F\. Xu, Z\. Kuang, M\. Song, X\. Wu, Y\. Zhu, and L\. A\. Tuan \(2025\)Graph\-R1: towards agentic GraphRAG framework via end\-to\-end reinforcement learning\.arXiv preprint arXiv:2507\.21892\.Cited by:[§2](https://arxiv.org/html/2608.05153#S2.p2.1)\.
- A\. Masoudifard, M\. Mowlavi Sorond, M\. Madadi, M\. Sabokrou, and E\. Habibi \(2024\)Leveraging graph\-RAG and prompt engineering to enhance LLM\-based automated requirement traceability and compliance checks\.arXiv preprint arXiv:2412\.08593\.Cited by:[§2](https://arxiv.org/html/2608.05153#S2.p3.1)\.
- F\. Niu, R\. Pan, L\. C\. Briand, and H\. Hu \(2025\)TVR: automotive system requirement traceability validation and recovery through retrieval\-augmented generation\.arXiv preprint arXiv:2504\.15427\.Cited by:[§1](https://arxiv.org/html/2608.05153#S1.p1.1),[§2](https://arxiv.org/html/2608.05153#S2.p3.1)\.
- RTCA, Inc\. \(2011\)DO\-178C: software considerations in airborne systems and equipment certification\.Technical reportRTCA\.Cited by:[§1](https://arxiv.org/html/2608.05153#S1.p1.1)\.
- D\. Ru, L\. Qiu, X\. Hu, T\. Zhang, P\. Shi,et al\.\(2024\)RAGChecker: a fine\-grained framework for diagnosing retrieval\-augmented generation\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track,Note:arXiv:2408\.08067Cited by:[§2](https://arxiv.org/html/2608.05153#S2.p4.4)\.
- SAE International \(2010\)ARP4754A: guidelines for development of civil aircraft and systems\.Technical reportSAE International\.Cited by:[§1](https://arxiv.org/html/2608.05153#S1.p1.1)\.
- Y\. Tang and Y\. Yang \(2024\)MultiHop\-RAG: benchmarking retrieval\-augmented generation for multi\-hop queries\.arXiv preprint arXiv:2401\.15391\.Cited by:[§2](https://arxiv.org/html/2608.05153#S2.p2.1)\.
- H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal \(2022\)MuSiQue: multihop questions via single\-hop question composition\.Transactions of the Association for Computational Linguistics \(TACL\)10,pp\. 539–554\.Cited by:[§2](https://arxiv.org/html/2608.05153#S2.p2.1),[§3\.2](https://arxiv.org/html/2608.05153#S3.SS2.p1.1)\.
- J\. Wallat, M\. Heuss, M\. de Rijke, and A\. Anand \(2024\)Correctness is not faithfulness in RAG attributions\.arXiv preprint arXiv:2412\.18004\.Cited by:[§2](https://arxiv.org/html/2608.05153#S2.p4.4),[§5\.2](https://arxiv.org/html/2608.05153#S5.SS2.p1.1),[§6](https://arxiv.org/html/2608.05153#S6.SS0.SSS0.Px3.p1.4)\.
- K\. Wataoka, T\. Takahashi, and R\. Ri \(2024\)Self\-preference bias in LLM\-as\-a\-judge\.InNeurIPS Safe Generative AI Workshop,Note:arXiv:2410\.21819Cited by:[§2](https://arxiv.org/html/2608.05153#S2.p4.4),[§6](https://arxiv.org/html/2608.05153#S6.SS0.SSS0.Px3.p1.4)\.
- Z\. Xiang, C\. Wu, Q\. Zhang, S\. Chen, Z\. Hong, X\. Huang, and J\. Su \(2026\)When to use graphs in RAG: a comprehensive analysis for graph retrieval\-augmented generation\.InProc\. International Conference on Learning Representations \(ICLR\),Note:arXiv:2506\.05690Cited by:[§1](https://arxiv.org/html/2608.05153#S1.p1.1),[§2](https://arxiv.org/html/2608.05153#S2.p2.1),[§5\.2](https://arxiv.org/html/2608.05153#S5.SS2.p1.1)\.
- S\. Yan, J\. Gu, Y\. Zhu, and Z\. Ling \(2024\)Corrective retrieval augmented generation\.arXiv preprint arXiv:2401\.15884\.Cited by:[§2](https://arxiv.org/html/2608.05153#S2.p1.1)\.