MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph-Augmented Generation

# MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph–Augmented Generation
Source: [https://arxiv.org/html/2606.26458](https://arxiv.org/html/2606.26458)
,Bao HoangMichigan State UniversityEast LansingMichiganUSA[hoangbao@msu\.edu](https://arxiv.org/html/2606.26458v1/mailto:[email protected]),Han LiuDalian University of TechnologyDalianLiaoningChina[liu\.han\.dut@gmail\.com](https://arxiv.org/html/2606.26458v1/mailto:[email protected]),Ting WangStony Brook UniversityStony BrookNew YorkUSA[wang@cs\.stonybrook\.edu](https://arxiv.org/html/2606.26458v1/mailto:[email protected])andFenglong MaThe Pennsylvania State UniversityUniversity ParkPennsylvaniaUSA[fenglong@psu\.edu](https://arxiv.org/html/2606.26458v1/mailto:[email protected])

\(2026\)

###### Abstract\.

Retrieval\-augmented generation \(RAG\) over knowledge graphs has emerged as a promising approach for grounding large language models, yet existing benchmarks largely overlook the challenges of retrieval in multimodal knowledge graph RAG \(MKG\-RAG\)\. In practice, retrieval is a critical bottleneck: multimodal knowledge is heterogeneous, difficult to align across modalities, and often poorly served by retrievers designed for unstructured corpora\. To address this gap, we introduceMKG\-RAG\-Bench, a cross\-domain benchmark explicitly designed to evaluate retrieval in MKG\-RAG\.MKG\-RAG\-Benchis constructed from two multimodal knowledge graphs spanning general and medical domains, and includes carefully aligned question\-answering datasets that support controlled evaluation of both retrieval and downstream generation\. The benchmark is built using an LLM\-based curation pipeline that filters low\-utility knowledge, generates structurally grounded queries with exact supervision, and systematically covers diverse modality configurations\. Through extensive experiments across representative retriever families and modality settings, we show that effective multimodal retrieval remains challenging yet crucial for end\-to\-end MKG\-RAG performance, and that retrieval quality strongly determines generation outcomes\. By isolating retrieval as a first\-class evaluation target,MKG\-RAG\-Benchprovides a principled foundation for diagnosing current limitations and advancing multimodal knowledge graph RAG systems\.111The benchmark can be accessed via:[https://github\.com/XiaochenWang\-PSU/MKG\-RAG\-Bench](https://github.com/XiaochenWang-PSU/MKG-RAG-Bench)\.

Multimodal retrieval\-augmented generation, multimodal knowledge graphs

††journalyear:2026††copyright:cc††conference:Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\.2; August 09–13, 2026; Jeju Island, Republic of Korea††booktitle:Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\.2 \(KDD ’26\), August 09–13, 2026, Jeju Island, Republic of Korea††doi:10\.1145/3770855\.3817570††isbn:979\-8\-4007\-2259\-2/2026/08††ccs:Computing methodologies Continuous space search††ccs:Computing methodologies Natural language generation## 1\.Introduction

Retrieval\-augmented generation \(RAG\) enhances large language models \(LLMs\) by retrieving information relevant to a given query from external sources and conditioning generation on this retrieved evidence\(Lewiset al\.,[2020](https://arxiv.org/html/2606.26458#bib.bib48)\)\. While RAG has proven effective in improving factual grounding, conventional approaches predominantly retrieve from unstructured textual corpora that are noisy, fragmented, and weakly connected\. These limitations often hinder reliable evidence selection and multi\-step reasoning, especially for complex queries\. To address these issues,knowledge graph–based RAG\(KG\-RAG\) has been proposed\(Edgeet al\.,[2024](https://arxiv.org/html/2606.26458#bib.bib49)\)\. By retrieving information from structured knowledge graphs, KG\-RAG enables more coherent, contextualized, and comprehensive knowledge access, leading to improved answer quality and faithfulness\(Wanget al\.,[2026](https://arxiv.org/html/2606.26458#bib.bib15)\)\. However, existing KG\-RAG methods are largely confined totextual knowledge graphs, which substantially limits their applicability in real\-world scenarios where critical information is conveyed through images, charts, tables, and other non\-textual modalities\.

![Refer to caption](https://arxiv.org/html/2606.26458v1/x1.png)Figure 1\.The proposed pipeline for benchmark construction using multimodal knowledge graphs and LLMs\.Multimodal KG\-RAG\.To overcome this limitation, we explore the task, multimodal KG\-RAG \(MKG\-RAG\), which integrates structured knowledge across multiple modalities to support more expressive retrieval and reasoning\. Formally, given a queryqq, a multimodal knowledge graph𝒢\\mathcal\{G\}, a retrieverRetriever, and a multimodal large language modelMLLM, the goal ofMKG\-RAGis to generate a responseℛ\\mathcal\{R\}, i\.e\.,

ℛ=MLLM​\(q,Retriever​\(q,𝒢\)\)\.\\mathcal\{R\}=\\textsf\{MLLM\}\(q,\\textsf\{Retriever\}\(q,\\mathcal\{G\}\)\)\.The most closely related setting is multimodal RAG\(Chenet al\.,[2022b](https://arxiv.org/html/2606.26458#bib.bib47)\), which follows the traditional RAG pipeline by retrieving multimodal content from unstructured external corpora \(e\.g\., Wikipedia\) rather than from a structured multimodal knowledge graph\. To the best of our knowledge,no existing benchmark or prior worksystematically studies retrieval and generation in theMKG\-RAGsetting\.

Motivation and Challenges\.Multimodal KG\-RAG is benchmark\-worthy because it evaluates a model’s ability to retrieve, align, and reason over*structured multimodal knowledge*, a critical capability for real\-world systems that is not captured by existing RAG, KG\-RAG, or multimodal benchmarks\. This work aims to introduce thefirstbenchmark for theMKG\-RAGtask, consisting of curated datasets and evaluation protocols that explicitly target retrieval quality and multimodal grounding\. Designing such a benchmark, however, poses several non\-trivial challenges:

- •Heterogeneous Retrieval\.Unlike conventional knowledge graphs, where both queries and target triplets are unimodal, multimodal knowledge graphs support heterogeneous modality combinations across queries and retrieval targets\. InMKG\-RAG, queries may be text\-only, image\-only, or multimodal, while relevant graph components may combine textual, visual, and numerical information in different ways\. These combinations induce distinct retrieval behaviors and demand different modeling strategies\. A meaningfulMKG\-RAGbenchmark must therefore systematically cover diverse multimodal retrieval scenarios to reflect realistic usage and to enable fair comparison among retrievers\.
- •Multimodal Knowledge Graph–Query Alignment\.InMKG\-RAG, relevant knowledge may not be explicitly aligned with the query at the surface level, particularly when the query and target nodes differ in modality or abstraction\. For example, a textual query may require grounding in visual evidence, while an image\-based query may necessitate retrieving semantically related textual entities and relations\. Addressing such cross\-modal and cross\-structural alignment requires joint reasoning over graph topology, semantic relationships, and modality\-specific representations\. Accordingly, aMKG\-RAGbenchmark must include queries that demand non\-trivial alignment beyond shallow modality matching\.
- •Benchmark Utility\.The primary goal of the benchmark is to evaluate both the effectiveness of multimodal retrieval and the grounding quality of multimodal LLMs\. This requires careful construction to ensure that the retrieved knowledge is informative and well\-aligned with downstream queries\. Multimodal knowledge graphs often contain generic or weakly informative triplets, which can lead to ambiguous or poorly grounded questions and obscure the contribution of retrieval\. A usefulMKG\-RAGbenchmark must prioritize high\-utility triplets and disentangle improvements due to effective retrieval from those arising from generative priors\.

Our Solution\.To address these challenges, we propose a cross\-domain, retrieval\-orientedMKG\-RAGbenchmark, namedMKG\-RAG\-Bench, comprising two multimodal knowledge graphs and corresponding multimodal question\-answering datasets\. The benchmark is designed to enable comprehensive evaluation across both the retrieval and generation stages of multimodal KG\-RAG\. The benchmark construction pipeline is shown in Figure[1](https://arxiv.org/html/2606.26458#S1.F1)\. Specifically, we use an LLM to select high\-utility triplets and employ heuristic strategies to ensure strong alignment between the selected triplets and the constructed queries\. The same LLM, with different prompts, is further leveraged to generate multimodal, knowledge\-grounded questions across diverse retrieval scenarios, thereby systematically covering heterogeneous modality combinations\. We conduct extensive experiments with representative retriever architectures across different multimodal settings, evaluating both effectiveness and efficiency throughout the retrieval and generation pipeline\.While this work primarily focuses on evaluatingMKG\-RAGusingtraining\-freeapproaches, the benchmark is also applicable to training\-based methods, including retriever training and MLLM fine\-tuning\.

Table 1\.Comparison of representative multimodal RAG and KG\-RAG benchmarks\.Contributions\.In summary, our main contributions are as follows:

- •We identify a critical research gap inMKG\-RAG, highlighting that retrieval, despite its central role, is underexplored and insufficiently evaluated in existing multimodal RAG research\.
- •We introduce the first, cross\-domain, retrieval\-orientedMKG\-RAGbenchmark comprising two multimodal knowledge graphs and corresponding question\-answering datasets, enabling systematic evaluation of retrieval and downstream generation\.
- •We propose a benchmark curation pipeline based on LLMs that directly addresses key challenges in heterogeneous retrieval, multimodal alignment, and benchmark utility\.
- •We develop a unified evaluation framework and conduct comprehensive experiments across diverse multimodal settings, demonstrating the importance of effective multimodal retrieval and enabling future research on more powerful retrievers forMKG\-RAG\.

## 2\.Preliminaries

### 2\.1\.Benchmark Comparisons

The most closely related line of work is multimodal RAG \(M\-RAG\), which focuses on retrieving multimodal content, such as textual passages and images, from large unstructured corpora\. Retrieved multimodal evidence can be directly incorporated into the input of multimodal large language models\(Maet al\.,[2024b](https://arxiv.org/html/2606.26458#bib.bib55);[Liet al\.,](https://arxiv.org/html/2606.26458#bib.bib57)\), or reorganized into ad\-hoc multimodal structures to further enhance generation\(Wanget al\.,[2025a](https://arxiv.org/html/2606.26458#bib.bib56)\)\. However, M\-RAG follows the conventional RAG paradigm and does not assume access to a structured multimodal knowledge graph\. As a result, retrieval is performed overunstructured corpora, and any induced structure is transient and query\-specific rather than explicitly represented in the knowledge source\.

Another related line of work is knowledge graph–based RAG \(KG\-RAG\), which aims to retrieve relevant triplets or subgraphs from a given knowledge graph to support grounded generation\. Existing KG\-RAG approaches\(Luoet al\.,[2023](https://arxiv.org/html/2606.26458#bib.bib46); Liet al\.,[2024](https://arxiv.org/html/2606.26458#bib.bib45)\), however, primarily focus ontextual knowledge graphsand do not explicitly model or retrieve multimodal information\. Consequently, they are not designed to evaluate or support retrieval and reasoning over structured multimodal knowledge\.

In contrast,MKG\-RAGoperates overpre\-defined multimodal knowledge graphs, where entities, relations, and cross\-modal connections are explicitly modeled\. This fundamentally changes the retrieval problem: instead of selecting isolated multimodal documents,MKG\-RAGrequires identifying relevant subgraphs that integrate heterogeneous modalities through structured relations\. Consequently, existing M\-RAG and KG\-RAG benchmarks are insufficient for evaluatingMKG\-RAG, as they do not capture multimodal graph\-based retrieval or disentangle retrieval quality from downstream generation\. Table[1](https://arxiv.org/html/2606.26458#S1.T1)compares representative benchmarks across multiple perspectives, highlighting the distinctive characteristics ofMKG\-RAG\.

### 2\.2\.Retrievers

Although theMKG\-RAGtask differs from conventional RAG, multimodal RAG, and KG\-RAG, many existing retrieval techniques can be directly adapted to this setting\. In this work, we categorize commonly used retrievers into four groups based on their underlying retrieval strategies:

- •Text\-only retrievers\(Robertson and Zaragoza,[2009](https://arxiv.org/html/2606.26458#bib.bib29); Karpukhinet al\.,[2020](https://arxiv.org/html/2606.26458#bib.bib30); Reimers and Gurevych,[2019](https://arxiv.org/html/2606.26458#bib.bib31)\)\. Text\-only retrievers treat both queries and candidate triplets as plain text and rank candidates using textual similarity, such as sparse lexical matching or dense semantic embeddings, without incorporating visual information\.
- •Fusion\-based multimodal retrievers\(Chenet al\.,[2022b](https://arxiv.org/html/2606.26458#bib.bib47); Meiet al\.,[2025](https://arxiv.org/html/2606.26458#bib.bib32)\)\. Fusion\-based retrievers encode textual and visual information into a shared representation space and perform retrieval by measuring cross\-modal similarity between multimodal queries and multimodal targets\.
- •Captioning\-based retrievers\(Gaoet al\.,[2022](https://arxiv.org/html/2606.26458#bib.bib27); Zhai,[2024](https://arxiv.org/html/2606.26458#bib.bib28)\)\. Captioning\-based retrievers first convert visual content into textual descriptions using an external image captioning model, and then reduce multimodal retrieval to text\-only retrieval over the generated captions\.
- •Reranking\-based retrievers\(Yan and Xie,[2024](https://arxiv.org/html/2606.26458#bib.bib25); Chenet al\.,[2025](https://arxiv.org/html/2606.26458#bib.bib26)\)\. Reranking approaches typically adopt a multi\-stage pipeline in which a lightweight first\-stage retriever filters a candidate set, followed by a more powerful reranker that performs fine\-grained relevance estimation and final selection\.

To ensure fair and controlled comparison, we evaluate all retrievers in atraining\-freesetting by directly using their pretrained weights, without fine\-tuning on the proposed benchmark\.

### 2\.3\.Preliminary Analysis

A seemingly straightforward approach to constructing a multimodal knowledge graph RAG benchmark is to directly combine existing multimodal knowledge graphs, such as MMKG\(Liuet al\.,[2019](https://arxiv.org/html/2606.26458#bib.bib44)\)and TIVA\-KG\(Wanget al\.,[2023](https://arxiv.org/html/2606.26458#bib.bib43)\), with modality\-relevant downstream tasks, e\.g\., A\-OKVQA\(Schwenket al\.,[2022](https://arxiv.org/html/2606.26458#bib.bib58)\)and K\-VQA\(Sanket Shah and Talukdar,[2019](https://arxiv.org/html/2606.26458#bib.bib42)\)\. However, as discussed in Section[1](https://arxiv.org/html/2606.26458#S1), such a naive combination fails to adequately capture the requirements of theMKG\-RAGtask and often leads to severe misalignment between the knowledge source, retrieval objectives, and downstream evaluation\. In particular, the knowledge required to answer a given query may not be present in the knowledge graph, while irrelevant retrieved content may dominate the model’s input, introducing noise and ultimately degrading performance\.

To empirically illustrate this issue, we conduct a preliminary study using MedMKG\(Wanget al\.,[2025b](https://arxiv.org/html/2606.26458#bib.bib20)\), a multimodal medical knowledge graph constructed from MIMIC clinical data, to support medical visual question answering on two widely used benchmarks, VQA\-RAD\(Lauet al\.,[2018](https://arxiv.org/html/2606.26458#bib.bib18)\)and SLAKE\(Liuet al\.,[2021](https://arxiv.org/html/2606.26458#bib.bib17)\)\. Specifically, we augment multimodal large language models with retrieval from MedMKG and evaluate their performance under different retrieval strategies\. The results are summarized in Figure[2](https://arxiv.org/html/2606.26458#S2.F2)\.

Across all experimental settings, models augmented with multimodal retrieval consistently underperform their RAG\-free counterparts\. Although multimodal retrievers outperform text\-only retrievers, they still fail to surpass generation without retrieval augmentation\. These results indicate that retrieval from MedMKG does not effectively support the downstream visual question answering tasks\. We attribute this behavior to two primary factors: \(1\) task\-critical knowledge required by the MLLM may be absent from the multimodal knowledge graph, and \(2\) task\-irrelevant knowledge present in the graph may be retrieved and injected into the model input, introducing substantial noise that interferes with generation\.

Taken together, these observations reveal a fundamental limitation of directly combining existing multimodal knowledge graphs with downstream multimodal tasks for MKG\-RAG\. The resulting misalignment between knowledge sources, retrieval targets, and evaluation objectives undermines the effectiveness of retrieval augmentation and obscures the true capabilities of multimodal KG\-RAG systems\. This motivates the need for a dedicatedMKG\-RAGbenchmark that explicitly aligns multimodal knowledge graphs, retrieval objectives, and downstream evaluation—precisely the goal of the benchmark proposed in this work\.

![Refer to caption](https://arxiv.org/html/2606.26458v1/x2.png)Figure 2\.BLEU \(%\) comparison on \(a\) VQA\-RAD and \(b\) SLAKE datasets\. RAG\-free consistently achieves the best performance across both benchmarks\.

## 3\.Benchmark Construction

Table 2\.Statistics ofMKG\-RAG\-Bench, where %MM refers to the percentage of multimodal data\.DatasetSplitRetrievalGenerationQueryTripletQuestionAnswer\# of Queries%MMAvgLen\# of Triplets%MMAvgLen\# of Questions%MMAvgLen\# of AnswersAvgLenMKG\-RAG\-Bench\-GTrain48,90830\.112\.225,51728\.86\.848,90830\.112\.261,0012\.5Val6,11331\.012\.225,51728\.86\.86,11331\.012\.27,5862\.5Test6,11530\.312\.225,51728\.86\.86,11530\.312\.27,7952\.5MKG\-RAG\-Bench\-MTrain4,78140\.616\.218,46849\.08\.74,78140\.616\.216,5643\.7Val59742\.216\.118,46849\.08\.759742\.216\.12,0303\.5Test59943\.116\.218,46849\.08\.759943\.116\.22,0743\.6

An effective benchmark should not only cover diverse data sources but also support rigorous evaluation across multiple stages of a task\. In the proposedMKG\-RAGbenchmark, we construct datasets from two multimodal knowledge graphs spanning distinct domains: MarKG\(Zhanget al\.,[2023](https://arxiv.org/html/2606.26458#bib.bib10)\)for the general domain and MedMKG\(Wanget al\.,[2025b](https://arxiv.org/html/2606.26458#bib.bib20)\)for the medical domain222The details of these two multimodal knowledge graphs are provided in Appendix[A](https://arxiv.org/html/2606.26458#A1)\.\. This cross\-domain design allows us to evaluate the generality of multimodal KG\-RAG methods under varying knowledge structures and modality distributions\. To enable faithful evaluation of both retrieval and generation in multimodal KG\-RAG, we design a principled data construction pipeline that filters out low\-utility knowledge, generates structurally aligned and knowledge\-grounded queries with exact supervision, and supports explicit evaluation of both retrieval and generation stages\.

### 3\.1\.Dataset Curation

As shown in Figure[1](https://arxiv.org/html/2606.26458#S1.F1), our construction pipeline consists of three stages: \(1\) LLM\-based utility filtering, \(2\) controlled completeness simulation via triplet masking, and \(3\) hybrid question synthesis\. Each stage is designed to address a specific challenge in benchmarking multimodal KG\-RAG\.

Step 1: LLM\-based Utility Filtering\.A central challenge in benchmarking multimodal KG\-RAG is ensuring that retrieval contributes meaningful and well\-grounded evidence to downstream tasks\. Multimodal knowledge graphs often contain triplets that are overly generic, definitional, or weakly informative, from which derived queries may be ambiguous or insufficiently grounded, limiting their usefulness for evaluating retrieval\.

Given a triplet, we prompt an LLM to assess its*retrieval utility*, i\.e\., whether it can support a clear and informative information\-seeking query\. Triplets deemed low\-utility are those that tend to yield questions answerable without external evidence or that fail to convey a specific retrieval intent\(Yihet al\.,[2016](https://arxiv.org/html/2606.26458#bib.bib35)\)\. For example, the triplet \(Sun, is\_a, star\) may lead to the question “What is the Sun?”, which neither requires retrieval nor benefits from structured knowledge\. We remove such triplets from the benchmark to ensure that generated queries are well specified and grounded in retrievable knowledge\. This filtering step produces a refined triplet set that better reflects realistic retrieval scenarios in which external knowledge access is necessary for accurate reasoning\. The prompt templates and decision criteria are detailed in Appendix[B](https://arxiv.org/html/2606.26458#A2)\.

Step 2: Controlled Completeness Simulation\.To construct queries that require knowledge retrieval while preserving unambiguous supervision, we simulate controlled incompleteness over the retained triplets\. Rather than presenting complete knowledge graph facts, we mask one component of each triplet and treat the remaining components as observed context, thereby mimicking information\-seeking queries\.

For each triplet\(h,r,t\)\(h,r,t\), we construct two masked variants: \(i\)*relation masking*, where the relationrris masked and must be inferred from the head and tail, and \(ii\)*tail masking*, where the tailttis masked and must be inferred from the head and relation\. We keep the head entity unchanged as a semantic anchor, which facilitates multimodal grounding when visual information is available and ensures that each query is uniquely supported by its originating knowledge\.

We avoid masking the head entity, as doing so would require entity disambiguation beyond the scope of retrieval evaluation and could introduce additional ambiguity\. This design allows us to focus on assessing retrievers’ ability to identify relevant relations and entities under realistic multimodal conditions\. The resulting masked triplets serve as structurally grounded templates for query generation\.

Step 3: Hybrid Question Synthesis\.Masked triplets provide structured but non\-linguistic representations and do not directly resemble natural user queries\. To generate realistic questions while preserving exact alignment with the underlying knowledge, we again employ GPT\-5\(Singhet al\.,[2025](https://arxiv.org/html/2606.26458#bib.bib50)\)to synthesize natural\-language questions from masked triplets under two complementary modes\.

Text\-only question synthesis\.For masked triplets derived from text\-only knowledge, GPT\-5 generates textual questions that explicitly correspond to the missing component\. For example, from\(penicillin,\[Mask\],bacterial infection\)\(\\textit\{penicillin\},\[\\textsc\{Mask\}\],\\textit\{bacterial infection\}\), the LLM generates: “What is the relationship between penicillin and bacterial infection?” This produces text\-only queries whose answers are uniquely grounded in the masked triplet\.

Image\-grounded question synthesis\.For masked triplets whose head entity is associated with an image, we provide the image to GPT\-5 and require the head entity to be referenced as\[Image\]\. This enforces explicit visual grounding and prevents trivial lexical matching on entity names\. For example,\(Eiffel Tower,located\_in,\[Mask\]\)\(\\textit\{Eiffel Tower\},\\textit\{located\\\_in\},\[\\textsc\{Mask\}\]\)yields the question: “![[Uncaptioned image]](https://arxiv.org/html/2606.26458v1/fig/Eiffel-Tower-Paris-France-800x1200.jpg)Which city is the landmark in the image located in?” To avoid shortcut retrieval caused by identical images appearing in both queries and knowledge graph nodes, we apply image augmentations to the question images in Appendix[C](https://arxiv.org/html/2606.26458#A3)\.

Across both modes, question synthesis preserves the semantics of the masked component while producing natural, modality\-appropriate queries, enabling retrieval evaluation under both text\-only and multimodal settings\.

### 3\.2\.Downstream Task Data

To enable explicit and fine\-grained evaluation of retrieval performance, we constructretrieval supervisiondirectly from the aligned query–triplet pairs\. For each synthesized question, the originating triplet is treated as the primary positive target\. Building on the retrieval dataset, we constructdownstream generation datain a supervision\-preserving manner\. Because each question is derived from a masked triplet, the masked component corresponds exactly to the missing information requested by the query\. We therefore treat the masked entity or relation as the ground\-truth answer\. This process yields question–answer \(QA\) pairs that are guaranteed to be supported by the aligned multimodal knowledge graph\. The resulting QA data includes both text\-only and visually grounded instances, reflecting the mixed\-modality nature ofMKG\-RAG\. By construction, successful generation requires both correct retrieval and faithful grounding, enabling meaningful end\-to\-end evaluation\. An example of our benchmark construction is provided in Appendix[D](https://arxiv.org/html/2606.26458#A4)\.

### 3\.3\.Dataset Statistics

After curating both retrieval and generation datasets from two domain\-specific multimodal knowledge graphs, we split the datasets for each domain and each stage into train/validation/test sets with an8:1:18\{:\}1\{:\}1ratio\. The split is performed by questions, so the same question is never shared across train/validation/test sets, while keeping the corpus fixed, following standard IR evaluation\. This split supports not only comprehensive evaluation but also retriever training when needed\. To reflect realistic MKG\-RAG usage, each split contains a mixture of text\-only and multimodal query–triplet pairs, rather than separating modalities into disjoint partitions\. As a result, we obtainMKG\-RAG\-Bench, a cross\-domain multimodal benchmark that supports training and evaluation under diverse settings in practical MKG\-RAG scenarios\. Table[2](https://arxiv.org/html/2606.26458#S3.T2)reports the key statistics ofMKG\-RAG\-Bench, whereMKG\-RAG\-Bench\-G represents the data obtained from the general multimodal knowledge graph MarKG, andMKG\-RAG\-Bench\-M is the dataset constructed based on the medical multimodal knowledge graph MedMKG\.

A notable difference between the two subsets lies in the query–triplet cardinality\. ForMKG\-RAG\-Bench\-G, the number of queries notably exceeds the number of triplets\. In contrast,MKG\-RAG\-Bench\-M contains substantially fewer queries than triplets\. This gap is primarily driven by the structural properties of the underlying medical KG, which exhibits more frequent1:n1\{:\}npatterns\. Popular clinical entities \(e\.g\.,lungorpneumonia\) often connect to many other concepts through the same relation, so a fixed incomplete form such as\(h,r,\[M​a​s​k\]\)\(h,r,\[Mask\]\)may correspond to multiple valid tails\. Consequently, many triplets share the same masked form and yield identical queries, resulting in fewer unique queries despite a larger triplet set\.

By contrast, the general\-domain KG is relatively sparse: relations between everyday entities are less densely connected and tend to produce more unique incomplete forms, leading to a higher diversity of queries per triplet\.

Table 3\.Retrieval performance \(%\) onMKG\-RAG\-Bench\-G dataset under different retrieval settings\.Table 4\.Retrieval performance \(%\) onMKG\-RAG\-Bench\-M dataset under different retrieval settings\.

## 4\.Benchmark Evaluation

### 4\.1\.Evaluation Settings

We evaluate the proposedMKG\-RAG\-Benchon both retrieval and generation tasks\. As our primary focus is on thetraining\-freesetting, all evaluations are conducted exclusively on the test splits reported in Table[2](https://arxiv.org/html/2606.26458#S3.T2), without using any training or validation data\. As described in Section[3\.1](https://arxiv.org/html/2606.26458#S3.SS1)\(Step 3\), the benchmark includes two types of queries—text\-only and multimodal—and the underlying multimodal knowledge graphs contain both text\-only and multimodal triplets333Note that MarKG and MedMKG do not include triplets of the form \(image, relation, image\)\.\. To comprehensively assess retrieval and generation performance under different modality combinations, we evaluate the following five settings, defined by the query modality and the scope of candidate triplets: \(S1\)all queries with all triplets, \(S2\)text\-only queries with text\-only triplets, \(S3\)text\-only queries with all triplets, \(S4\)multimodal queries with multimodal triplets, and \(S5\)multimodal queries with all triplets\.

Setting S1 provides an overall evaluation of different retrieval techniques under the most general and realistic scenario\. Comparing S2 and S3 allows us to assess whether incorporating multimodal triplets benefits retrieval and generation for text\-only queries\. Similarly, comparing S4 and S5 evaluates the contribution of text\-only triplets when handling multimodal queries\. Together, these settings enable systematic analysis of how multimodal information influences retrieval and generation performance across different query types and knowledge graph configurations\.

### 4\.2\.Retrieval Evaluation

Evaluation Configuration\.The objective of retrieval evaluation is to assess the effectiveness of different retrieval techniques introduced in Section[2\.2](https://arxiv.org/html/2606.26458#S2.SS2), includingtext\-only retrievers\(Huet al\.,[2025](https://arxiv.org/html/2606.26458#bib.bib51); Liet al\.,[2023](https://arxiv.org/html/2606.26458#bib.bib52); Heet al\.,[2024](https://arxiv.org/html/2606.26458#bib.bib38)\),fusion\-based multimodal retrievers\(Chenet al\.,[2022b](https://arxiv.org/html/2606.26458#bib.bib47); Meiet al\.,[2025](https://arxiv.org/html/2606.26458#bib.bib32)\),captioning\-based retrievers\(Gaoet al\.,[2022](https://arxiv.org/html/2606.26458#bib.bib27); Zhai,[2024](https://arxiv.org/html/2606.26458#bib.bib28)\), andreranking\-based retrievers\(Yan and Xie,[2024](https://arxiv.org/html/2606.26458#bib.bib25); Chenet al\.,[2025](https://arxiv.org/html/2606.26458#bib.bib26)\)\. In addition, we include a basicrandom retrieveras a simple lower\-bound baseline for the retrieval task\. For fair comparison, all retrievers are implemented using a shared CLIP encoder\(Radfordet al\.,[2021](https://arxiv.org/html/2606.26458#bib.bib53)\)444[https://huggingface\.co/sentence\-transformers/clip\-ViT\-B\-32](https://huggingface.co/sentence-transformers/clip-ViT-B-32)to obtain unified representations\. Additionally, thecaptioning\-based retrieversare implemented with a BLIP model\(Liet al\.,[2022](https://arxiv.org/html/2606.26458#bib.bib41)\)555[https://huggingface\.co/Salesforce/blip\-image\-captioning\-base](https://huggingface.co/Salesforce/blip-image-captioning-base)\. Both queries and candidate triplets are embedded into the same representation space, and candidates are ranked based oncosine similarity\. We report standard retrieval metrics, including NDCG@​K@K, Precision@​K@K, and Recall@​K@Kin the main experiments\. Implementation details are provided in Appendix[E](https://arxiv.org/html/2606.26458#A5)\.

Retrieval Results and Key Findings\.We evaluate retrieval performance on bothMKG\-RAG\-Bench\-G andMKG\-RAG\-Bench\-M under different query and triplet compositions\. Detailed results are reported in Table[3](https://arxiv.org/html/2606.26458#S3.T3)and Table[4](https://arxiv.org/html/2606.26458#S3.T4), with efficiency analysis provided in Appendix[F](https://arxiv.org/html/2606.26458#A6)\. Note thatOthers\*means the baselines mentioned in the evaluation configuration\. Rather than enumerating numerical comparisons, we summarize the results through a set of key findings that highlight the challenges and opportunities of multimodal KG\-RAG retrieval\.

Finding 1:A modality gap persists in retrieval\.For text\-only queries retrieving from text\-only triplets, all retrievers exhibit nearly identical performance, as they effectively reduce to the same text\-based retrieval pipeline\. When the candidate space is expanded to include both text\-only and multimodal triplets, performance changes only marginally, and top\-ranked results remain dominated by text\-only triplets\. This observation suggests that multimodal triplets rarely serve as effective matches for purely textual queries and primarily increase the size of the candidate pool without contributing useful evidence\. More broadly, this behavior reveals a persistent*modality gap*in the retrieval space: cross\-modal matching between textual queries and multimodal knowledge does not emerge naturally from hybrid indexing alone\. This finding indicates that simply incorporating multimodal knowledge into a KG is insufficient for enabling cross\-modal retrieval, and that explicit mechanisms are required to bridge modalities when such interactions are desired\.

Finding 2:Multimodal embedding is critical for visually grounded retrieval\.Once queries require visual grounding, unimodal retrieval strategies become unreliable\. Text\-only retrievers degrade sharply because they lack the ability to incorporate image evidence\. Captioning\-based retrievers also underperform, as the captioning process introduces an information bottleneck: omissions or inaccuracies in generated captions directly propagate into retrieval errors\. In contrast, embedding\-based multimodal retrievers remain effective in this regime\. Fusion\-based methods benefit from jointly encoding visual and textual signals into a shared representation space, enabling more robust cross\-modal alignment\. Reranking\-based approaches further improve performance by refining candidate sets using stronger multimodal matching\. These results indicate that the central capability for effective MKG\-RAG retrieval is not improved text ranking, but reliable multimodal representation learning that preserves and aligns visual evidence with KG semantics\.

Finding 3:Retrieval difficulty and method preference are domain\-dependent\.Visually grounded retrieval is consistently more challenging onMKG\-RAG\-Bench\-M than onMKG\-RAG\-Bench\-G\. A plausible explanation is that medical images tend to be visually homogeneous and require sensitivity to subtle patterns, while the associated terminology is dense and fine\-grained\. As a result, generic similarity signals are weaker and more easily confounded\. This domain discrepancy also explains differences in method preference\. OnMKG\-RAG\-Bench\-G, fusion\-based retrieval is often sufficient, as coarse semantic alignment yields strong performance\. OnMKG\-RAG\-Bench\-M, reranking\-based approaches provide greater benefits, as refinement helps disambiguate subtle visual cues and specialized language\. These observations highlight that retrieval strategies effective in general domains may not transfer directly to specialized domains without adaptation\.

Implications for MKG\-RAG retriever design\.Together, these findings suggest several directions for future retriever development\. First, bridging the modality gap requires explicit cross\-modal alignment objectives or supervision, rather than relying on mixed\-modality indexing to induce cross\-modal matching implicitly\. Second, multimodal retrieval should prioritize representation learning that preserves fine\-grained visual information and aligns it directly with KG semantics, as lossy caption\-based conversions consistently limit performance\. Third, domain\-aware refinement becomes increasingly important in settings with subtle visual distinctions and specialized terminology, motivating stronger reranking or verifier\-style matching modules and the use of domain\-adapted visual and textual encoders\. Overall, effective MKG\-RAG retrieval is likely to require the integration of explicit cross\-modal alignment, robust multimodal embeddings, and domain\-sensitive refinement\.

Table 5\.Generation results \(%\) onMKG\-RAG\-Bench\-G andMKG\-RAG\-Bench\-M withK=5K=5\.
### 4\.3\.Generation Evaluation

Evaluation Configuration\.For generation evaluation, we include ageneration\-onlybaseline \(i\.e\., RAG\-free\) that queries the LLM without retrieval augmentation, and evaluate RAG variants built on the same retrievers, modality settings, and implementations described in Section[4\.2](https://arxiv.org/html/2606.26458#S4.SS2)\. For each query, we provide the top\-KKretrieved triplets as additional context to the generator, usingK=5K\{=\}5to balance evaluation fidelity with computational cost and token budget\. We use GPT\-5\(Singhet al\.,[2025](https://arxiv.org/html/2606.26458#bib.bib50)\)as the generation model\. Generation is evaluated as an open\-ended \(visual\) question answering task using Exact Match \(EM\), token\-level F1, Contains@​1@1, and BLEU\-1\.

Generation Results and Key Findings\.Table[5](https://arxiv.org/html/2606.26458#S4.T5)reports generation performance onMKG\-RAG\-Bench\-G andMKG\-RAG\-Bench\-M under different query and knowledge modality combinations\. Rather than enumerating individual scores, we summarize the results through several key findings that reveal how retrieval quality and modality interact to shape end\-to\-end MKG\-RAG performance\.

Finding 4:Retrieval provides genuine but uneven utility for generation\.Compared to the generation\-only baseline, retrieval\-augmented generation yields consistent improvements across most metrics and settings, confirming that retrieved multimodal knowledge can be effectively consumed by the generator\. Notably, even Random retrieval occasionally improves performance, indicating that when relevant triplets appear in the context, the generator can exploit them\. This serves as a sanity check for the alignment between constructed knowledge, queries, and answers, and validates the overall quality of the benchmark\.

Finding 5:Generation gains are substantially weaker in multimodal regimes\.Retrieval augmentation delivers the largest gains when both queries and supporting knowledge are textual\. In contrast, for multimodal queries, improvements are often marginal unless a strong multimodal retriever is used, and even then absolute gains remain limited—particularly onMKG\-RAG\-Bench\-M\. These results indicate that end\-to\-end generation is highly sensitive to whether retrieval can reliably supply visually grounded evidence\. Simply incorporating multimodal knowledge into the pipeline does not automatically translate into proportional generation improvements, revealing a key bottleneck in current MKG\-RAG systems\.

Finding 6:Generation performance closely tracks retrieval quality\.Generation trends closely mirror retrieval behavior\. Text\-only and captioning\-based retrieval pipelines provide limited gains for multimodal queries because they either ignore visual information or compress it through lossy captions, constraining evidence quality before generation begins\. In contrast, embedding\-based multimodal retrievers yield the most reliable improvements in multimodal settings\. Fusion\-based methods are generally more effective onMKG\-RAG\-Bench\-G, while reranking\-based approaches offer additional benefits onMKG\-RAG\-Bench\-M, where subtle visual distinctions and specialized terminology demand stronger refinement\. This alignment highlights retrieval quality as a primary driver of end\-to\-end MKG\-RAG generation performance\.

Implications: toward graph\-aware multimodal generation\.The comparatively modest gains in multimodal settings suggest that current multimodal retrievers, largely designed for unstructured corpus retrieval, do not fully exploit the structure and semantics of MKG\. Improving MKG\-RAG generation, therefore, likely requires retrieval methods that are explicitly graph\-aware, such as relation\-sensitive matching, neighborhood\- or path\-based evidence aggregation, and graph\-constrained candidate selection to reduce distractors\. Beyond retrieval, there is also headroom in how evidence is presented to the generator, including improved triplet selection, ordering, and representation formats that better convey structured graph information\. Overall, these findings suggest that progress in MKG\-RAG will depend less on scaling generic multimodal retrieval and more on developing graph\-aware, domain\-sensitive retrieval and evidence integration strategies\.

Table 6\.Generation results \(%\) for validating the effectiveness of LLM\-based utility filtering strategy\.Table 7\.Cross\-LLM consistency analysis\.
### 4\.4\.Ablation on LLM\-based Utility Filtering

A key design choice inMKG\-RAG\-Benchis to filter out low\-utility triplets that tend to produce ambiguous or weakly grounded queries\. To validate this choice, we conduct an ablation study comparing questions derived fromfilteredtriplets and those derived from theremainingtriplets retained in the benchmark\. We randomly sample 150 multimodal questions from each group and evaluate MKG\-RAG performance under Setting \(S4\) using our strongest Fusion\-based retriever, with both RAG\-free and retrieval\-augmented generation\. Results are reported in Table[6](https://arxiv.org/html/2606.26458#S4.T6)\. Across all metrics, questions derived from filtered triplets consistently yield lower performance than those derived from remaining triplets, even without retrieval augmentation, indicating that such questions are intrinsically difficult and poorly specified\. Although retrieval improves performance in both cases, gains on filtered\-triplet questions are substantially smaller, suggesting that retrieval cannot compensate for unclear or weakly grounded information needs\. These results demonstrate that low\-utility triplets confound reliable MKG\-RAG evaluation, and therefore validate the necessity of LLM\-based utility filtering to ensure clear, grounded queries and faithful assessment of retrieval and generation performance\.

### 4\.5\.Discussion on LLM Dependency

To evaluate whether the pipeline depends heavily on a specific LLM, we conduct a cross\-LLM consistency analysis\. Besides the GPT\-based construction pipeline, we apply Qwen3\.5\(Team,[2026](https://arxiv.org/html/2606.26458#bib.bib13)\)and Gemini 2\.5 Flash\(Comaniciet al\.,[2025](https://arxiv.org/html/2606.26458#bib.bib14)\)to a 300\-sample subset, including 150 text\-only samples and 150 multimodal samples\. We compare their outputs with GPT\-based results using filtering alignment and question\-level semantic similarity measured by BERTScore\. As shown in Table[7](https://arxiv.org/html/2606.26458#S4.T7), both alternative LLMs achieve high filtering alignment with GPT\-based outputs, with Qwen3\.5 reaching 99\.7% and Gemini 2\.5 Flash reaching 97\.0%\. The generated questions also show strong semantic consistency, with a BERTScore of 0\.8818 for both models\. These results indicate that the filtering and synthesis process is robust to the choice of LLM backbone\.

We further conduct a human evaluation to assess the reliability of the filtering and question synthesis results\. Human annotators reach 88% agreement with the filtering decisions, and the generated questions receive an average quality and diversity score of 3\.97 out of 5\. Together, the cross\-LLM and human evaluation results show that the benchmark construction process maintains stable behavior across different LLMs and introduces limited model\-specific bias in practice\.

## 5\.Conclusion

This work demonstrates that progress in MKG\-RAG is increasingly constrained by retrieval, a component that has received limited attention largely due to the lack of suitable benchmarks\. To address this gap, we introduceMKG\-RAG\-Bench, a cross\-domain benchmark explicitly designed to evaluate retrieval in MKG\-RAG, comprising two multimodal knowledge graphs and carefully aligned question\-answering datasets\.MKG\-RAG\-Benchis constructed through an LLM\-based curation pipeline that filters low\-utility knowledge and systematically covers diverse modality configurations, enabling controlled and realistic retrieval evaluation\. Extensive experiments across representative retriever families reveal that effectively leveraging multimodal signals during retrieval remains both challenging and critical for end\-to\-end performance\. By isolating and diagnosing these challenges,MKG\-RAG\-Benchprovides a principled foundation for developing, evaluating, and advancing more effective retrievers for MKG\-RAG\. We hope this benchmark will catalyze future research on graph\-aware, domain\-adaptive multimodal retrieval methods and more effective integration of structured knowledge into multimodal generation\.

## Acknowledgements

This research was partially supported by a 2025/2026 Rising Researcher Grant from Penn State’s Institute for Computational & Data Sciences \(RRID:SCR\_025154\) and the National Science Foundation under Grant No\. 2333790 and 2238275\.

## References

- O\. Bodenreider \(2004\)The unified medical language system \(UMLS\): integrating biomedical terminology\.Nucleic Acids Res\.32\(Database issue\),pp\. D267–70\(en\)\.Cited by:[2nd item](https://arxiv.org/html/2606.26458#A1.I1.i2.p1.1)\.
- J\. Chen, R\. Xu, Z\. Fu, W\. Shi, Z\. Li, X\. Zhang, C\. Sun, L\. Li, Y\. Xiao, and H\. Zhou \(2022a\)E\-kar: a benchmark for rationalizing natural language analogical reasoning\.arXiv preprint arXiv:2203\.08480\.Cited by:[1st item](https://arxiv.org/html/2606.26458#A1.I1.i1.p1.1)\.
- W\. Chen, H\. Hu, X\. Chen, P\. Verga, and W\. Cohen \(2022b\)Murag: multimodal retrieval\-augmented generator for open question answering over images and text\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 5558–5570\.Cited by:[§1](https://arxiv.org/html/2606.26458#S1.p2.6),[2nd item](https://arxiv.org/html/2606.26458#S2.I1.i2.p1.1),[§4\.2](https://arxiv.org/html/2606.26458#S4.SS2.p1.3)\.
- Z\. Chen, C\. Xu, Y\. Qi, X\. Jiang, and J\. Guo \(2025\)VLM is a strong reranker: advancing multimodal retrieval\-augmented generation via knowledge\-enhanced reranking and noise\-injected training\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 8140–8158\.Cited by:[4th item](https://arxiv.org/html/2606.26458#S2.I1.i4.p1.1),[§4\.2](https://arxiv.org/html/2606.26458#S4.SS2.p1.3)\.
- G\. Comanici, E\. Bieber, M\. Schaekermann, I\. Pasupat, N\. Sachdeva, I\. Dhillon, M\. Blistein, O\. Ram, D\. Zhang, E\. Rosen,et al\.\(2025\)Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv preprint arXiv:2507\.06261\.Cited by:[§4\.5](https://arxiv.org/html/2606.26458#S4.SS5.p1.1)\.
- K\. Dong, Y\. Chang, S\. Huang, Y\. Wang, R\. Tang, and Y\. Liu \(2025\)Benchmarking retrieval\-augmented multimodal generation for document question answering\.arXiv preprint arXiv:2505\.16470\.Cited by:[Table 1](https://arxiv.org/html/2606.26458#S1.T1.1.1.12.10.1)\.
- A\. Drozd, A\. Gladkova, and S\. Matsuoka \(2016\)Word embeddings, analogies, and machine learning: beyond king\-man\+ woman= queen\.InProceedings of coling 2016,pp\. 3519–3530\.Cited by:[1st item](https://arxiv.org/html/2606.26458#A1.I1.i1.p1.1)\.
- D\. Edge, H\. Trinh, N\. Cheng, J\. Bradley, A\. Chao, A\. Mody, S\. Truitt, D\. Metropolitansky, R\. O\. Ness, and J\. Larson \(2024\)From local to global: a graph rag approach to query\-focused summarization\.arXiv preprint arXiv:2404\.16130\.Cited by:[§1](https://arxiv.org/html/2606.26458#S1.p1.1)\.
- F\. Gao, Q\. Ping, G\. Thattai, A\. Reganti, Y\. N\. Wu, and P\. Natarajan \(2022\)Transform\-retrieve\-generate: natural language\-centric outside\-knowledge visual question answering\.InCVPR 2022,pp\. 5057–5067\.Cited by:[3rd item](https://arxiv.org/html/2606.26458#S2.I1.i3.p1.1),[§4\.2](https://arxiv.org/html/2606.26458#S4.SS2.p1.3)\.
- Y\. Gu, S\. Kase, M\. Vanni, B\. Sadler, P\. Liang, X\. Yan, and Y\. Su \(2021\)Beyond iid: three levels of generalization for question answering on knowledge bases\.InProceedings of the web conference 2021,pp\. 3477–3488\.Cited by:[Table 1](https://arxiv.org/html/2606.26458#S1.T1.1.1.8.6.1)\.
- X\. He, Y\. Tian, Y\. Sun, N\. Chawla, T\. Laurent, Y\. LeCun, X\. Bresson, and B\. Hooi \(2024\)G\-retriever: retrieval\-augmented generation for textual graph understanding and question answering\.Advances in Neural Information Processing Systems37,pp\. 132876–132907\.Cited by:[§4\.2](https://arxiv.org/html/2606.26458#S4.SS2.p1.3)\.
- Y\. Hu, Z\. Lei, Z\. Zhang, B\. Pan, C\. Ling, and L\. Zhao \(2025\)GRAG: graph retrieval\-augmented generation\.External Links:2405\.16506,[Link](https://arxiv.org/abs/2405.16506)Cited by:[§4\.2](https://arxiv.org/html/2606.26458#S4.SS2.p1.3)\.
- A\. E\. W\. Johnson, T\. J\. Pollard, S\. J\. Berkowitz, N\. R\. Greenbaum, M\. P\. Lungren, C\. Deng, R\. G\. Mark, and S\. Horng \(2019\)MIMIC\-CXR, a de\-identified publicly available database of chest radiographs with free\-text reports\.Sci\. Data6\(1\),pp\. 317\(en\)\.Cited by:[2nd item](https://arxiv.org/html/2606.26458#A1.I1.i2.p1.1)\.
- V\. Karpukhin, B\. Oğuz, S\. Min, P\. Lewis, L\. Wu, S\. Edunov, D\. Chen, and W\. Yih \(2020\)Dense passage retrieval for open\-domain question answering\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2004.04906),[Link](https://arxiv.org/abs/2004.04906)Cited by:[1st item](https://arxiv.org/html/2606.26458#S2.I1.i1.p1.1)\.
- J\. J\. Lau, S\. Gayen, A\. Ben Abacha, and D\. Demner\-Fushman \(2018\)A dataset of clinically generated visual questions and answers about radiology images\.Scientific data5\(1\),pp\. 1–10\.Cited by:[§2\.3](https://arxiv.org/html/2606.26458#S2.SS3.p2.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.\(2020\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in neural information processing systems33,pp\. 9459–9474\.Cited by:[§1](https://arxiv.org/html/2606.26458#S1.p1.1)\.
- J\. Li, D\. Li, C\. Xiong, and S\. Hoi \(2022\)Blip: bootstrapping language\-image pre\-training for unified vision\-language understanding and generation\.InInternational conference on machine learning,pp\. 12888–12900\.Cited by:[§4\.2](https://arxiv.org/html/2606.26458#S4.SS2.p1.3)\.
- M\. Li, S\. Miao, and P\. Li \(2024\)Simple is effective: the roles of graphs and large language models in knowledge\-graph\-based retrieval\-augmented generation\.arXiv preprint arXiv:2410\.20724\.Cited by:[§2\.1](https://arxiv.org/html/2606.26458#S2.SS1.p2.1)\.
- S\. Li, Y\. Gao, H\. Jiang, Q\. Yin, Z\. Li, X\. Yan, C\. Zhang, and B\. Yin \(2023\)Graph reasoning for question answering with triplet retrieval\.InACL \(Findings\),Cited by:[§4\.2](https://arxiv.org/html/2606.26458#S4.SS2.p1.3)\.
- \[20\]Y\. Li, Y\. Li, X\. Wang, Y\. Jiang, Z\. Zhang, X\. Zheng, H\. Wang, H\. Zheng, F\. Huang, J\. Zhou,et al\.Benchmarking multimodal retrieval augmented generation with dynamic vqa dataset and self\-adaptive planning agent\.InThe Thirteenth ICLR,Cited by:[Table 1](https://arxiv.org/html/2606.26458#S1.T1.1.1.5.3.1),[§2\.1](https://arxiv.org/html/2606.26458#S2.SS1.p1.1)\.
- B\. Liu, L\. Zhan, L\. Xu, L\. Ma, Y\. Yang, and X\. Wu \(2021\)SLAKE: a semantically\-labeled knowledge\-enhanced dataset for medical visual question answering\.External Links:2102\.09542,[Link](https://arxiv.org/abs/2102.09542)Cited by:[§2\.3](https://arxiv.org/html/2606.26458#S2.SS3.p2.1)\.
- Y\. Liu, H\. Li, A\. Garcia\-Duran, M\. Niepert, D\. Onoro\-Rubio, and D\. S\. Rosenblum \(2019\)MMKG: multi\-modal knowledge graphs\.InEuropean Semantic Web Conference,pp\. 459–474\.Cited by:[§2\.3](https://arxiv.org/html/2606.26458#S2.SS3.p1.1)\.
- L\. Luo, Y\. Li, G\. Haffari, and S\. Pan \(2023\)Reasoning on graphs: faithful and interpretable large language model reasoning\.arXiv preprint arXiv:2310\.01061\.Cited by:[§2\.1](https://arxiv.org/html/2606.26458#S2.SS1.p2.1)\.
- Y\. Ma, Y\. Zang, L\. Chen, M\. Chen, Y\. Jiao, X\. Li, X\. Lu, Z\. Liu, Y\. Ma, X\. Dong,et al\.\(2024a\)Mmlongbench\-doc: benchmarking long\-context document understanding with visualizations\.Advances in Neural Information Processing Systems37,pp\. 95963–96010\.Cited by:[Table 1](https://arxiv.org/html/2606.26458#S1.T1.1.1.11.9.1)\.
- Z\. Ma, T\. Lan, R\. Tu, Y\. Hu, Y\. Zhu, T\. Zhang, H\. Huang, Z\. Wu, and X\. Mao \(2024b\)Multi\-modal retrieval augmented multi\-modal generation: datasets, evaluation metrics and strong baselines\.arXiv preprint arXiv:2411\.16365\.Cited by:[Table 1](https://arxiv.org/html/2606.26458#S1.T1.1.1.1.1),[§2\.1](https://arxiv.org/html/2606.26458#S2.SS1.p1.1)\.
- K\. Marino, M\. Rastegari, A\. Farhadi, and R\. Mottaghi \(2019\)Ok\-vqa: a visual question answering benchmark requiring external knowledge\.InProceedings of CVPR,pp\. 3195–3204\.Cited by:[Table 1](https://arxiv.org/html/2606.26458#S1.T1.1.1.3.1.1)\.
- L\. Mei, S\. Mo, Z\. Yang, and C\. Chen \(2025\)A survey of multimodal retrieval\-augmented generation\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2504.08748),[Link](https://arxiv.org/abs/2504.08748)Cited by:[2nd item](https://arxiv.org/html/2606.26458#S2.I1.i2.p1.1),[§4\.2](https://arxiv.org/html/2606.26458#S4.SS2.p1.3)\.
- A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark,et al\.\(2021\)Learning transferable visual models from natural language supervision\.InInternational conference on machine learning,pp\. 8748–8763\.Cited by:[§4\.2](https://arxiv.org/html/2606.26458#S4.SS2.p1.3)\.
- N\. Reimers and I\. Gurevych \(2019\)Sentence\-bert: sentence embeddings using siamese bert\-networks\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.1908.10084),[Link](https://arxiv.org/abs/1908.10084)Cited by:[1st item](https://arxiv.org/html/2606.26458#S2.I1.i1.p1.1)\.
- S\. Robertson and H\. Zaragoza \(2009\)The probabilistic relevance framework: bm25 and beyond\.Foundations and Trends® in Information Retrieval3\(4\),pp\. 333–389\.External Links:ISSN 1554\-0677,[Link](http://dx.doi.org/10.1561/1500000019),[Document](https://dx.doi.org/10.1561/1500000019)Cited by:[1st item](https://arxiv.org/html/2606.26458#S2.I1.i1.p1.1)\.
- N\. Y\. Sanket Shah and P\. P\. Talukdar \(2019\)KVQA: knowledge\-aware visual question answering\.InAAAI,Cited by:[§2\.3](https://arxiv.org/html/2606.26458#S2.SS3.p1.1)\.
- C\. Schuhmann, R\. Beaumont, R\. Vencu, C\. Gordon, R\. Wightman, M\. Cherti, T\. Coombes, A\. Katta, C\. Mullis, M\. Wortsman, P\. Schramowski, S\. Kundurthy, K\. Crowson, L\. Schmidt, R\. Kaczmarczyk, and J\. Jitsev \(2022\)LAION\-5b: an open large\-scale dataset for training next generation image\-text models\.External Links:2210\.08402,[Link](https://arxiv.org/abs/2210.08402)Cited by:[1st item](https://arxiv.org/html/2606.26458#A1.I1.i1.p1.1)\.
- D\. Schwenk, A\. Khandelwal, C\. Clark, K\. Marino, and R\. Mottaghi \(2022\)A\-okvqa: a benchmark for visual question answering using world knowledge\.InECCV,pp\. 146–162\.Cited by:[§2\.3](https://arxiv.org/html/2606.26458#S2.SS3.p1.1)\.
- A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram,et al\.\(2025\)Openai gpt\-5 system card\.arXiv preprint arXiv:2601\.03267\.Cited by:[§3\.1](https://arxiv.org/html/2606.26458#S3.SS1.p7.1),[§4\.3](https://arxiv.org/html/2606.26458#S4.SS3.p1.3)\.
- A\. Talmor and J\. Berant \(2018\)The web as a knowledge\-base for answering complex questions\.arXiv preprint arXiv:1803\.06643\.Cited by:[Table 1](https://arxiv.org/html/2606.26458#S1.T1.1.1.7.5.1)\.
- Q\. Team \(2026\)Qwen3\.5: accelerating productivity with native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§4\.5](https://arxiv.org/html/2606.26458#S4.SS5.p1.1)\.
- J\. Wang, X\. Yang, K\. Sun, P\. Suresh, S\. Sharma, A\. Czyzewski, D\. Andersen, S\. Appini, A\. Banerjee, S\. Choudhary,et al\.\(2025a\)CRAG\-mm: multi\-modal multi\-turn comprehensive rag benchmark\.arXiv preprint arXiv:2510\.26160\.Cited by:[Table 1](https://arxiv.org/html/2606.26458#S1.T1.1.1.4.2.1),[§2\.1](https://arxiv.org/html/2606.26458#S2.SS1.p1.1)\.
- X\. Wang, Z\. Wu, Y\. Zhong, X\. Zhang, S\. Wang, and F\. Ma \(2026\)GPR: empowering generation with graph\-pretrained retriever\.InProceedings of the ACM Web Conference 2026,pp\. 8349–8352\.Cited by:[§1](https://arxiv.org/html/2606.26458#S1.p1.1)\.
- X\. Wang, Y\. Zhong, L\. Zhang, L\. Dai, T\. Wang, and F\. Ma \(2025b\)MEDMKG: benchmarking medical knowledge exploitation with multimodal knowledge graph\.External Links:2505\.17214,[Link](https://arxiv.org/abs/2505.17214)Cited by:[2nd item](https://arxiv.org/html/2606.26458#A1.I1.i2.p1.1),[§2\.3](https://arxiv.org/html/2606.26458#S2.SS3.p2.1),[§3](https://arxiv.org/html/2606.26458#S3.p1.1)\.
- X\. Wang, B\. Meng, H\. Chen, Y\. Meng, K\. Lv, and W\. Zhu \(2023\)TIVA\-kg: a multimodal knowledge graph with text, image, video and audio\.InProceedings of MM’23,New York, NY, USA,pp\. 2391–2399\.External Links:ISBN 9798400701085,[Link](https://doi.org/10.1145/3581783.3612266),[Document](https://dx.doi.org/10.1145/3581783.3612266)Cited by:[§2\.3](https://arxiv.org/html/2606.26458#S2.SS3.p1.1)\.
- N\. Wasserman, R\. Pony, O\. Naparstek, A\. R\. Goldfarb, E\. Schwartz, U\. Barzelay, and L\. Karlinsky \(2025\)REAL\-mm\-rag: a real\-world multi\-modal retrieval benchmark\.InAnnual Meeting of the Association for Computational Linguistics,Cited by:[Table 1](https://arxiv.org/html/2606.26458#S1.T1.1.1.9.7.1)\.
- Y\. Yan and W\. Xie \(2024\)EchoSight: advancing visual\-language models with wiki knowledge\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 1538–1551\.Cited by:[4th item](https://arxiv.org/html/2606.26458#S2.I1.i4.p1.1),[§4\.2](https://arxiv.org/html/2606.26458#S4.SS2.p1.3)\.
- W\. Yih, M\. Richardson, C\. Meek, M\. Chang, and J\. Suh \(2016\)The value of semantic parse labeling for knowledge base question answering\.InProceedings of the 54th ACL \(Volume 2: Short Papers\),pp\. 201–206\.Cited by:[Table 1](https://arxiv.org/html/2606.26458#S1.T1.1.1.6.4.1),[§3\.1](https://arxiv.org/html/2606.26458#S3.SS1.p3.1)\.
- W\. Zhai \(2024\)Self\-adaptive multimodal retrieval\-augmented generation\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2410.11321),[Link](https://arxiv.org/abs/2410.11321)Cited by:[3rd item](https://arxiv.org/html/2606.26458#S2.I1.i3.p1.1),[§4\.2](https://arxiv.org/html/2606.26458#S4.SS2.p1.3)\.
- N\. Zhang, L\. Li, X\. Chen, X\. Liang, S\. Deng, and H\. Chen \(2023\)Multimodal analogical reasoning over knowledge graphs\.External Links:2210\.00312,[Link](https://arxiv.org/abs/2210.00312)Cited by:[1st item](https://arxiv.org/html/2606.26458#A1.I1.i1.p1.1),[§3](https://arxiv.org/html/2606.26458#S3.p1.1)\.
- A\. Zou, W\. Yu, H\. Zhang, K\. Ma, D\. Cai, Z\. Zhang, H\. Zhao, and D\. Yu \(2025\)Docbench: a benchmark for evaluating llm\-based document reading systems\.InProceedings of the 4th International Workshop on Knowledge\-Augmented Methods for Natural Language Processing,pp\. 359–373\.Cited by:[Table 1](https://arxiv.org/html/2606.26458#S1.T1.1.1.10.8.1)\.

Utility Filtering PromptYou curate and write natural\-language queries for a KG retrieval dataset\.Modality: \{multimodal or text\-only\}\.At query time, the head is given as: \{IMAGE or TEXT\}\.Task: This group corresponds to \{FIXED \(head, relation\) with MULTIPLE possible tailsORFIXED \(head, tail\) with MULTIPLE possible relations\}\.You must decide if this group is meaningful for generating questions that ask for the missing part\.Rules:1\) Output MUST be JSON:`\{"keep": bool, "reason": str\}`\.2\) Return ONLY JSON \(no extra keys\)\.3\) Decide keep based primarily on relation\_text and tail examples\.4\) Because head\_text may be uninformative for images, DO NOT reject just because head\_text is generic\.5\) Set keep=false only when the tails are mostly placeholders or administrative/meta concepts, e\.g\.,tails like ‘thing/object/entity/space/place/environment/location/category’, or when the relation is too vague\.Input:GROUP TYPE: \{mask\_tail or mask\_relation\}head\_text: \{head\_text\}relation\_text: \{relation\_text\}tail\_text \(if mask\_relation\): \{tail\_text\}candidates\_count: \{cand\_total\}examples \(random 3\): \{cand\_examples\}candidates\_sample: \{cand\_sample\}

Question Generation PromptYou curate and write natural\-language queries for a KG retrieval dataset\.Modality: \{multimodal or text\-only\}\.At query time, the head is given as: \{IMAGE or TEXT\}\.Task: This group corresponds to \{FIXED \(head, relation\) with MULTIPLE possible tailsORFIXED \(head, tail\) with MULTIPLE possible relations\}\.Generate exactly K WH\-questions that ask for the missing part\.Rules:1\) Output MUST be JSON:`\{"questions": \[\.\.\]\}`\.2\) \{prefix\_rule\}3\) \{head\_leak\_rule\}4\) Do NOT copy relation\_text \(or candidate relation strings\) verbatim; paraphrase\.5\) If mask\_tail: Questions MUST NOT mention any specific tail candidates\.6\)≤\\leq25 words per question\. Avoid yes/no\.7\) Refer to the head as ‘\{head\_ref\}’ \(or equivalent if multimodal\)\.8\) If mask\_relation: You MAY mention tail\_text\.Input:GROUP TYPE: \{mask\_tail or mask\_relation\}head\_text: \{head\_text\}relation\_text: \{relation\_text\}tail\_text \(if mask\_relation\): \{tail\_text\}candidates\_count: \{cand\_total\}examples \(random 3\): \{cand\_examples\}candidates\_sample: \{cand\_sample\}K=\{k\}

Figure 3\.Two\-agent prompting for query construction: filtering low\-quality triplets and generating WH\-questions for the remaining triplets\.## Appendix ASource Knowledge Graphs

We use the following two knowledge graphs:

- •MarKG\.MarKG\(Zhanget al\.,[2023](https://arxiv.org/html/2606.26458#bib.bib10)\)is a multimodal knowledge graph constructed to support multimodal analogical reasoning, where the entities involved in an analogy \(e\.g\., head/tail/query entities\) may come from different modalities\. It is released together with the Multimodal Analogical Reasoning dataset \(MARS\) and is built by expanding and annotating seed entities and relations from E\-KAR\(Chenet al\.,[2022a](https://arxiv.org/html/2606.26458#bib.bib39)\)and BATS\(Drozdet al\.,[2016](https://arxiv.org/html/2606.26458#bib.bib40)\)\. In particular, MarKG links external entities to Wikidata and associates entities with images collected from LAION\-5B\(Schuhmannet al\.,[2022](https://arxiv.org/html/2606.26458#bib.bib16)\)\. Overall, MarKG contains 11,292 entities and 192 relation types, forming 34,420 triplets, with 76,424 linked images, providing multimodal grounding for general\-domain analogy\.
- •MedMKG\.For the multimodal knowledge graph, we use MedMKG\(Wanget al\.,[2025b](https://arxiv.org/html/2606.26458#bib.bib20)\)\. MedMKG is a multimodal knowledge graph that fuses rich imaging data from MIMIC\-CXR\(Johnsonet al\.,[2019](https://arxiv.org/html/2606.26458#bib.bib22)\)with structured clinical knowledge from the Unified Medical Language System \(UMLS\)\(Bodenreider,[2004](https://arxiv.org/html/2606.26458#bib.bib23)\), using both rule\-based tools and large language models for precise concept extraction and relationship modeling\. To enhance graph quality and compactness, Neighbor\-aware Filtering \(NaF\), a filtering algorithm specifically designed for multimodal knowledge graphs, is applied to MedMKG\. MedMKG comprises 53,587 edges, 4,868 images, and 262 relations over 3,148 concepts\.

Table 8\.Retrieval efficiency measured by Queries per Second \(QPS\) under an online serving setting with cached embeddings and warm\-started retrievers\. We conduct the experiments with all settings as detailed in Section[4\.1](https://arxiv.org/html/2606.26458#S4.SS1)\.
## Appendix BPrompts for Data Construction

DuringMKG\-RAG\-Benchconstruction, we prompt GPT\-5 twice: first to validate knowledge utility, and then to translate triplets into natural\-language queries\. The corresponding prompts are shown in Figure[3](https://arxiv.org/html/2606.26458#A0.F3)\.

## Appendix CImage Augmentation for Visual Questions

To prevent exact image duplication from serving as a shortcut in multimodal retrieval, we apply image augmentation to the images used in visual questions\. For each question image, we randomly apply one augmentation from the following set:swapping to another image in the knowledge graph that refers to the same entity, random cropping, rotation, or color jitter\. These augmentations ensure that the image in the visual question differs from the image associated with the target triplet, eliminating trivial matches and better simulating realistic retrieval settings where an exact duplicate of the query image is unlikely to appear in the knowledge graph\.

## Appendix DExample ofMKG\-RAG\-BenchConstruction

We use two examples to illustrate how the proposed pipeline generates benchmark data with high\-utility triplets\.

- •Text\-only Query\.We take \(Antigua and Barbuda, member\_of, \[MASK\]\) as example\. The pipeline will generate a query first, which is“Of which global or intergovernmental bodies is Antigua and Barbuda a member?”For theretrievaltask, the ground truth data include several high\-utility triples, such as \{\(Antigua and Barbuda, member\_of, Commonwealth of Nations\), \(Antigua and Barbuda, member\_of, United Nations\), \(Antigua and Barbuda, member\_of, World Trade Organization\)\. For thegenerationtask, the answers are \{Commonwealth of Nations, United Nations, World Trade Organization\}\.
- •Multimodal Query\.If there is a high\-utility multimodal triplet \(People’s Republic of China![[Uncaptioned image]](https://arxiv.org/html/2606.26458v1/fig/china-flag-icon.png), railway\_traffic\_side, left\), the entity “left” is then masked, leading to the generation of the multimodal query:![[Uncaptioned image]](https://arxiv.org/html/2606.26458v1/fig/tiananmen.jpg)“On which side do trains operate in the subject’s country?”Note that the image entities can be different\. For theretrievaltask, the ground triplet is the original high\-utility triplet, and for thegenerationtask, the answer is the mask “left”\.

## Appendix EDetails about Retriever Implementation

The proposedMKG\-RAG\-Benchis ageneralbenchmark, and any retrievers can be used\. However, in the current version, we implement the following retrievers in the experiments\.

For text\-only queries and triplets, all retrievers that share the same CLIP implementation produce identical embeddings by encoding text solely with CLIP’s text encoder\. As a result, they achieve the same performance on text\-only query–text\-only triplet retrieval due to this homogeneous encoding\. For multimodal retrieval, the text\-only retriever ignores visual signals and encodes only the textual entity associated with the image\. The captioning\-based retriever first uses BLIP to convert the image into a textual description, and then encodes the resulting text with CLIP’s text encoder\.

The remaining two retrievers adopt more advanced multimodal strategies\. We implement the fusion retriever using a late\-fusion design, where the visual embedding from CLIP’s image encoder is averaged with the text embedding of the textual part of each multimodal triplet\. The reranking retriever performs a two\-stage procedure: it first retrieves candidates using image\-embedding similarity, and then reranks the shortlisted triplets using text\-embedding similarity\.

## Appendix FRetrieval Efficiency Analysis

An additional study analyzes the retrieval efficiency of the baseline retrievers\. We report Queries per Second \(QPS\) under an online serving setting where all embeddings are cached and the retrievers are warm\-started\. The results are shown in Table[8](https://arxiv.org/html/2606.26458#A1.T8)\. Overall, the findings highlight a clearefficiency–effectiveness trade\-off: in the multimodal setting, fusion\-based and reranking\-based retrievers exhibit roughly an order\-of\-magnitude lower throughput \(about 0\.1×\\timesQPS\) than the remaining baselines\. By contrast, captioning\-based retrieval benefits substantially from caching and achieves competitive throughput, approaching that of the text\-only retriever\. Taken together, this study quantifies the efficiency cost of stronger multimodal modeling and provides practical guidance for deploying MKG\-RAG systems under real\-time constraints\.