VISPATH：面向多模态知识图谱问答的视觉意图引导路径推理

# VisPath: Visual-Intent-Guided Path Reasoning for Multimodal Knowledge Graph Question Answering
Source: [https://arxiv.org/html/2609.20843](https://arxiv.org/html/2609.20843)
2ndZhengpin Li\*3rdMengzhe Jia4thYang Li5thWentao Zhang\*Jinke Wu and Zhengpin Li contributed equally to this work\.

###### Abstract

Knowledge graph question answering \(KGQA\) enables models to answer natural\-language questions through structured graph reasoning and has achieved substantial progress across many benchmarks and applications\. Recently, multimodal KGQA \(MM\-KGQA\) has attracted increasing attention because many questions require jointly using multimodal inputs and KG evidence\. However, existing MM\-KGQA methods typically use multimodal information only for starting entity grounding or evidence retrieval, after which multi\-hop reasoning degenerates into text\-only graph search\. As a result, they cannot exploit multimodal cues that become important at intermediate hops\. To address this limitation, we proposeVisPath, a visual\-intent\-guided path reasoning framework for MM\-KGQA\.VisPathfirst identifies a reliable starting entity by combining multimodal grounding with graph\-structural cues\. It then performs intent\-guided path discovery by recomputing hop\-specific multimodal intent from the input, question, and current partial paths, so that each expansion is guided by the current reasoning state\. The discovered paths are further refined through reasoning\-chain pruning, which evaluates candidate paths as complete evidence chains based on their consistency with the question, reasoning sketch, and hop\-specific intent\. Finally,VisPathchecks whether the selected evidence is sufficient for answer generation\. We further constructVisPath\-Bench, a benchmark for evaluating multimodal multi\-hop reasoning over KGs, covering questions that require two to four hops over KG paths\. Extensive experiments onVisPath\-Bench and three additional multimodal QA benchmarks show thatVisPathconsistently outperforms strong KGQA, vision–language, and retrieval\-augmented baselines\. Notably, with GPT\-4o as the backbone,VisPathsurpasses GPT\-5\.4 onVisPath\-Bench, achieving a 10\.6% relative improvement in average accuracy and a 13\.1% improvement at 2\-hop reasoning\.

\*\*footnotetext:Jinke Wu and Zhengpin Li contributed equally to this work\.## IIntroduction

Knowledge graphs \(KGs\) have become a widely adopted form of structured knowledge representation\[[16](https://arxiv.org/html/2609.20843#bib.bib45),[32](https://arxiv.org/html/2609.20843#bib.bib54)\]\. By organizing real\-world knowledge into entities, relations, and factual triples, KGs provide an explicit and interpretable foundation for knowledge access and reasoning\. Due to their structured nature, KGs have been widely used in search engines, recommender systems, question answering, and knowledge\-intensive reasoning tasks\[[36](https://arxiv.org/html/2609.20843#bib.bib21),[26](https://arxiv.org/html/2609.20843#bib.bib4),[58](https://arxiv.org/html/2609.20843#bib.bib38),[46](https://arxiv.org/html/2609.20843#bib.bib37),[24](https://arxiv.org/html/2609.20843#bib.bib3)\]\. Compared with unstructured text corpora, KGs make relational dependencies more explicit and allow the reasoning process to be traced through supporting facts\[[29](https://arxiv.org/html/2609.20843#bib.bib24),[25](https://arxiv.org/html/2609.20843#bib.bib5)\]\.

One important application of KGs is knowledge graph question answering \(KGQA\)\. KGQA aims to answer natural\-language questions by locating relevant entities, traversing relations, and reasoning over graph facts\. For example, given the question “Where was the architect of the Government Museum, Chandigarh born?”, a KGQA system needs to first locate the entityGovernment Museum, Chandigarh, follow the relationarchitecttoLe Corbusier, and then followplace of birthto infer the answerLa Chaux\-de\-Fonds\. This example illustrates the key strength of KGQA: the answer is not generated only from textual patterns, but is derived through an explicit reasoning path over structured facts\. Existing KGQA methods have achieved substantial progress, ranging from semantic parsing\[[6](https://arxiv.org/html/2609.20843#bib.bib41),[54](https://arxiv.org/html/2609.20843#bib.bib15)\]and neural graph reasoning\[[43](https://arxiv.org/html/2609.20843#bib.bib43),[14](https://arxiv.org/html/2609.20843#bib.bib19)\]to recent LLM\-based graph exploration methods\[[44](https://arxiv.org/html/2609.20843#bib.bib23),[9](https://arxiv.org/html/2609.20843#bib.bib27),[45](https://arxiv.org/html/2609.20843#bib.bib28)\]\. With these advances, KGQA methods have achieved state\-of\-the\-art performance across many benchmarks and application scenarios, demonstrating the effectiveness of structured graph reasoning for knowledge\-intensive question answering\.

Despite the progress of KGQA, most existing methods are designed for unimodal settings\[[44](https://arxiv.org/html/2609.20843#bib.bib23),[29](https://arxiv.org/html/2609.20843#bib.bib24),[45](https://arxiv.org/html/2609.20843#bib.bib28)\]\. They typically assume that both the question and the knowledge graph are textual or symbolic, and that the starting entity can be identified from the question itself\. However, many real\-world questions are naturally multimodal, where the entity to be queried may only be specified by an image rather than by its textual name\. In such cases, a model must first ground the visual content to the correct graph entity and then reason over the KG to obtain the answer\. This makes standard KGQA methods insufficient, since they do not explicitly model the interaction between visual grounding and graph reasoning\.

Recently, several studies have begun to explore multimodal knowledge graph question answering \(MM\-KGQA\)\. In this task, a model is required to answer a question by jointly using visual inputs and structured KG evidence\. Existing efforts show that multimodal evidence is important for connecting visual content with encyclopedic knowledge and for answering questions beyond what can be inferred from the image alone\. Some methods construct multimodal KGs from documents or retrieve entity\- and relation\-level evidence for visual question answering\[[55](https://arxiv.org/html/2609.20843#bib.bib52)\]\. Meanwhile, several benchmarks have been proposed to evaluate multi\-hop visual questions that require external knowledge beyond the image\[[46](https://arxiv.org/html/2609.20843#bib.bib37),[58](https://arxiv.org/html/2609.20843#bib.bib38),[32](https://arxiv.org/html/2609.20843#bib.bib54)\]\.

However, existing MM\-KGQA methods still have an essential limitation\. Visual information is often used only at the beginning of the reasoning process, such as for starting entity grounding or evidence retrieval\. Once the starting entity or relevant evidence is obtained, the subsequent reasoning process usually degenerates into text\-only graph search\. This decoupled paradigm makes later reasoning heavily dependent on the correctness of the initial grounding\. More importantly, it prevents the model from using visual cues that may become relevant at intermediate hops, even though such cues can be crucial for selecting the correct relation or path\.

To address this limitation, we proposeVisPath, a visual\-intent\-guided path reasoning framework for MM\-KGQA\. Instead of treating multimodal information as a one\-time signal for starting entity grounding,VisPathkeeps the interaction among the multimodal input, the question, and the multimodal KG \(MMKG\) throughout the reasoning process\. First, for starting entity identification,VisPathuses the multimodal referring expression in the given question to guide object recognition from the multimodal input, and combines the recognized object with graph\-structural cues from the graph to select an entity that is both multimodally consistent and reasoning\-supportive\. Next, for path discovery, it recomputes a hop\-specific intent from the multimodal input, the question, and the current partial paths, so that each expansion step is guided by the current multimodal reasoning state rather than by the question alone\. Then, for reasoning\-chain pruning, it evaluates candidate paths as complete evidence chains by considering their consistency with the question, the global reasoning sketch, and the multimodal intent of the current hop\. Finally, before producing the answer,VisPathsummarizes the selected graph paths under the multimodal context and checks whether the resulting multimodal evidence is sufficient to support answer generation\. In this way,VisPathenables multimodal information to participate not only in entity grounding, but also in relation selection, path pruning, and answer verification, leading to reasoning paths that are both KG\-faithful and multimodally grounded across the entire reasoning process\.

We further introduceVisPath\-Bench, a new dataset designed to evaluate multimodal multi\-hop reasoning over KGs\. Existing MM\-KGQA datasets mainly focus on shallow reasoning\[[32](https://arxiv.org/html/2609.20843#bib.bib54),[46](https://arxiv.org/html/2609.20843#bib.bib37)\], making it difficult to assess whether a model can maintain multimodal grounding across deeper graph paths\. To fill this gap,VisPath\-Bench contains questions that require two to four hops of reasoning over KG paths\. Each question is constructed so that the starting entity cannot be directly read from the question text, but must be grounded from the multimodal input, while the final answer must be derived through structured graph reasoning\. In this way,VisPath\-Bench provides a more challenging and systematic resource for evaluating whether models can jointly perform multimodal grounding, multi\-hop path reasoning, and evidence\-faithful answer generation under realistic and complex settings\.

Our contributions are summarized as follows\.

- •We proposeVisPath, a visual\-intent\-guided path reasoning method that keeps visual evidence involved in hop\-by\-hop KG reasoning\.
- •We constructVisPath\-Bench, a benchmark for evaluating image\-grounded multi\-hop reasoning over MMKGs\.
- •Extensive experiments show thatVisPath, using GPT\-4o as the LLM backbone, consistently outperforms strong baselines, surpassing GPT\-5\.4\[[35](https://arxiv.org/html/2609.20843#bib.bib64)\]onVisPath\-Bench with a 10\.6% relative improvement in average accuracy and a 13\.1% relative improvement at 2\-hop reasoning\.

## IIRelated Work

### II\-AMultimodal Knowledge Graphs

Multimodal knowledge graphs \(MMKGs\) extend conventional knowledge graphs \(KGs\) by associating entities, relations, or triples with evidence from non\-textual modalities, such as images, audio, and video\[[27](https://arxiv.org/html/2609.20843#bib.bib1),[59](https://arxiv.org/html/2609.20843#bib.bib6)\]\. Existing MMKG resources differ mainly in the type of multimodal evidence they attach and the granularity at which such evidence is organized\. At the entity level, IKRL\[[52](https://arxiv.org/html/2609.20843#bib.bib39)\]pioneers image\-enhanced entity representations by encoding entity images into the embedding space\. MMKG\[[28](https://arxiv.org/html/2609.20843#bib.bib2)\]aligns FB15K\[[8](https://arxiv.org/html/2609.20843#bib.bib58)\]with DBpedia\[[3](https://arxiv.org/html/2609.20843#bib.bib59)\]and YAGO\[[41](https://arxiv.org/html/2609.20843#bib.bib60)\], and enriches each entity with images and numerical literals\. Richpedia\[[47](https://arxiv.org/html/2609.20843#bib.bib7)\]links encyclopedic entities to images crawled at scale, while VisualSem\[[1](https://arxiv.org/html/2609.20843#bib.bib8)\]grounds a multilingual entity inventory built on BabelNet\[[34](https://arxiv.org/html/2609.20843#bib.bib61)\]\. Later resources move beyond whole\-entity images toward finer and richer grounding\. AspectMMKG\[[57](https://arxiv.org/html/2609.20843#bib.bib9)\]attaches aspect\-specific images to capture different facets of an entity\. VISTA\[[19](https://arxiv.org/html/2609.20843#bib.bib10)\]grounds not only entities but also relational triples with visual evidence\. TIVA\-KG\[[49](https://arxiv.org/html/2609.20843#bib.bib11)\]extends triple\-level grounding across text, image, video, and audio\. MMEKG\[[31](https://arxiv.org/html/2609.20843#bib.bib40)\]further builds an event\-centric MMKG that connects textual and visual events\. Another line of work organizes multimodal knowledge for unified representation or downstream use\. Examples include UKnow\[[12](https://arxiv.org/html/2609.20843#bib.bib12)\]for unified entity\- and concept\-level cross\-modal knowledge, M2ConceptBase\[[56](https://arxiv.org/html/2609.20843#bib.bib13)\]for fine\-grained concept\-image alignment, and VAT\-KG\[[37](https://arxiv.org/html/2609.20843#bib.bib14)\]for concept\-centric multimodal knowledge\.

### II\-BRetrieval\-Augmented Generation

Retrieval\-augmented generation \(RAG\) grounds large language model \(LLM\) outputs in retrieved external evidence to reduce hallucination on knowledge\-intensive tasks\[[20](https://arxiv.org/html/2609.20843#bib.bib46)\]\. Later studies make retrieval more adaptive and agentic\. Self\-RAG\[[2](https://arxiv.org/html/2609.20843#bib.bib50)\]retrieves evidence on demand and critiques its own generations through reflection tokens, while Search\-o1\[[21](https://arxiv.org/html/2609.20843#bib.bib51)\]integrates an agentic search workflow into long\-form reasoning\. In the multimodal setting, OmniSearch\[[23](https://arxiv.org/html/2609.20843#bib.bib34)\]plans sub\-question chains with retrieval actions for dynamically changing visual questions, and MMSearch\-R1\[[51](https://arxiv.org/html/2609.20843#bib.bib35)\]trains a multimodal model with reinforcement learning to perform on\-demand, multi\-turn image and text search\. Another line of work incorporates graph structures into retrieval\. GraphRAG\[[10](https://arxiv.org/html/2609.20843#bib.bib47)\]builds an LLM\-derived entity graph over a text corpus and answers global questions through hierarchical community summaries\. LightRAG\[[13](https://arxiv.org/html/2609.20843#bib.bib48)\]couples graph\-structured text indexing with a dual\-level retrieval scheme, and G\-Retriever\[[15](https://arxiv.org/html/2609.20843#bib.bib49)\]retrieves question\-relevant subgraphs from textual graphs for answering\.

Limitation\.Traditional RAG retrieves free\-form textual evidence, making it difficult to organize and verify the relational dependencies required by multi\-hop questions\. Graph\-based RAG introduces graph structures into retrieval, but they are mainly constructed for conventional KGs and not designed for MMKGs, where reasoning requires grounding visual evidence\.

### II\-CKnowledge Graph Question Answering

Traditional KGQA\.Knowledge graph question answering \(KGQA\) aims to answer natural\-language questions by reasoning over a structured knowledge graph\. Before the rise of LLMs, KGQA methods mainly followed two paradigms: semantic parsing and information\-retrieval\-based reasoning\. Semantic parsing methods translate a question into an executable logical form or query graph\. For example, SEMPRE\[[6](https://arxiv.org/html/2609.20843#bib.bib41)\]learns logical\-form mapping from question–answer pairs over Freebase\[[7](https://arxiv.org/html/2609.20843#bib.bib62)\], while STAGG\[[54](https://arxiv.org/html/2609.20843#bib.bib15)\]formulates KGQA as staged query\-graph generation\. Later methods, such as QGG\[[18](https://arxiv.org/html/2609.20843#bib.bib16)\]and RNG\-KBQA\[[53](https://arxiv.org/html/2609.20843#bib.bib17)\], further improve query\-graph construction and candidate ranking\. Another line of work retrieves or propagates information over a question\-specific subgraph\. KV\-MemNN\[[33](https://arxiv.org/html/2609.20843#bib.bib42)\]reads facts from a key–value memory\. GraftNet\[[43](https://arxiv.org/html/2609.20843#bib.bib43)\]performs graph convolution over a fused KB\-and\-text subgraph, and PullNet\[[42](https://arxiv.org/html/2609.20843#bib.bib44)\]further retrieves such subgraphs iteratively for multi\-hop questions\. EmbedKGQA\[[39](https://arxiv.org/html/2609.20843#bib.bib18)\]matches questions with pretrained KG embeddings to reach distant answers\. NSM\[[14](https://arxiv.org/html/2609.20843#bib.bib19)\]and TransferNet\[[40](https://arxiv.org/html/2609.20843#bib.bib20)\]perform multi\-step relation reasoning with intermediate supervision\.

LLM\-based KGQA\.LLMs have reshaped KGQA by enabling models to reason over KG facts in natural\-language form\[[36](https://arxiv.org/html/2609.20843#bib.bib21)\]\. Early LLM\-based methods retrieve question\-relevant triples and provide them to the model as textual context\[[4](https://arxiv.org/html/2609.20843#bib.bib22)\]\. To better exploit graph structure, later studies treat the LLM as an agent that iteratively explores the KG\. Think\-on\-Graph\[[44](https://arxiv.org/html/2609.20843#bib.bib23)\]performs beam search over relations and entities from the starting entity\. Reasoning\-on\-Graph\[[29](https://arxiv.org/html/2609.20843#bib.bib24)\]first generates relation paths as plans and then retrieves supporting facts\. Chain\-of\-Knowledge\[[22](https://arxiv.org/html/2609.20843#bib.bib25)\]interleaves structured and unstructured retrieval during chain\-of\-thought reasoning\. Think\-on\-Graph 2\.0\[[30](https://arxiv.org/html/2609.20843#bib.bib26)\]further couples graph traversal with document retrieval\. More recent efforts emphasize adaptive planning and path\-level reasoning\. Plan\-on\-Graph\[[9](https://arxiv.org/html/2609.20843#bib.bib27)\]decomposes a question into sub\-objectives and self\-corrects erroneous exploration\. Paths\-over\-Graph\[[45](https://arxiv.org/html/2609.20843#bib.bib28)\]constructs multi\-hop reasoning paths and prunes them using graph structure\. A few recent studies introduce multimodal evidence into multimodal knowledge graph question answering \(MM\-KGQA\)\. mKG\-RAG\[[55](https://arxiv.org/html/2609.20843#bib.bib52)\]constructs MMKGs from documents and retrieves entity\- and relation\-level evidence for visual question answering\. ReasonVQA\[[46](https://arxiv.org/html/2609.20843#bib.bib37)\]and MMhops\-R1\[[58](https://arxiv.org/html/2609.20843#bib.bib38)\]further build large\-scale benchmarks whose questions require multi\-hop reasoning grounded in encyclopedic knowledge beyond the image itself\.

Limitation\.Despite this progress, existing MM\-KGQA methods typically use visual information only for starting entity grounding\. After that, reasoning still largely degenerates into text\-only graph search, making it difficult to exploit visual cues that become relevant at intermediate hops\.

## IIIPreliminaries

### III\-AMultimodal Knowledge Graph

We denote a multimodal knowledge graph \(MMKG\) as𝒢=\(ℰ,ℛ,ℱ,𝒳,𝒱\)\\mathcal\{G\}=\(\\mathcal\{E\},\\mathcal\{R\},\\mathcal\{F\},\\mathcal\{X\},\\mathcal\{V\}\), whereℰ\\mathcal\{E\},ℛ\\mathcal\{R\}, andℱ\\mathcal\{F\}are the sets of entities, relations, and facts, respectively\.𝒳\\mathcal\{X\}denotes the set of visual resources, and𝒱\\mathcal\{V\}maps each entity to its associated visual resources\. In this paper, we focus on the visual modality following existing studies on MMKGs\. Each fact inℱ\\mathcal\{F\}is represented as a triple\(esubj,r,eobj\)\(e\_\{\\mathrm\{subj\}\},r,e\_\{\\mathrm\{obj\}\}\), whereesubj,eobj∈ℰe\_\{\\mathrm\{subj\}\},e\_\{\\mathrm\{obj\}\}\\in\\mathcal\{E\}are the subject and object, respectively, andr∈ℛr\\in\\mathcal\{R\}\. A reasoning path of lengthddin an MMKG is denoted asp=\(e0,r1,e1,…,rd,ed\)p=\(e\_\{0\},r\_\{1\},e\_\{1\},\\ldots,r\_\{d\},e\_\{d\}\), where\(ei−1,ri,ei\)∈ℱ\(e\_\{i\-1\},r\_\{i\},e\_\{i\}\)\\in\\mathcal\{F\}for everyi=1,…,di=1,\\ldots,d\. Here,e0e\_\{0\}is the starting entity of the path, andede\_\{d\}is the endpoint entity\. For example,Government Museum, Chandigarh→architectLe Corbusier→place of birthLa Chaux\-de\-Fonds\\textit\{Government Museum, Chandigarh\}\\xrightarrow\{\\textit\{architect\}\}\\textit\{Le Corbusier\}\\xrightarrow\{\\textit\{place of birth\}\}\\textit\{La Chaux\-de\-Fonds\}is a length\-2 path, whereGovernment Museum, Chandigarhis the head entity andLa Chaux\-de\-Fondsis the endpoint entity\.

### III\-BProblem Formulation

In this paper, we study multimodal knowledge graph question answering \(MM\-KGQA\)\. Given a multimodal query\(I,Q,𝒢\)\(I,Q,\\mathcal\{G\}\), whereIIis an input image,QQis a natural\-language question related toII, and𝒢\\mathcal\{G\}is an MMKG, the goal is to return the correct answerAAby jointly grounding the visual content inIIand reasoning over𝒢\\mathcal\{G\}\. The answer should be supported by a reasoning path in𝒢\\mathcal\{G\}that starts from the image\-related starting entity and reaches evidence relevant toQQ\. For example, given an imageIIofGovernment Museum, Chandigarhand the questionQQ“Where was the architect of the building shown in the image born?”, the model is expected to identifyGovernment Museum, Chandigarhas the starting entity, follow the pathGovernment Museum, Chandigarh→architectLe Corbusier→place of birthLa Chaux\-de\-Fonds\\textit\{Government Museum, Chandigarh\}\\xrightarrow\{\\textit\{architect\}\}\\textit\{Le Corbusier\}\\xrightarrow\{\\textit\{place of birth\}\}\\textit\{La Chaux\-de\-Fonds\}, and return the answerA=La Chaux\-de\-FondsA=\\textit\{La Chaux\-de\-Fonds\}\.

![Refer to caption](https://arxiv.org/html/2609.20843v1/x1.png)Figure 1:Framework of the proposedVisPath\.VisPathanswers queries through four coordinated stages: \(a\) visual\- and graph\-aware entity identification that selects a structurally valid starting entity from image\-related candidates, \(b\) intent\-guided reasoning path discovery that uses hop\-specific visual intent to guide relation expansion and Top\-K selection, \(c\) reasoning\-chain pruning that evaluates candidate paths with respect to the question sketch and chain\-level coherence, and \(d\) visual\-grounded answering that checks answerability with the image and selected paths before producing the final answer\.

## IVMethod

Existing MM\-KGQA methods typically use multimodal information, such as visual cues from the imageII, only for starting entity grounding\. Once the starting entitye0e\_\{0\}is identified, graph reasoning is reduced to triple\-only path search over\(e0,Q,𝒢\)\(e\_\{0\},Q,\\mathcal\{G\}\)\. The subsequent process expands candidate relations, reaches intermediate entities, and selects reasoning paths solely within the knowledge graph, without further access toII\. Such a decoupled process makes later reasoning heavily dependent on the correctness of the initial grounding\. It also prevents the model from using visual cues that may become important at intermediate hops\.

To address this limitation, we propose a novel MM\-KGQA method namedVisPath, which answers a multimodal query\(I,Q,𝒢\)\(I,Q,\\mathcal\{G\}\)by keeping visual evidence involved throughout the reasoning process\. Specifically,VisPathfirst performs*visual\- and graph\-aware starting entity identification*to locate the image\-related starting entity, providing a reliable graph entry point for subsequent multi\-hop reasoning\. Starting from this entity, it then conducts*intent\-guided reasoning path discovery*, which progressively explores the MMKG by guiding each path expansion toward image\-relevant paths using visual intent\. The discovered paths are further processed by*reasoning\-chain pruning*, which evaluates candidates as complete reasoning chains rather than isolated edges and retains the most globally coherent paths\. Finally,VisPathperforms*visual\-grounded question answering*, where the selected paths are summarized with respect to the visual content, checked for answerability, and then used to either produce the final answer or continue the search when the current evidence is insufficient\. In this way,VisPathreduces the dependence on perfect initial grounding, keeps graph reasoning continuously aligned with visual evidence, and produces answers that are both path\-faithful and image\-grounded\. We illustrateVisPathin Figure[1](https://arxiv.org/html/2609.20843#S3.F1)and provide the whole reasoning process in Algorithm[1](https://arxiv.org/html/2609.20843#algorithm1)\.

### IV\-AVisual\- and Graph\-aware Starting Entity Identification

Existing MM\-KGQA methods usually identify the starting entity by matching the visual content of the image with entity names in the MMKG\. Relying only on such visual\-name matching can be unreliable, since the object recognized from the image may be described with a different name, alias, or level of granularity from the entity stored in the graph\. Moreover, even when an entity has the highest visual\-name similarity, it may not provide relations that are useful for answeringQQ\. Selecting such an entity as the starting point can mislead the subsequent graph reasoning process\. To address this issue,VisPathidentifies the starting entityeSe\_\{S\}for the query\(I,Q,𝒢\)\(I,Q,\\mathcal\{G\}\)by jointly considering visual evidence and graph\-structural cues\. In this way, the selected entity is not only visually consistent with the image, but also structurally suitable as the entry point for multi\-hop reasoning\.

Visual\-aware candidate ranking\.We first prompt a text language modelLLM​\(⋅\)\\texttt\{LLM\}\(\\cdot\)to extract the expression inQQthat refers to the visual object inII, together with its bare head noun, denoted asHEAD\\mathrm\{HEAD\}\. We use onlyHEAD\\mathrm\{HEAD\}for visual recognition, so that the vision model focuses on the object depicted in the image rather than being biased by other reasoning clues inQQ\. Given the imageIIand the extractedHEAD\\mathrm\{HEAD\}, we use a vision–language modelVLM​\(⋅\)\\texttt\{VLM\}\(\\cdot\)to recognize the depicted object and produce a textual object name:

o=VLM​\(I,HEAD\),o=\\texttt\{VLM\}\\left\(I,\\mathrm\{HEAD\}\\right\),\(1\)whereoodenotes the object name predicted from the image\. Sinceoois generated by visual recognition, it may not exactly match the entity names stored in the MMKG\. We therefore encodeoowith a sentence encoderSenBERT​\(⋅\)\\texttt\{SenBERT\}\(\\cdot\)\[[38](https://arxiv.org/html/2609.20843#bib.bib53)\]and retrieve similar entity names from𝒢\\mathcal\{G\}using a pre\-built entity\-name embedding index\. We then provide the retrieved entity names, the predicted object nameoo, and the questionQQtoLLM​\(⋅\)\\texttt\{LLM\}\(\\cdot\)\. The language model filters and ranks the retrieved entities by considering both their semantic similarity toooand their compatibility with the image\-referring expression inQQ\. The ranked entities returned byLLM​\(⋅\)\\texttt\{LLM\}\(\\cdot\)form the initial candidate set, denoted as\{eQ,1,…,eQ,K\}\\\{e\_\{Q,1\},\\dots,e\_\{Q,K\}\\\}, where earlier entities are considered more likely to be the starting entity for answeringQQ\. Rather than directly accepting the top\-ranked entity, we keep the entire ranked candidate set for the subsequent validation step\. This preserves alternative visually plausible entities and allows revising the initial ranking when the top candidate is not suitable for downstream reasoning\.

Graph\-aware validation\.Visual similarity can identify entities that are visually or lexically close to the object in the image, but it does not indicate whether these entities are suitable starting points for answeringQQ\. This distinction is important in MMKG reasoning\. A visually plausible candidate may be disconnected from the required answer path, or it may only expose outgoing relations that are irrelevant to the question\. Therefore, we further validate each candidate by checking whether its outgoing relations are compatible with the reasoning need expressed inQQ\. For each candidateeQ,ke\_\{Q,k\}, we compute a graph\-aware alignment score:

sQ,k=maxr∈ℛeQ,k⁡cos⁡\(SenBERT​\(r\),SenBERT​\(Q\)\),s\_\{Q,k\}=\\max\_\{r\\in\\mathcal\{R\}\_\{e\_\{Q,k\}\}\}\\cos\\big\(\\texttt\{SenBERT\}\(r\),\\texttt\{SenBERT\}\(Q\)\\big\),\(2\)whereℛeQ,k\\mathcal\{R\}\_\{e\_\{Q,k\}\}denotes the set of outgoing relations ofeQ,ke\_\{Q,k\}\. This score measures whether the candidate provides at least one outgoing relation that can plausibly initiate the answer path\. We use this score to correct the visual linking result\. DenoteeQ,1e\_\{Q,1\}as the top\-ranked candidate from visual\-aware candidate linking\. LeteQ,k⋆e\_\{Q,k^\{\\star\}\}be the candidate with the highest graph\-aware alignment score, wherek⋆=arg⁡max2≤k≤K⁡sQ,kk^\{\\star\}=\\arg\\max\_\{2\\leq k\\leq K\}s\_\{Q,k\}\. We replaceeQ,1e\_\{Q,1\}witheQ,k⋆e\_\{Q,k^\{\\star\}\}only when the latter is clearly more suitable for graph reasoning:

sQ,k⋆−sQ,1\>δ,δ\>0\.s\_\{Q,k^\{\\star\}\}\-s\_\{Q,1\}\>\\delta,\\quad\\delta\>0\.\(3\)Otherwise, we keepeQ,1e\_\{Q,1\}as the starting entity\. After this correction, we obtain an updated ranked candidate list\{e~Q,1,…,e~Q,K\}\\\{\\tilde\{e\}\_\{Q,1\},\\dots,\\tilde\{e\}\_\{Q,K\}\\\}and take its first element as the starting entity, i\.e\.,eS=e~Q,1e\_\{S\}=\\tilde\{e\}\_\{Q,1\}\. This conservative correction avoids changing reliable visual links unnecessarily, while still allowing the model to recover when the visually top\-ranked entity is not structurally useful for subsequent reasoning\.

### IV\-BIntent\-Guided Reasoning Path Discovery

After identifying the starting entityeSe\_\{S\}, existing KGQA methods usually rely only on the questionQQto determine which relations should be followed in subsequent hops\. As a result, path search is conducted purely over the graph, and the image is no longer involved in the reasoning process\. This question\-only strategy is insufficient for MMKG reasoning, because the question may not explicitly specify every relation needed in the answer path\. In such cases, some intermediate reasoning clues must still be inferred from the image\. For example, consider the question “Which league does the team shown on the player’s jersey belong to?” Even if the player is correctly grounded as the starting entity, the graph may contain many outgoing relations, such asplace of birth,position,national team, and multipleclubrelations\. The question does not explicitly indicate which team should be followed\. Instead, the relevant team must be identified from the jersey logo or other visual details in the image\. Therefore, visual evidence remains necessary after grounding: it guides the first hop toward the visually indicatedclubrelation, and the next hop can then follow theleaguerelation from that club\.

To this end, we propose an intent\-guided reasoning path discovery module that keeps the image involved after the starting entity has been identified\. Starting fromeSe\_\{S\}, the module progressively discovers reasoning paths over the MMKG\. At each hop, it infers the reasoning intent for the next step by jointly considering the imageII, the questionQQ, and the current partial paths\. This intent is then used to select the most relevant outgoing edges from the current entities\. The selected edges extend the current paths and produce a new set of candidate reasoning paths for the next hop\. In this way,VisPathcan use visual evidence to decide which relation should be followed at each reasoning state, avoid blindly expanding irrelevant graph branches, and discover reasoning paths that are both graph\-valid and image\-relevant\.

Formally, let𝒫\(i−1\)\\mathcal\{P\}^\{\(i\-1\)\}denote the set of partial reasoning paths retained after hopi−1i\-1, and letℰ\(i−1\)\\mathcal\{E\}^\{\(i\-1\)\}denote their endpoint entities\. We initialize the search with𝒫\(0\)=\{\(eS\)\}\\mathcal\{P\}^\{\(0\)\}=\\\{\(e\_\{S\}\)\\\}andℰ\(0\)=\{eS\}\\mathcal\{E\}^\{\(0\)\}=\\\{e\_\{S\}\\\}\. At each hopii, we first useVLM​\(⋅\)\\texttt\{VLM\}\(\\cdot\)to generate a hop\-specific reasoning intent:

qQ\(i\)=VLM​\(I,Q,𝒫\(i−1\)\),q\_\{Q\}^\{\(i\)\}=\\texttt\{VLM\}\\left\(I,Q,\\mathcal\{P\}^\{\(i\-1\)\}\\right\),\(4\)whereqQ\(i\)q\_\{Q\}^\{\(i\)\}is a short textual description of what the next reasoning step should seek\. Since this intent is conditioned on the image, the question, and the current partial paths, it can adapt to different reasoning stages\. In the jersey example, the first hop may focus on the club indicated by the player’s jersey, while the next hop focuses on the league associated with that club\.

Given the hop\-specific intentqQ\(i\)q\_\{Q\}^\{\(i\)\}, we compute an intent\-matching score for each outgoing edge from the current endpoint entities\. For an entitye∈ℰ\(i−1\)e\\in\\mathcal\{E\}^\{\(i\-1\)\}and one of its outgoing edges\(e,r,et\)\(e,r,e\_\{t\}\), we define:

cr,qi=cos⁡\(SenBERT​\(r∥et\),SenBERT​\(qQ\(i\)\)\),c\_\{r,q\_\{i\}\}=\\cos\\left\(\\texttt\{SenBERT\}\\left\(r\\mathbin\{\\\|\}e\_\{t\}\\right\),\\texttt\{SenBERT\}\\left\(q\_\{Q\}^\{\(i\)\}\\right\)\\right\),\(5\)where∥\\mathbin\{\\\|\}denotes textual concatenation, andete\_\{t\}is the endpoint entity reached by relationrr\. We include the endpoint entity together with the relation because the relation name alone may be too coarse to distinguish visually relevant edges from irrelevant ones\. For each entitye∈ℰ\(i−1\)e\\in\\mathcal\{E\}^\{\(i\-1\)\}, we keep only the top\-M1M\_\{1\}outgoing edges with the highest intent\-matching scores\. The collected edge set is denoted asℛ\(i−1\)\\mathcal\{R\}^\{\(i\-1\)\}\. This operation limits the branching factor at each hop and prevents high\-degree entities from introducing excessive irrelevant candidates\.

Finally, we construct new candidate paths by extending each retained partial path with the selected outgoing edges from its endpoint entity:

𝒫cand\(i\)=⋃p∈𝒫\(i−1\)\{p⊕\(end⁡\(p\),r,et\)∣r∈ℛ\(i−1\)\}\.\{\\mathcal\{P\}\}^\{\(i\)\}\_\{\\mathrm\{cand\}\}=\\bigcup\_\{p\\in\\mathcal\{P\}^\{\(i\-1\)\}\}\\left\\\{p\\oplus\(\\operatorname\{end\}\(p\),r,e\_\{t\}\)\\mid r\\in\\mathcal\{R\}^\{\(i\-1\)\}\\right\\\}\.\(6\)Here,end⁡\(p\)\\operatorname\{end\}\(p\)denotes the endpoint entity of pathpp, andp⊕\(end⁡\(p\),r,et\)p\\oplus\(\\operatorname\{end\}\(p\),r,e\_\{t\}\)appends relationrrand endpoint entityete\_\{t\}topp\. Since at mostM1M\_\{1\}outgoing edges are retained for each endpoint entity, the number of generated candidate paths is bounded by

\|𝒫cand\(i\)\|≤M1​\|𝒫\(i−1\)\|\.\\left\|\\mathcal\{P\}\_\{\\mathrm\{cand\}\}^\{\(i\)\}\\right\|\\leq M\_\{1\}\\left\|\\mathcal\{P\}^\{\(i\-1\)\}\\right\|\.\(7\)This bounded construction allowsVisPathto explore visually relevant paths hop by hop while controlling the growth of the candidate space\.

### IV\-CReasoning\-Chain Pruning

The intent\-guided discovery module generates a candidate path set𝒫cand\(i\)\\mathcal\{P\}\_\{\\mathrm\{cand\}\}^\{\(i\)\}at hopii\. Although each expanded edge is selected according to the hop\-specific intent, such local matching does not guarantee that the whole path is suitable for answering the question\. A path may contain edges that are individually plausible, but their composition may deviate from the expected reasoning direction\. In other words, locally optimal expansions at different hops may not form a globally coherent reasoning chain\. Therefore, we further prune and select candidate paths at the reasoning\-chain level, so that the retained paths are consistent with the questionQQ, the visual evidence fromII, and the hop\-specific intentqQ\(i\)q\_\{Q\}^\{\(i\)\}as a whole\.

To provide a global reference for chain\-level selection, we first useLLM​\(⋅\)\\texttt\{LLM\}\(\\cdot\)to analyze the questionQQinto a reasoning sketch𝒮Q\\mathcal\{S\}\_\{Q\}and a predicted reasoning depthD^\\widehat\{D\}\. We useD^\\widehat\{D\}only when it is smaller than the pre\-defined maximum search depthDmaxD\_\{\\max\}; otherwise, the search is bounded byDmaxD\_\{\\max\}\. The sketch describes the coarse relation\-level reasoning direction, whileD^\\widehat\{D\}estimates the number of hops required to reach the answer\. For example, for the question “Where was the architect of the building shown in the image born?”, the question can be summarized into a two\-hop sketch, ‘architect→\\rightarrowplace of birth”, withD^=2\\widehat\{D\}=2\. This sketch does not determine the exact entities in the path, but it provides a global semantic guide for judging whether a candidate path follows the expected reasoning chain\.

To evaluate whether a candidate path is globally coherent, we compare the whole path with the multimodal reasoning objective\. Specifically, we first convert each candidate path into a textual relation chain, and then measure its semantic consistency with the question, the reasoning sketch, and the hop\-specific visual intent\. For each pathp∈𝒫cand\(i\)p\\in\\mathcal\{P\}\_\{\\mathrm\{cand\}\}^\{\(i\)\}, the chain\-level score is defined as

σp,i=cos⁡\(SenBERT​\(Verbalize​\(p\)\),SenBERT​\(Q∥𝒮Q∥qQ\(i\)\)\),\\footnotesize\\sigma\_\{p,i\}=\\cos\\left\(\\texttt\{SenBERT\}\\left\(\\texttt\{Verbalize\}\(p\)\\right\),\\texttt\{SenBERT\}\\left\(Q\\mathbin\{\\\|\}\\mathcal\{S\}\_\{Q\}\\mathbin\{\\\|\}q\_\{Q\}^\{\(i\)\}\\right\)\\right\),\(8\)whereVerbalize​\(p\)\\texttt\{Verbalize\}\(p\)transforms the structured pathppinto a natural\-language relation chain\. Here,QQspecifies the answer objective,𝒮Q\\mathcal\{S\}\_\{Q\}provides the global reasoning sketch, andqQ\(i\)q\_\{Q\}^\{\(i\)\}provides the image\-derived intent at hopii\. Thus,σp,i\\sigma\_\{p,i\}measures whether the complete path is compatible with the multimodal reasoning goal, rather than only checking whether its latest edge matches the current hop intent\. We then select the top\-M2M\_\{2\}paths according toσp,i\\sigma\_\{p,i\}and denote the selected path set as𝒫chain\(i\)\\mathcal\{P\}\_\{\\mathrm\{chain\}\}^\{\(i\)\}\. This step removes paths whose complete relation chains are weakly aligned with the expected reasoning direction, even if some of their individual edges were selected by the hop\-level intent\.

After semantic scoring, we further employ a language model to examine the remaining paths as complete reasoning chains and select those that are most likely to provide valid evidence for answering the question\. If the language model returns an empty set, we fall back to the highest\-scoring paths in𝒫chain\(i\)\\mathcal\{P\}\_\{\\mathrm\{chain\}\}^\{\(i\)\}\. We then retain at mostM3M\_\{3\}paths according to their chain\-level scores and denote the retained path set as𝒫\(i\)\\mathcal\{P\}^\{\(i\)\}\. By moving from locally intent\-matched edge expansion to globally coherent chain selection,VisPathreduces the risk of accumulating plausible but inconsistent reasoning steps, while keeping the search focused on paths that are both visually grounded and semantically aligned with the question\.

### IV\-DVisual\-Grounded Question Answering

After reasoning\-chain pruning and selection,VisPathobtains a retained path set𝒫\(i\)\\mathcal\{P\}^\{\(i\)\}at hopii\. These paths represent the most promising graph evidence, but their relations and entities still need to be interpreted together with the image before we can determine whether they are sufficient to answerQQ\. Therefore, this module verifies the retained paths against the visual content and decides whether the current multimodal evidence can support answer generation or whether the search should continue to the next hop\.

We first summarize the retained paths under the visual context\. Given the imageII, the questionQQ, and the retained path set𝒫\(i\)\\mathcal\{P\}^\{\(i\)\}, we use a vision–language modelVLM​\(⋅\)\\texttt\{VLM\}\(\\cdot\)to generate an image\-grounded evidence statement:

zi=VLM​\(I,Q,𝒫\(i\)\),z\_\{i\}=\\texttt\{VLM\}\\left\(I,Q,\\mathcal\{P\}^\{\(i\)\}\\right\),\(9\)whereziz\_\{i\}summarizes how the selected graph paths relate to the visual content\. This step is necessary because the retained paths describe symbolic graph relations and entities, while the final answer should remain consistent with the depicted object and the visual evidence inII\.

We then queryVLM​\(⋅\)\\texttt\{VLM\}\(\\cdot\)to assess whether the image\-grounded evidence statementziz\_\{i\}is sufficient to answerQQ\. The model returns an answerability verdict and, if the verdict is positive, a candidate answerAiA\_\{i\}\. We formulate this step as

\(ηi,Ai\)=VLM​\(I,Q,zi\),\(\\eta\_\{i\},A\_\{i\}\)=\\texttt\{VLM\}\\left\(I,Q,z\_\{i\}\\right\),\(10\)whereηi∈\{0,1\}\\eta\_\{i\}\\in\\\{0,1\\\}indicates whether the current multimodal evidence is sufficient to answerQQ, andAiA\_\{i\}denotes the candidate answer generated fromziz\_\{i\}whenηi=1\\eta\_\{i\}=1\.VisPathstops normal path expansion only when the evidence is judged answerable and the predicted reasoning depthD^\\widehat\{D\}has been reached\. Accordingly, we define the stopping hop as

istop=min⁡\{i∣ηi=1∧i≥D^\}\.i\_\{\\mathrm\{stop\}\}=\\min\\left\\\{i\\mid\\eta\_\{i\}=1\\ \\wedge\\ i\\geq\\widehat\{D\}\\right\\\}\.\(11\)If such a hop exists, the final answer is generated from the corresponding image\-grounded evidence\. If the evidence becomes answerable before reachingD^\\widehat\{D\}, we record the candidate answer as a fallback but continue the search\. This prevents a shallow path from prematurely terminating the reasoning process before the expected number of hops is completed\.

When neither a valid stopping hop nor a fallback hop exists,VisPathperforms terminal answer selection\. This step ranks the endpoint entities appearing in the retained paths and selects the most plausible answer candidate\. When reference images are available, the top candidates are further verified againstIIusingVLM​\(⋅\)\\texttt\{VLM\}\(\\cdot\)\. If no reliable graph candidate can be obtained, the answer is inferred directly from the image\. In this way,VisPathkeeps visual evidence involved until the final answer is produced, while avoiding unnecessary hops once sufficient multimodal evidence has been obtained\.

Input :imageII, questionQQ, MMKG𝒢\\mathcal\{G\}, retention sizesM1M\_\{1\},M2M\_\{2\},M3M\_\{3\}, depth budgetDmaxD\_\{\\max\}

Output :answer

AA
recognize object

oofrom

IIand rank entity candidates

\{eQ,1,…,eQ,K\}\\\{e\_\{Q,1\},\\dots,e\_\{Q,K\}\\\}
⊳\\trianglerightEq\. \([1](https://arxiv.org/html/2609.20843#S4.E1)\): visual recognition

select the starting entity

eSe\_\{S\}by graph\-aware validation

⊳\\trianglerightEqs\. \([2](https://arxiv.org/html/2609.20843#S4.E2)\)\-\-\([3](https://arxiv.org/html/2609.20843#S4.E3)\)

1prompt the LLM for the reasoning sketch

𝒮Q\\mathcal\{S\}\_\{Q\}and predicted depth

D^\\widehat\{D\};

D^←min⁡\(D^,Dmax\)\\widehat\{D\}\\leftarrow\\min\(\\widehat\{D\},\\,D\_\{\\max\}\)
2

𝒫\(0\)←\{\(eS\)\}\\mathcal\{P\}^\{\(0\)\}\\leftarrow\\\{\(e\_\{S\}\)\\\};

z⋆←⊥z^\{\\star\}\\leftarrow\\bot
3for*i=1i=1toDmaxD\_\{\\max\}*do

qQ\(i\)←VLM​\(I,Q,𝒫\(i−1\)\)q\_\{Q\}^\{\(i\)\}\\leftarrow\\texttt\{VLM\}\(I,Q,\\mathcal\{P\}^\{\(i\-1\)\}\)
⊳\\trianglerightEq\. \([4](https://arxiv.org/html/2609.20843#S4.E4)\): hop intent

keep the top\-

M1M\_\{1\}intent\-matched edges per frontier entity

⊳\\trianglerightEq\. \([5](https://arxiv.org/html/2609.20843#S4.E5)\): edge scoring

extend

𝒫\(i−1\)\\mathcal\{P\}^\{\(i\-1\)\}with retained edges into

𝒫cand\(i\)\\mathcal\{P\}\_\{\\mathrm\{cand\}\}^\{\(i\)\}
⊳\\trianglerightEq\. \([6](https://arxiv.org/html/2609.20843#S4.E6)\): path extension

retain the top\-

M2M\_\{2\}paths as

𝒫chain\(i\)\\mathcal\{P\}\_\{\\mathrm\{chain\}\}^\{\(i\)\}by chain scores

σp,i\\sigma\_\{p,i\}
⊳\\trianglerightEq\. \([8](https://arxiv.org/html/2609.20843#S4.E8)\): chain reranking

4examine

𝒫chain\(i\)\\mathcal\{P\}\_\{\\mathrm\{chain\}\}^\{\(i\)\}with the LLM and retain at most

M3M\_\{3\}paths as

𝒫\(i\)\\mathcal\{P\}^\{\(i\)\}
zi←VLM​\(I,Q,𝒫\(i\)\)z\_\{i\}\\leftarrow\\texttt\{VLM\}\(I,Q,\\mathcal\{P\}^\{\(i\)\}\)
⊳\\trianglerightEq\. \([9](https://arxiv.org/html/2609.20843#S4.E9)\): evidence statement

5if*ziz\_\{i\}supports answeringQQ*then

z⋆←ziz^\{\\star\}\\leftarrow z\_\{i\}
⊳\\trianglerightEq\. \([10](https://arxiv.org/html/2609.20843#S4.E10)\): answerability

6if*i≥D^i\\geq\\widehat\{D\}*then

break

⊳\\trianglerightEq\. \([11](https://arxiv.org/html/2609.20843#S4.E11)\): stop rule

7

8

9

10if*z⋆≠⊥z^\{\\star\}\\neq\\bot*then

11generate the final answer

AAfrom

\(I,Q,z⋆\)\(I,Q,z^\{\\star\}\)
12

13else

14select

AAfrom visited path entities with visual verification

15

return*AA*

Algorithm 1VisPathReasoning Process

## VExperiments

### V\-AVisPath\-Bench construction

Existing MMKGQA benchmarks are dominated by two\-hop questions and contain few examples that require deeper reasoning chains\. To evaluate multimodal multi\-hop reasoning more systematically, we constructVisPath\-Bench from the Wikidata5M MMKG in three stages\. \(i\)*Sampling*: we sample topic entities that are associated with an image through the WikidataP18property and have gold reasoning paths of length two to four\. \(ii\)*Question generation*: for each sampled path, we generate a natural\-language question whose answer is the path endpoint\. The topic entity is referred to only through a visual referring expression, so that the starting entity must be grounded from the image rather than read directly from the question text\. \(iii\)*Single\-answer filtering*: we retain only questions whose gold path leads to a single unambiguous answer entity, and discard paths whose relations allow multiple valid endpoints\. The resulting benchmark contains 2,000 questions, including 500 two\-hop, 1,200 three\-hop, and 300 four\-hop questions, covering 52 distinct Wikidata relations\. We provide the prompt templates in Figure[2](https://arxiv.org/html/2609.20843#S5.F2)and distribution of the question length in Figure[3](https://arxiv.org/html/2609.20843#S5.F3)\.

TABLE I:Statistics of the four multi\-hop VQA benchmarks\. Avg\.\|Q\|\|Q\|and Avg\.\|A\|\|A\|denote the average question and answer lengths in words, respectively\. \#Relations reports the number of distinct relations appearing in the gold reasoning paths\.Dataset\#QuestionsHop DistributionAvg\.\|Q\|\|Q\|\#RelationsAvg\.\|A\|\|A\|Image Source2\-Hop3\-Hop4\-HopReasonVQA1,9513671,584–13\.8491\.8Visual GenomeMMhops1,90787482920415\.6–1\.7WikipediaE\-VQA1,4001,400––16\.6–1\.7iNat/GLDv2VisPath\-Bench2,0005001,20030018\.4522\.3Wikidata \(P18\)
### V\-BExperimental Setup

![Refer to caption](https://arxiv.org/html/2609.20843v1/x2.png)Figure 2:Prompt templates used in the construction ofVisPath\-Bench\.Benchmarks\.We evaluateVisPathon four multi\-hop visual question answering benchmarks, as summarized in Table[I](https://arxiv.org/html/2609.20843#S5.T1)\. These include ourVisPath\-Bench, described below, and three benchmarks adapted from prior work\. ReasonVQA\[[46](https://arxiv.org/html/2609.20843#bib.bib37)\]and MMhops\[[58](https://arxiv.org/html/2609.20843#bib.bib38)\]are entity\-centric multi\-hop VQA datasets\. To make them compatible with our path\-based evaluation protocol, we re\-ground each question onto the knowledge graph by resolving its topic entity and gold answer to KG nodes\. E\-VQA\[[32](https://arxiv.org/html/2609.20843#bib.bib54)\]is a two\-hop encyclopedic VQA dataset built over fine\-grained natural and landmark images\. Together, the four benchmarks cover two to four reasoning hops and involve diverse image sources, including Visual Genome, Wikipedia, iNaturalist, Google Landmarks, and the Wikidata imagery used in our own benchmark\.

Baselines\.We compareVisPathwith four families of baselines: \(i\) classic KGQA methods, including EmbedKGQA\[[39](https://arxiv.org/html/2609.20843#bib.bib18)\]; \(ii\) vision–language models evaluated zero\-shot on\(I,Q\)\(I,Q\), including open\-source models Qwen2\.5\-VL\-32B/72B\[[5](https://arxiv.org/html/2609.20843#bib.bib31)\]and GLM\-4\.6V\[[11](https://arxiv.org/html/2609.20843#bib.bib65)\], as well as closed\-source models GPT\-4o\[[17](https://arxiv.org/html/2609.20843#bib.bib32)\]and GPT\-5\.4; \(iii\) multi\-hop KG reasoning methods, including MindMap\[[50](https://arxiv.org/html/2609.20843#bib.bib29)\], ToG\[[44](https://arxiv.org/html/2609.20843#bib.bib23)\], and PoG\[[45](https://arxiv.org/html/2609.20843#bib.bib28)\]; and \(iv\) retrieval\-augmented and agentic methods, including OmniSearch\[[23](https://arxiv.org/html/2609.20843#bib.bib34)\]and MMSearch\-R1\[[51](https://arxiv.org/html/2609.20843#bib.bib35)\]\. The text\-only KGQA methods in family \(iii\) cannot access the image and therefore require a starting entity\. For a controlled comparison, we provide these text\-only KGQA baselines with the gold starting entity, since they cannot access the image for entity grounding\. In contrast,VisPathmust infer the starting entity from the multimodal input before performing graph reasoning\. Thus, the observed improvements are achieved under a stricter setting forVisPath, further demonstrating the effectiveness of incorporating multimodal evidence into hop\-by\-hop path reasoning\.

![Refer to caption](https://arxiv.org/html/2609.20843v1/x3.png)Figure 3:Distribution of question lengths on the four benchmarks\.Metrics\.Since the answers are free\-form text and may involve aliases, hypernyms, or multilingual variants, exact match can underestimate model performance and may unfairly penalize verbose outputs from VLM baselines\. We therefore report open\-ended semantic accuracy, denoted as SBERT@0\.3, computed with Sentence\-BERT embeddings\[[38](https://arxiv.org/html/2609.20843#bib.bib53)\]\. A prediction is considered correct if the cosine similarity between its sentence embedding and that of the gold answer is at least0\.30\.3\. We report both per\-hop and overall accuracy, with all results micro\-averaged over questions\. The similarity threshold follows the open\-ended evaluation protocol used in prior multimodal QA studies\[[32](https://arxiv.org/html/2609.20843#bib.bib54),[46](https://arxiv.org/html/2609.20843#bib.bib37)\]\.

Implementation details\.VisPathuses GPT\-4o as the text language model, Qwen2\.5\-VL\-72B as the vision–language model, and all\-MiniLM\-L6\-v2\[[38](https://arxiv.org/html/2609.20843#bib.bib53)\]as the sentence encoder\. We adopt Wikidata5M\[[48](https://arxiv.org/html/2609.20843#bib.bib63)\]as the underlying knowledge graph and build an approximate nearest\-neighbor index over its entity names for efficient entity retrieval\. The main hyperparameters are the per\-hop edge retention sizeM1=30M\_\{1\}=30for path expansion, the path reranking sizeM2=30M\_\{2\}=30for chain\-level scoring, the chain retention sizeM3=3M\_\{3\}=3for beam\-style selection, the override thresholdδ=0\.05\\delta\{=\}0\.05in Eq\. \([3](https://arxiv.org/html/2609.20843#S4.E3)\), which grows by0\.010\.01per rank for lower\-ranked candidates to keep overrides conservative, and the maximum search depthDmax=3D\_\{\\max\}=3\.

### V\-CMain Results

TABLE II:Main results on four multi\-hop VQA benchmarks\. Metric: SBERT@0\.3\.Bold: best in column;underline: second\-best\.MethodVisPath\-BenchReasonVQAMMhopsE\-VQA2\-hop3\-hop4\-hopAvg2\-hop3\-hopAvg2\-hop3\-hop4\-hopAvg2\-hop*Classic KGQA*EmbedKGQA\(ACL’20\)16\.211\.14\.711\.428\.615\.918\.39\.86\.97\.48\.35\.9*Open\-source Vision–Language Models*Qwen2\.5\-VL\-32B\(2025\)58\.649\.546\.051\.349\.251\.451\.070\.667\.870\.269\.361\.7Qwen2\.5\-VL\-72B\(2025\)62\.052\.649\.754\.546\.657\.155\.175\.472\.475\.074\.065\.0GLM\-4\.6V\(2025\)51\.850\.048\.350\.252\.060\.558\.980\.379\.677\.979\.766\.4*Closed\-source Vision–Language Models*GPT\-4o\(2024\)57\.650\.147\.051\.563\.152\.854\.979\.174\.377\.576\.862\.8GPT\-5\.4\(2026\)65\.456\.852\.758\.369\.660\.762\.481\.778\.976\.079\.966\.0*Multi\-Hop Reasoning*MindMap\(ACL’24\)68\.450\.649\.754\.951\.247\.648\.352\.452\.446\.151\.641\.0ToG \(GPT\-4o\)\(ICLR’24\)34\.016\.713\.720\.632\.422\.124\.027\.622\.717\.224\.311\.4PoG \(GPT\-4o\)\(WWW’25\)55\.236\.725\.339\.631\.632\.132\.034\.126\.016\.729\.417\.1*Retrieval\-Augmented / Agent*OmniSearch\(ICLR’25\)38\.036\.637\.037\.037\.943\.142\.150\.738\.628\.443\.137\.4MMSearch\-R1\-7B\(ACL’26\)44\.238\.135\.039\.245\.551\.150\.060\.948\.043\.153\.332\.3VisPath\(GPT\-4o\)74\.062\.457\.064\.581\.564\.868\.086\.583\.179\.884\.370\.4We evaluateVisPathagainst a comprehensive set of baselines on four benchmarks under the SBERT@0\.3 metric, as shown in Table[II](https://arxiv.org/html/2609.20843#S5.T2)\. The results first show thatVisPath\-Bench is more challenging than existing benchmarks for current baselines\. For example, strong zero\-shot VLMs suffer clear performance drops onVisPath\-Bench compared with other datasets: Qwen2\.5\-VL\-72B drops by26\.4%26\.4\\%relative to its performance on MMhops, and GLM\-4\.6V drops by37\.0%37\.0\\%\. This indicates that recognizing the depicted entity or relying on dense encyclopedic knowledge is insufficient for our benchmark, where the answer must be derived through image\-grounded multi\-hop paths\.

The results also show that reasoning depth substantially affects all methods\. As the number of hops increases, text\-only path\-reasoning methods degrade sharply because they decode relation sequences mainly from the question and cannot revise them with multimodal evidence\. For instance, PoG drops by54\.2%54\.2\\%from two\-hop to four\-hop questions onVisPath\-Bench, while ToG drops by59\.7%59\.7\\%\. Even strong VLM baselines show notable degradation as hop depth increases, suggesting that implicit reasoning from the input alone is not enough for deep MM\-KGQA\.

In contrast,VisPathdelivers more uniform performance across benchmarks and reasoning depths\. OnVisPath\-Bench,VisPathachieves a10\.6%10\.6\\%relative improvement in average accuracy over the strongest baseline\. Its advantage holds consistently across different hop levels, with relative improvements of13\.1%13\.1\\%,9\.9%9\.9\\%, and8\.2%8\.2\\%at two, three, and four hops, respectively\. These results demonstrate that keeping multimodal evidence involved in hop\-by\-hop path discovery and reasoning\-chain selection enablesVisPathto remain robust as questions become deeper and more compositional\.

### V\-DBackbone Study

Tables[III](https://arxiv.org/html/2609.20843#S5.T3)and[IV](https://arxiv.org/html/2609.20843#S5.T4)evaluate the robustness ofVisPathunder different text and vision backbones\. For the text LLM, GPT\-4o and GPT\-4o\-mini achieve comparable performance onVisPath\-Bench, suggesting that our path reasoning framework does not rely solely on the strongest proprietary text model\. However, the gap becomes clearer on ReasonVQA, where GPT\-4o provides stronger cross\-benchmark generalization\. DeepSeek\-V3 and Qwen3\-32B remain competitive on two\-hop questions, but their performance drops more substantially at deeper hops, indicating that long\-range path planning and reasoning\-chain selection require stronger instruction following and semantic judgment\. For the vision backbone, Qwen2\.5\-VL\-72B achieves the best overall performance across both benchmarks\. Qwen3\-VL\-32B remains competitive on two\-hop questions but degrades more on deeper reasoning, while Qwen3\-VL\-8B shows a larger drop, especially onVisPath\-Bench\. These results show that both components are important: the vision model affects multimodal grounding and hop\-specific intent estimation, while the text LLM is crucial for maintaining coherent multi\-hop reasoning and selecting globally valid evidence paths\.

TABLE III:Performance ofVisPathwith different text LLM backbones\. The vision model is fixed to Qwen2\.5\-VL\-72B\.TABLE IV:Performance ofVisPathwith different vision model backbones\. The text LLM is fixed to GPT\-4o\.
### V\-ETopic Grounding Analysis

TABLE V:Starting\-entity grounding accuracy on ReasonVQA\. Strict accuracy requires the predicted entity QID to match the gold QID, while soft accuracy additionally accepts predictions whose entity label has SBERT similarity≥0\.7\\geq 0\.7to the gold label\.Table[V](https://arxiv.org/html/2609.20843#S5.T5)evaluates starting\-entity grounding on ReasonVQA\. Here,*Top\-1 Selection*denotes a direct grounding strategy that simply takes the highest\-ranked entity predicted by the vision\-language model, without applying graph\-aware self\-correcting validation\. This setting measures how reliable visual\-name matching alone is for recovering the correct starting entity\. As shown in the table, Top\-1 Selection with Qwen2\.5\-VL\-72B achieves only24\.124\.1strict accuracy and25\.125\.1soft accuracy, indicating that grounding the starting entity from the multimodal input is a non\-trivial bottleneck\. When applying the self\-correcting validation inVisPathunder the same Qwen2\.5\-VL\-72B backbone, strict accuracy increases to42\.742\.7and soft accuracy increases to48\.348\.3, corresponding to an18\.618\.6\-point strict gain and a23\.223\.2\-point soft gain\. This confirms that visual similarity alone is insufficient, while checking whether candidate entities expose question\-relevant outgoing relations can substantially improve grounding reliability\. The results also show that grounding performance varies across vision backbones: replacing Qwen2\.5\-VL\-72B with GPT\-4o further improves strict accuracy to56\.256\.2, and GLM\-4\.6V achieves the best performance with57\.257\.2strict accuracy and64\.864\.8soft accuracy\. These results demonstrate that both multimodal recognition quality and graph\-aware validation are important for accurate starting\-entity grounding, and they also explain why robust downstream path reasoning is necessary when the initial grounding remains imperfect\.

### V\-FAblation Study

TABLE VI:Component ablation onVisPath\-Bench \(SBERT@0\.3\)\. Each row removes one component fromVisPath\.Table[VI](https://arxiv.org/html/2609.20843#S5.T6)reports the component ablation results onVisPath\-Bench, where one component is removed from the full method at a time\. Removing*Graph\-aware validation*disables the relation\-compatibility correction in Eq\. \([3](https://arxiv.org/html/2609.20843#S4.E3)\), so the starting entity is directly selected from the top\-ranked multimodal grounding candidate\. Removing*Visual intent*discards the hop\-specific intentqQ\(i\)q\_\{Q\}^\{\(i\)\}generated by Eq\. \([4](https://arxiv.org/html/2609.20843#S4.E4)\); consequently, outgoing edges in Eq\. \([5](https://arxiv.org/html/2609.20843#S4.E5)\) are matched only against the question text during path discovery in Section[IV\-B](https://arxiv.org/html/2609.20843#S4.SS2)\. Removing*Reasoning\-chain pruning*eliminates the chain\-level scoring in Eq\. \([8](https://arxiv.org/html/2609.20843#S4.E8)\) and the subsequent LLM\-based chain examination in Section[IV\-C](https://arxiv.org/html/2609.20843#S4.SS3), so candidate paths are retained solely according to hop\-level edge scores\. Removing*Evidence checking*disables the answerability verification in Eq\. \([10](https://arxiv.org/html/2609.20843#S4.E10)\) and the stopping rule in Eq\. \([11](https://arxiv.org/html/2609.20843#S4.E11)\), forcing the search to run until the full depth budget is reached before answer generation\.

The results support two main observations\. First, all components contribute to the overall performance, with graph\-aware validation being the most critical one\. Removing this component decreases the average score from64\.564\.5to56\.256\.2and consistently degrades performance at all hop depths\. This finding is consistent with the grounding analysis in Section[V\-E](https://arxiv.org/html/2609.20843#S5.SS5): once an incorrect starting entity is selected, later reasoning stages have limited ability to recover the correct answer path\. Removing visual intent and reasoning\-chain pruning also leads to clear performance drops, reducing the average score by5\.55\.5and6\.06\.0points, respectively\. This shows that hop\-level multimodal guidance and chain\-level consistency filtering address complementary failure modes: the former helps select relevant relations at each step, while the latter prevents locally plausible edges from forming globally inconsistent paths\. Second, evidence checking has a different role from the other components\. Removing it leaves the 2\-hop score unchanged, decreases the 3\-hop score from62\.462\.4to61\.661\.6, and slightly increases the 4\-hop score from57\.057\.0to58\.058\.0\. This is because the answerability gate may occasionally be over\-confident on deeper questions and stop the search with an answer supported by a partial evidence chain\. Without this gate, the model always exhausts the depth budget and may therefore reach the deeper evidence required by some 4\-hop questions\. However, for shallower questions, evidence checking helps avoid unnecessary expansion beyond an already sufficient answer path, which explains the drop when it is removed\. Overall, the full model achieves the best average performance while also avoiding unnecessary deeper exploration, showing that evidence checking provides a useful balance between answer quality and search efficiency\.

### V\-GCost and Efficiency

TABLE VII:Per\-question cost onVisPath\-Bench: text\-LLM calls, vision calls, text tokens, and wall\-clock time\.Table[VII](https://arxiv.org/html/2609.20843#S5.T7)reports the per\-question cost ofVisPathand representative baselines onVisPath\-Bench\. AlthoughVisPathintroduces additional VLM calls to keep multimodal evidence involved during hop\-by\-hop reasoning, its overall cost remains controllable\. With GPT\-4o\-mini,VisPathuses5\.85\.8LLM calls and5\.25\.2VLM calls per question, while consuming only3\.63\.6k text tokens, which is lower than ToG and OmniSearch and comparable to PoG\. Its wall\-clock time is also close to existing multi\-hop and retrieval\-augmented baselines, showing that the additional multimodal reasoning steps do not lead to prohibitive overhead\. Using GPT\-4o increases the runtime because of the stronger and slower backbone, but the number of calls and token usage remain within a moderate range\. These results indicate thatVisPathachieves stronger multimodal multi\-hop reasoning while maintaining a practical and manageable inference cost\.

### V\-HHyperparameter Study

![Refer to caption](https://arxiv.org/html/2609.20843v1/x4.png)\(a\)Edge retentionM1M\_\{1\}
![Refer to caption](https://arxiv.org/html/2609.20843v1/x5.png)\(b\)Chain retentionM3M\_\{3\}
![Refer to caption](https://arxiv.org/html/2609.20843v1/x6.png)\(c\)Depth budgetDmaxD\_\{\\max\}

Figure 4:Sensitivity to the main hyperparameters onVisPath\-Bench \(Avg\. SBERT@0\.3\)\. Stars mark the default setting\.![Refer to caption](https://arxiv.org/html/2609.20843v1/x7.png)Figure 5:Qualitative example: the candidate pathsVisPathenumerates and the path it selects to answer the query\.![Refer to caption](https://arxiv.org/html/2609.20843v1/x8.png)Figure 6:Prompt template of question analysis\.![Refer to caption](https://arxiv.org/html/2609.20843v1/x9.png)Figure 7:Prompt template of intent generation\.![Refer to caption](https://arxiv.org/html/2609.20843v1/x10.png)Figure 8:Example of generating statement\.![Refer to caption](https://arxiv.org/html/2609.20843v1/x11.png)Figure 9:Prompt template for relation\-chain schema judgingFigure[4](https://arxiv.org/html/2609.20843#S5.F4)studies the sensitivity ofVisPathto three key hyperparameters: the per\-hop edge retention sizeM1M\_\{1\}from Section[IV\-B](https://arxiv.org/html/2609.20843#S4.SS2), the chain retention sizeM3M\_\{3\}from Section[IV\-C](https://arxiv.org/html/2609.20843#S4.SS3), and the maximum search depthDmaxD\_\{\\max\}, which truncates the predicted depthD^\\widehat\{D\}from the query analysis when the prediction is overly aggressive\. Overall,VisPathis not highly sensitive to these settings\. IncreasingM1M\_\{1\}from1515to3030improves the average accuracy by covering more useful outgoing edges at each hop, while further increasing it to6060brings no additional gain, suggesting that a moderate retention size already covers the relevant candidates and larger values mainly admit irrelevant ones\. For the chain retention size, performance increases substantially fromM3=1M\_\{3\}\{=\}1toM3=3M\_\{3\}\{=\}3, because keeping a single chain makes the search fragile to one wrong selection, while a small set of globally coherent chains provides sufficient redundancy\. Further increasingM3M\_\{3\}to55leaves the result nearly unchanged\. The depth budget has the smallest impact, as the predicted depthD^\\widehat\{D\}rarely exceeds the actual hop count of the benchmark questions, so enlargingDmaxD\_\{\\max\}changes the executed search for only a small fraction of queries\. These observations show thatVisPathis robust to hyperparameter choices, and the default setting ofM1=30M\_\{1\}\{=\}30,M3=3M\_\{3\}\{=\}3, andDmax=3D\_\{\\max\}\{=\}3provides a good balance between reasoning coverage and cost\.

### V\-ICase Study

Figure[5](https://arxiv.org/html/2609.20843#S5.F5)illustrates a complete reasoning process ofVisPathon a representative query\. Given the input image and the question “Which administrative body is twinned with the birthplace of the architect of the building shown here?”,VisPathfirst grounds the visual content to the starting entityGovernment Museum, Chandigarh\. It then performs three\-hop reasoning, where each hop is guided by a multimodal intent\.

At the first hop,VisPathidentifies that the question requires finding the architect of the depicted building\. Based on the visual appearance of the building and the current reasoning state, it expands candidate relations from the starting entity and selects the path leading toLe Corbusier\. At the second hop, the reasoning intent shifts to the architect’s birthplace\. Among the candidate paths expanded fromLe Corbusier,VisPathselects the path reachingLa Chaux\-de\-Fonds\. At the third hop, the intent further changes to the administrative body twinned with that birthplace\. The model reranks the candidate continuations and retains the path that leads toFrameries, which is returned as the final answer\.

This example shows howVisPathkeeps multimodal evidence involved throughout the reasoning process rather than using it only for initial grounding\. The same input image supports different reasoning intents at different hops, including identifying the building’s architect, locating the architect’s birthplace, and finding the corresponding twinned administrative body\. In contrast, a text\-only method must infer the entire relation sequence from the question alone and cannot adjust its path search using multimodal evidence at intermediate steps\. By repeatedly re\-anchoring the search to the multimodal input and selecting paths at the reasoning\-chain level,VisPathmaintains a coherent evidence path from the starting entity to the final answer\.

### V\-JPrompt Templates

To improve the reproducibility ofVisPath, we present the key prompt templates used in our reasoning pipeline\. Specifically, Figure[6](https://arxiv.org/html/2609.20843#S5.F6)shows the query analysis prompt, which decomposes the input question into a reasoning sketch𝒮Q\\mathcal\{S\}\_\{Q\}and predicts the required reasoning depthD^\\widehat\{D\}\. Figure[7](https://arxiv.org/html/2609.20843#S5.F7)presents the hop\-specific intent generation prompt, which produces the visual reasoning intentqQ\(i\)q\_\{Q\}^\{\(i\)\}at each hop based on the image, the question, and the current partial path\. Figure[8](https://arxiv.org/html/2609.20843#S5.F8)illustrates the prompt for generating the image\-grounded evidence statementziz\_\{i\}, where selected KG paths are summarized together with the visual observation\. All these prompts require strict JSON outputs to ensure stable parsing and controllable downstream execution\. In addition, we provide the prompt templates used for relation\-chain schema judging in Figure[9](https://arxiv.org/html/2609.20843#S5.F9)\.

## VIConclusion

In this paper, we focus on multimodal knowledge graph question answering \(MM\-KGQA\), where a model must jointly ground multimodal input and perform multi\-hop reasoning over structured KG evidence\. We proposeVisPath, a visual\-intent\-guided path reasoning framework\.VisPathkeeps multimodal information involved throughout the reasoning process by combining graph\-aware starting entity identification, hop\-specific intent\-guided path discovery, reasoning\-chain pruning, and evidence checking before answer generation\. This design allows multimodal evidence to support not only entity grounding, but also relation selection, path filtering, and answer verification\. We also introduceVisPath\-Bench, a benchmark designed to evaluate multimodal multi\-hop reasoning over KGs\. Experiments onVisPath\-Bench and three other multimodal QA benchmarks show thatVisPathconsistently improves over strong baselines\. These results demonstrate the importance of keeping multimodal evidence in the reasoning loop for faithful MM\-KGQA\.

A promising future direction is to extendVisPathto temporal MMKGQA, where both multimodal evidence and KG facts evolve over time\. In this setting, models need to identify not only visually grounded entities and reasoning paths, but also temporally valid evidence\. ExtendingVisPathwith time\-aware multimodal intent may further support faithful reasoning over dynamic MMKG\.

## References

- \[1\]H\. Alberts, N\. Huang, Y\. Deshpande,et al\.\(2021\)VisualSem: a high\-quality knowledge graph for vision and language\.InProceedings of the 1st Workshop on Multilingual Representation Learning,pp\. 138–152\.Cited by:[§II\-A](https://arxiv.org/html/2609.20843#S2.SS1.p1.1)\.
- \[2\]A\. Asai, Z\. Wu, Y\. Wang,et al\.\(2024\)Self\-RAG: learning to retrieve, generate, and critique through self\-reflection\.InInternational Conference on Learning Representations,Cited by:[§II\-B](https://arxiv.org/html/2609.20843#S2.SS2.p1.1)\.
- \[3\]S\. Auer, C\. Bizer, G\. Kobilarov,et al\.\(2007\)DBpedia: a nucleus for a web of open data\.InProceedings of the 6th International Semantic Web Conference,Vol\.4825,pp\. 722–735\.Cited by:[§II\-A](https://arxiv.org/html/2609.20843#S2.SS1.p1.1)\.
- \[4\]J\. Baek, A\. F\. Aji, and A\. Saffari\(2023\)Knowledge\-augmented language model prompting for zero\-shot knowledge graph question answering\.InProceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations,pp\. 78–106\.Cited by:[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p2.1)\.
- \[5\]S\. Bai, K\. Chen, X\. Liu,et al\.\(2025\)Qwen2\.5\-VL technical report\.arXiv preprint arXiv:2502\.13923\.Cited by:[§V\-B](https://arxiv.org/html/2609.20843#S5.SS2.p2.1)\.
- \[6\]J\. Berant, A\. Chou, R\. Frostig,et al\.\(2013\)Semantic parsing on Freebase from question\-answer pairs\.InProceedings of the 2013 Conference on Empirical Methods in Natural Language Processing,pp\. 1533–1544\.Cited by:[§I](https://arxiv.org/html/2609.20843#S1.p2.1),[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p1.1)\.
- \[7\]K\. Bollacker, C\. Evans, P\. Paritosh,et al\.\(2008\)Freebase: a collaboratively created graph database for structuring human knowledge\.InProceedings of the ACM SIGMOD International Conference on Management of Data,pp\. 1247–1250\.Cited by:[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p1.1)\.
- \[8\]A\. Bordes, N\. Usunier, A\. Garcia\-Durán,et al\.\(2013\)Translating embeddings for modeling multi\-relational data\.Advances in Neural Information Processing Systems26,pp\. 2787–2795\.Cited by:[§II\-A](https://arxiv.org/html/2609.20843#S2.SS1.p1.1)\.
- \[9\]L\. Chen, P\. Tong, Z\. Jin,et al\.\(2024\)Plan\-on\-Graph: self\-correcting adaptive planning of large language model on knowledge graphs\.Advances in Neural Information Processing Systems37,pp\. 37665–37691\.Cited by:[§I](https://arxiv.org/html/2609.20843#S1.p2.1),[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p2.1)\.
- \[10\]D\. Edge, H\. Trinh, N\. Cheng,et al\.\(2024\)From local to global: a graph RAG approach to query\-focused summarization\.arXiv preprint arXiv:2404\.16130\.Cited by:[§II\-B](https://arxiv.org/html/2609.20843#S2.SS2.p1.1)\.
- \[11\]GLM\-V Team\(2025\)GLM\-4\.5V and GLM\-4\.1V\-Thinking: towards versatile multimodal reasoning with scalable reinforcement learning\.arXiv preprint arXiv:2507\.01006\.Cited by:[§V\-B](https://arxiv.org/html/2609.20843#S5.SS2.p2.1)\.
- \[12\]B\. Gong, S\. Tan, Y\. Feng,et al\.\(2024\)UKnow: a unified knowledge protocol with multimodal knowledge graph datasets for reasoning and vision\-language pre\-training\.Advances in Neural Information Processing Systems37,pp\. 9612–9633\.Cited by:[§II\-A](https://arxiv.org/html/2609.20843#S2.SS1.p1.1)\.
- \[13\]Z\. Guo, L\. Xia, Y\. Yu,et al\.\(2025\)LightRAG: simple and fast retrieval\-augmented generation\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 10746–10761\.Cited by:[§II\-B](https://arxiv.org/html/2609.20843#S2.SS2.p1.1)\.
- \[14\]G\. He, Y\. Lan, J\. Jiang,et al\.\(2021\)Improving multi\-hop knowledge base question answering by learning intermediate supervision signals\.InProceedings of the 14th ACM International Conference on Web Search and Data Mining,pp\. 553–561\.Cited by:[§I](https://arxiv.org/html/2609.20843#S1.p2.1),[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p1.1)\.
- \[15\]X\. He, Y\. Tian, Y\. Sun,et al\.\(2024\)G\-Retriever: retrieval\-augmented generation for textual graph understanding and question answering\.Advances in Neural Information Processing Systems37\.Cited by:[§II\-B](https://arxiv.org/html/2609.20843#S2.SS2.p1.1)\.
- \[16\]A\. Hogan, E\. Blomqvist, M\. Cochez,et al\.\(2021\)Knowledge graphs\.ACM Computing Surveys54\(4\),pp\. 1–37\.Cited by:[§I](https://arxiv.org/html/2609.20843#S1.p1.1)\.
- \[17\]A\. Hurst, A\. Lerer, A\. P\. Goucher,et al\.\(2024\)GPT\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[§V\-B](https://arxiv.org/html/2609.20843#S5.SS2.p2.1)\.
- \[18\]Y\. Lan and J\. Jiang\(2020\)Query graph generation for answering multi\-hop complex questions from knowledge bases\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 969–974\.Cited by:[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p1.1)\.
- \[19\]J\. Lee, C\. Chung, H\. Lee,et al\.\(2023\)VISTA: visual\-textual knowledge graph representation learning\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 7314–7328\.Cited by:[§II\-A](https://arxiv.org/html/2609.20843#S2.SS1.p1.1)\.
- \[20\]P\. Lewis, E\. Perez, A\. Piktus,et al\.\(2020\)Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.Advances in Neural Information Processing Systems33,pp\. 9459–9474\.Cited by:[§II\-B](https://arxiv.org/html/2609.20843#S2.SS2.p1.1)\.
- \[21\]X\. Li, G\. Dong, J\. Jin,et al\.\(2025\)Search\-o1: agentic search\-enhanced large reasoning models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 5420–5438\.Cited by:[§II\-B](https://arxiv.org/html/2609.20843#S2.SS2.p1.1)\.
- \[22\]X\. Li, R\. Zhao, Y\. K\. Chia,et al\.\(2024\)Chain\-of\-Knowledge: grounding large language models via dynamic knowledge adapting over heterogeneous sources\.InInternational Conference on Learning Representations,Cited by:[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p2.1)\.
- \[23\]Y\. Li, Y\. Li, X\. Wang,et al\.\(2025\)Benchmarking multimodal retrieval augmented generation with dynamic VQA dataset and self\-adaptive planning agent\.InInternational Conference on Learning Representations,pp\. 95582–95604\.Cited by:[§II\-B](https://arxiv.org/html/2609.20843#S2.SS2.p1.1),[§V\-B](https://arxiv.org/html/2609.20843#S5.SS2.p2.1)\.
- \[24\]Z\. Li, M\. Jia, Z\. Wei, and J\. Wang\(2024\)Beyond smoothness: a general optimization framework for graph neural networks with negative laplacian regularization\.Neural Networks180,pp\. 106704\.Cited by:[§I](https://arxiv.org/html/2609.20843#S1.p1.1)\.
- \[25\]Z\. Li, M\. Lin, J\. Wang, and S\. Wang\(2025\)Fairness\-aware prompt tuning for graph neural networks\.InProceedings of the ACM on Web Conference 2025,pp\. 3586–3597\.Cited by:[§I](https://arxiv.org/html/2609.20843#S1.p1.1)\.
- \[26\]Z\. Li and J\. Wang\(2024\)Spectral graph neural networks with generalized laguerre approximation\.InIEEE International Conference on Acoustics, Speech and Signal Processing,pp\. 7760–7764\.Cited by:[§I](https://arxiv.org/html/2609.20843#S1.p1.1)\.
- \[27\]W\. Liang, P\. D\. Meo, Y\. Tang,et al\.\(2024\)A survey of multi\-modal knowledge graphs: technologies and trends\.ACM Computing Surveys56\(11\),pp\. 1–41\.Cited by:[§II\-A](https://arxiv.org/html/2609.20843#S2.SS1.p1.1)\.
- \[28\]Y\. Liu, H\. Li, A\. Garcia\-Duran,et al\.\(2019\)MMKG: multi\-modal knowledge graphs\.InEuropean Semantic Web Conference,pp\. 459–474\.Cited by:[§II\-A](https://arxiv.org/html/2609.20843#S2.SS1.p1.1)\.
- \[29\]L\. Luo, Y\. Li, G\. Haffari,et al\.\(2024\)Reasoning on graphs: faithful and interpretable large language model reasoning\.InInternational Conference on Learning Representations,Cited by:[§I](https://arxiv.org/html/2609.20843#S1.p1.1),[§I](https://arxiv.org/html/2609.20843#S1.p3.1),[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p2.1)\.
- \[30\]S\. Ma, C\. Xu, X\. Jiang,et al\.\(2025\)Think\-on\-Graph 2\.0: deep and faithful large language model reasoning with knowledge\-guided retrieval augmented generation\.InInternational Conference on Learning Representations,Cited by:[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p2.1)\.
- \[31\]Y\. Ma, Z\. Wang, M\. Li,et al\.\(2022\)MMEKG: multi\-modal event knowledge graph towards universal representation across modalities\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations,pp\. 231–239\.Cited by:[§II\-A](https://arxiv.org/html/2609.20843#S2.SS1.p1.1)\.
- \[32\]T\. Mensink, J\. Uijlings, L\. Castrejon,et al\.\(2023\)Encyclopedic VQA: visual questions about detailed properties of fine\-grained categories\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 3113–3124\.Cited by:[§I](https://arxiv.org/html/2609.20843#S1.p1.1),[§I](https://arxiv.org/html/2609.20843#S1.p4.1),[§I](https://arxiv.org/html/2609.20843#S1.p7.1),[§V\-B](https://arxiv.org/html/2609.20843#S5.SS2.p1.1),[§V\-B](https://arxiv.org/html/2609.20843#S5.SS2.p3.1)\.
- \[33\]A\. H\. Miller, A\. Fisch, J\. Dodge,et al\.\(2016\)Key\-value memory networks for directly reading documents\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,pp\. 1400–1409\.Cited by:[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p1.1)\.
- \[34\]R\. Navigli and S\. P\. Ponzetto\(2012\)BabelNet: the automatic construction, evaluation and application of a wide\-coverage multilingual semantic network\.Artificial Intelligence193,pp\. 217–250\.Cited by:[§II\-A](https://arxiv.org/html/2609.20843#S2.SS1.p1.1)\.
- \[35\]OpenAI\(2026\)GPT\-5\.4 Thinking system card\.Note:https://deploymentsafety\.openai\.com/gpt\-5\-4\-thinkingCited by:[3rd item](https://arxiv.org/html/2609.20843#S1.I1.i3.p1.1)\.
- \[36\]S\. Pan, L\. Luo, Y\. Wang,et al\.\(2024\)Unifying large language models and knowledge graphs: a roadmap\.IEEE Transactions on Knowledge and Data Engineering36\(7\),pp\. 3580–3599\.Cited by:[§I](https://arxiv.org/html/2609.20843#S1.p1.1),[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p2.1)\.
- \[37\]H\. Park, J\. Seo, M\. Jang,et al\.\(2025\)VAT\-KG: knowledge\-intensive multimodal knowledge graph dataset for retrieval\-augmented generation\.arXiv preprint arXiv:2506\.21556\.Cited by:[§II\-A](https://arxiv.org/html/2609.20843#S2.SS1.p1.1)\.
- \[38\]N\. Reimers and I\. Gurevych\(2019\)Sentence\-BERT: sentence embeddings using Siamese BERT\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,pp\. 3982–3992\.Cited by:[§IV\-A](https://arxiv.org/html/2609.20843#S4.SS1.p2.22),[§V\-B](https://arxiv.org/html/2609.20843#S5.SS2.p3.1),[§V\-B](https://arxiv.org/html/2609.20843#S5.SS2.p4.6)\.
- \[39\]A\. Saxena, A\. Tripathi, and P\. Talukdar\(2020\)Improving multi\-hop question answering over knowledge graphs using knowledge base embeddings\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 4498–4507\.Cited by:[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p1.1),[§V\-B](https://arxiv.org/html/2609.20843#S5.SS2.p2.1)\.
- \[40\]J\. Shi, S\. Cao, L\. Hou,et al\.\(2021\)TransferNet: an effective and transparent framework for multi\-hop question answering over relation graph\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 4149–4158\.Cited by:[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p1.1)\.
- \[41\]F\. M\. Suchanek, G\. Kasneci, and G\. Weikum\(2007\)YAGO: a core of semantic knowledge\.InProceedings of the 16th International Conference on World Wide Web,pp\. 697–706\.Cited by:[§II\-A](https://arxiv.org/html/2609.20843#S2.SS1.p1.1)\.
- \[42\]H\. Sun, T\. Bedrax\-Weiss, and W\. W\. Cohen\(2019\)PullNet: open domain question answering with iterative retrieval on knowledge bases and text\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,pp\. 2380–2390\.Cited by:[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p1.1)\.
- \[43\]H\. Sun, B\. Dhingra, M\. Zaheer,et al\.\(2018\)Open domain question answering using early fusion of knowledge bases and text\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp\. 4231–4242\.Cited by:[§I](https://arxiv.org/html/2609.20843#S1.p2.1),[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p1.1)\.
- \[44\]J\. Sun, C\. Xu, L\. Tang,et al\.\(2024\)Think\-on\-Graph: deep and responsible reasoning of large language model on knowledge graph\.InInternational Conference on Learning Representations,Cited by:[§I](https://arxiv.org/html/2609.20843#S1.p2.1),[§I](https://arxiv.org/html/2609.20843#S1.p3.1),[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p2.1),[§V\-B](https://arxiv.org/html/2609.20843#S5.SS2.p2.1)\.
- \[45\]X\. Tan, X\. Wang, Q\. Liu,et al\.\(2025\)Paths\-over\-Graph: knowledge graph empowered large language model reasoning\.InProceedings of the ACM on Web Conference,pp\. 3505–3522\.Cited by:[§I](https://arxiv.org/html/2609.20843#S1.p2.1),[§I](https://arxiv.org/html/2609.20843#S1.p3.1),[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p2.1),[§V\-B](https://arxiv.org/html/2609.20843#S5.SS2.p2.1)\.
- \[46\]D\. T\. Tran, T\. Tran, M\. Hauswirth,et al\.\(2025\)ReasonVQA: a multi\-hop reasoning benchmark with structural knowledge for visual question answering\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 18793–18803\.Cited by:[§I](https://arxiv.org/html/2609.20843#S1.p1.1),[§I](https://arxiv.org/html/2609.20843#S1.p4.1),[§I](https://arxiv.org/html/2609.20843#S1.p7.1),[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p2.1),[§V\-B](https://arxiv.org/html/2609.20843#S5.SS2.p1.1),[§V\-B](https://arxiv.org/html/2609.20843#S5.SS2.p3.1)\.
- \[47\]M\. Wang, H\. Wang, G\. Qi,et al\.\(2020\)Richpedia: a large\-scale, comprehensive multi\-modal knowledge graph\.Big Data Research22\(100159\)\.Cited by:[§II\-A](https://arxiv.org/html/2609.20843#S2.SS1.p1.1)\.
- \[48\]X\. Wang, T\. Gao, Z\. Zhu,et al\.\(2021\)KEPLER: a unified model for knowledge embedding and pre\-trained language representation\.Transactions of the Association for Computational Linguistics9,pp\. 176–194\.Cited by:[§V\-B](https://arxiv.org/html/2609.20843#S5.SS2.p4.6)\.
- \[49\]X\. Wang, B\. Meng, H\. Chen,et al\.\(2023\)TIVA\-KG: a multimodal knowledge graph with text, image, video and audio\.InProceedings of the 31st ACM International Conference on Multimedia,pp\. 2391–2399\.Cited by:[§II\-A](https://arxiv.org/html/2609.20843#S2.SS1.p1.1)\.
- \[50\]Y\. Wen, Z\. Wang, and J\. Sun\(2024\)MindMap: knowledge graph prompting sparks graph of thoughts in large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,pp\. 10370–10388\.Cited by:[§V\-B](https://arxiv.org/html/2609.20843#S5.SS2.p2.1)\.
- \[51\]J\. Wu, Z\. Deng, W\. Li,et al\.\(2025\)MMSearch\-R1: incentivizing LMMs to search\.arXiv preprint arXiv:2506\.20670\.Cited by:[§II\-B](https://arxiv.org/html/2609.20843#S2.SS2.p1.1),[§V\-B](https://arxiv.org/html/2609.20843#S5.SS2.p2.1)\.
- \[52\]R\. Xie, Z\. Liu, H\. Luan,et al\.\(2017\)Image\-embodied knowledge representation learning\.InProceedings of the 26th International Joint Conference on Artificial Intelligence,pp\. 3140–3146\.Cited by:[§II\-A](https://arxiv.org/html/2609.20843#S2.SS1.p1.1)\.
- \[53\]X\. Ye, S\. Yavuz, K\. Hashimoto,et al\.\(2022\)RNG\-KBQA: generation augmented iterative ranking for knowledge base question answering\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics,pp\. 6032–6043\.Cited by:[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p1.1)\.
- \[54\]W\. Yih, M\. Chang, X\. He,et al\.\(2015\)Semantic parsing via staged query graph generation: question answering with knowledge base\.InProceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing,pp\. 1321–1331\.Cited by:[§I](https://arxiv.org/html/2609.20843#S1.p2.1),[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p1.1)\.
- \[55\]X\. Yuan, L\. Ning, Q\. Ye,et al\.\(2025\)mKG\-RAG: leveraging multimodal knowledge graphs in retrieval\-augmented generation for knowledge\-intensive VQA\.arXiv preprint arXiv:2508\.05318\.Cited by:[§I](https://arxiv.org/html/2609.20843#S1.p4.1),[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p2.1)\.
- \[56\]Z\. Zha, J\. Wang, Z\. Li,et al\.\(2024\)M2ConceptBase: a fine\-grained aligned concept\-centric multimodal knowledge base\.InProceedings of the 33rd ACM International Conference on Information and Knowledge Management,pp\. 3113–3123\.Cited by:[§II\-A](https://arxiv.org/html/2609.20843#S2.SS1.p1.1)\.
- \[57\]J\. Zhang, J\. Wang, X\. Wang,et al\.\(2023\)AspectMMKG: a multi\-modal knowledge graph with aspect\-aware entities\.InProceedings of the 32nd ACM International Conference on Information and Knowledge Management,pp\. 3361–3370\.Cited by:[§II\-A](https://arxiv.org/html/2609.20843#S2.SS1.p1.1)\.
- \[58\]T\. Zhang, Z\. Zhang, Z\. Ma,et al\.\(2026\)MMhops\-R1: multimodal multi\-hop reasoning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 28391–28399\.Cited by:[§I](https://arxiv.org/html/2609.20843#S1.p1.1),[§I](https://arxiv.org/html/2609.20843#S1.p4.1),[§II\-C](https://arxiv.org/html/2609.20843#S2.SS3.p2.1),[§V\-B](https://arxiv.org/html/2609.20843#S5.SS2.p1.1)\.
- \[59\]X\. Zhu, Z\. Li, X\. Wang,et al\.\(2022\)Multi\-modal knowledge graph construction and application: a survey\.IEEE Transactions on Knowledge and Data Engineering36\(2\),pp\. 715–735\.Cited by:[§II\-A](https://arxiv.org/html/2609.20843#S2.SS1.p1.1)\.