ColGraphRAG: Late-Interaction Evidence Retrieval for Multimodal GraphRAG

# Late-Interaction Evidence Retrieval for Multimodal GraphRAG
Source: [https://arxiv.org/html/2607.16208](https://arxiv.org/html/2607.16208)
###### Abstract

Graph\-grounded multimodal question answering organizes text, tables, and images in a structured evidence graph, yet end\-to\-end accuracy depends on which multimodal assets are ranked highly enough to enter downstream reasoning; for graph\-linked images, single\-vector bi\-encoder similarity can discard patch\- and token\-level structure needed for fine\-grained alignment\. We evaluate replacing the visual candidate\-ranking operator over graph\-linked image nodes with late\-interaction MaxSim\-style multi\-vector scoring in the ColBERT/ColPali lineage, while keeping offline graph construction, text\- and table\-side retrieval, structured extraction, and downstream reasoning unchanged\. On MultimodalQA, this change is associated with improved retrieval\-stage point estimates for graph\-linked image candidates and downstream QA gains, with larger movement where visual evidence matters most and mixed trends on text\-dominant questions; we interpret the pattern as mechanism\-level evidence for graph\-linked visual evidence inclusion, while broader validation and finer graph\-level diagnostics remain important future work\.

## 1Introduction

Multimodal question answering over long, visually rich documents requires evidence from text, tables, and figures that may only become decisive when combined\. Graph\-grounded systems link assets in an evidence graph and reason over a subgraph, yet downstream stages only see evidence promoted early enough; decisive figures or tables that rank too low are often irrecoverable\. Recent long\-document and visually grounded QA benchmarks reinforce this setting, where evidence is scattered across pages and modalities\[[4](https://arxiv.org/html/2607.16208#bib.bib4),[5](https://arxiv.org/html/2607.16208#bib.bib5),[18](https://arxiv.org/html/2607.16208#bib.bib18),[19](https://arxiv.org/html/2607.16208#bib.bib19),[21](https://arxiv.org/html/2607.16208#bib.bib21),[20](https://arxiv.org/html/2607.16208#bib.bib20),[22](https://arxiv.org/html/2607.16208#bib.bib22),[23](https://arxiv.org/html/2607.16208#bib.bib23)\]\.

A recurring weak point is the*ranking of graph\-linked image candidates before evidence assembly*\. Query\-driven multimodal GraphRAG\-style stacks often still rank images with CLIP\-style bi\-encoders\[[12](https://arxiv.org/html/2607.16208#bib.bib12)\]that map each query–image pair to one similarity score per side, pooling away local structure that matters when questions refer to specific regions or fine visual cues\. Fine\-grained late\-interaction multimodal retrievers improve visually grounded retrieval in RA\-VQA\[[16](https://arxiv.org/html/2607.16208#bib.bib16),[17](https://arxiv.org/html/2607.16208#bib.bib17)\], complementing pooled bi\-encoder design\.

We study the retrieval\-to\-graph bottleneck by replacing the graph\-linked visual ranker within the same GraphRAG pipeline\. At inference, ColGraphRAG operates on the same pre\-built graphGGas the pooled\-ranking baseline but replaces the graph\-linked visual ranker with ColBERT/ColPali\-style MaxSim scoring over multiple query vectors and local visual units, using deterministic template\-based visual query phrases; offline construction and non\-visual retrieval match the baseline\.

On MultimodalQA, replacing pooled visual ranking is associated with improved retrieval\-stage point estimates on graph\-linked image candidates and higher aggregate EM/F1 point estimates relative to the same system with single\-vector ranking; modality\-level patterns include text\-dominant subsets where scores need not move favorably\. We read these results as mechanism\-level evidence that finer\-grained visual ranking can improve graph\-linked evidence inclusion, without claiming full\-benchmark superiority or statistical significance\.

Using the sameGGand non\-visual retrieval lets us ask whether finer query–image matching improves the evidence available to downstream graph reasoning\.

#### Contributions\.

- •We formulate graph\-linked image candidate ranking as an evidence\-inclusion bottleneck in multimodal GraphRAG: relevant visual nodes may exist in the graph but fail to enter structured extraction if pooled ranking suppresses them\.
- •We introduce ColGraphRAG, which replaces pooled single\-vector visual candidate ranking with late\-interaction MaxSim scoring over graph\-linked image candidates while reusing the same graph, non\-visual retrieval, extraction, and answering stack\.
- •We evaluate the retrieval\-to\-graph effect across candidate\-list metrics, end\-task QA scores, and modality breakdowns on MultimodalQA, with WebQA and ViDoRe v3 serving as contextual and retrieval\-native evaluations\.

## 2Related work

### 2\.1Multimodal QA over long and visually rich documents

Multimodal question answering studies how to answer questions requiring evidence from heterogeneous sources such as text, tables, and images\[[1](https://arxiv.org/html/2607.16208#bib.bib1)\]\. In long or multi\-page settings, relevant evidence is often dispersed across passages, structured tables, and visually rich content, making coordinated retrieval across modalities essential\[[2](https://arxiv.org/html/2607.16208#bib.bib2),[3](https://arxiv.org/html/2607.16208#bib.bib3)\]\. Recent long\-document benchmarks explicitly stress multi\-page understanding and visually grounded reading\[[4](https://arxiv.org/html/2607.16208#bib.bib4),[5](https://arxiv.org/html/2607.16208#bib.bib5)\]; retrieval\-oriented resources further characterize long multimodal documents\[[18](https://arxiv.org/html/2607.16208#bib.bib18)\], realistic multimodal RAG settings\[[19](https://arxiv.org/html/2607.16208#bib.bib19)\], and multi\-page document QA\[[22](https://arxiv.org/html/2607.16208#bib.bib22),[23](https://arxiv.org/html/2607.16208#bib.bib23),[21](https://arxiv.org/html/2607.16208#bib.bib21),[20](https://arxiv.org/html/2607.16208#bib.bib20)\]\. These benchmarks motivate our setting: evidence is heterogeneous, but downstream reasoning only sees assets promoted by retrieval\. We therefore study graph\-linked visual evidence routing in MultimodalQA within a graph\-grounded QA pipeline where the ranking interface is the only changed component\.

### 2\.2Graph\-grounded or graph\-modeled multimodal retrieval and reasoning

Multi\-hop QA benchmarks such as HotpotQA\[[6](https://arxiv.org/html/2607.16208#bib.bib6)\]and MuSiQue\[[7](https://arxiv.org/html/2607.16208#bib.bib7)\]stress structured, multi\-step evidence assembly\. Multimodal graph encoders, query\-conditioned graph construction, and layout\-aware graph modeling further organize heterogeneous evidence for retrieval\-augmented reasoning\[[8](https://arxiv.org/html/2607.16208#bib.bib8),[9](https://arxiv.org/html/2607.16208#bib.bib9),[11](https://arxiv.org/html/2607.16208#bib.bib11)\]\. Those lines primarily reshape how graphs are built or traversed; ColGraphRAG targets the retrieval\-to\-graph interface that decides which graph\-linked visual candidates structured extraction and downstream reasoning can use\.

### 2\.3Late\-interaction retrieval and local visual grounding

Single\-vector bi\-encoder retrieval is efficient but can miss fine\-grained matches between query concepts and local visual evidence\[[12](https://arxiv.org/html/2607.16208#bib.bib12)\]\. Late\-interaction models preserve multiple query and document vectors and aggregate local matches with MaxSim\-style scoring\[[13](https://arxiv.org/html/2607.16208#bib.bib13),[14](https://arxiv.org/html/2607.16208#bib.bib14)\]\. Multimodal extensions study fine\-grained late interaction for visually grounded retrieval in RA\-VQA\[[16](https://arxiv.org/html/2607.16208#bib.bib16),[17](https://arxiv.org/html/2607.16208#bib.bib17)\]\. Vision–language document retrieval increasingly uses patch\- or token\-level representations for pages and documents\[[15](https://arxiv.org/html/2607.16208#bib.bib15)\]\. These methods motivate our scorer choice\. We apply MaxSim directly to graph\-linked image candidates induced by the upstream evidence graph, placing fine\-grained visual matching at the retrieval\-to\-graph boundary where ranking decides which image evidence enters downstream graph stages\. We do not propose a new graph constructor; we isolate the scorer at the retrieval\-to\-graph boundary while using the same graph\-grounded stack\. We follow the ColBERT/ColPali multi\-vector MaxSim*interface*\[[13](https://arxiv.org/html/2607.16208#bib.bib13),[15](https://arxiv.org/html/2607.16208#bib.bib15)\]; Section[3](https://arxiv.org/html/2607.16208#S3)formalizes the scorer used in ColGraphRAG\.

## 3Method

![Refer to caption](https://arxiv.org/html/2607.16208v1/x1.png)Figure 1:Overview of ColGraphRAG\. Given a multimodal knowledge base, ColGraphRAG derives a query\-aware pattern graph and forms a multimodal retrieval graph using text, table, and image evidence\. The figure usesGpG\_\{p\}andGmG\_\{m\}as compact visual labels; in the method, these objects are written asGp​\(q\)G\_\{p\}\(q\)andGm​\(q\)G\_\{m\}\(q\)to emphasize query conditioning\. The visual branch ranks graph\-linked image candidatesx∈𝒳img​\(G\)x\\in\\mathcal\{X\}\_\{\\mathrm\{img\}\}\(G\)\. Query\-side tokensQv=\{qiv\}i=1mQ^\{v\}=\\\{q\_\{i\}^\{v\}\\\}\_\{i=1\}^\{m\}encode deterministic visual phrasesCvis​\(q\)=ImagePhrases​\(q,κ​\(q\),G\)C\_\{\\mathrm\{vis\}\}\(q\)=\\mathrm\{ImagePhrases\}\(q,\\kappa\(q\),G\), while each candidate exposes local visual unitsU​\(x\)=\{uj\}j=1nxU\(x\)=\\\{u\_\{j\}\\\}\_\{j=1\}^\{n\_\{x\}\}\. Late\-interaction MaxSim produces an ordered top\-kkgraph\-linked image candidate listEk​\(q\)E\_\{k\}\(q\)\(Sec\.[3\.3](https://arxiv.org/html/2607.16208#S3.SS3)\)\. Retrieved evidence is assembled intoGe​\(q\)G\_\{e\}\(q\), expanded into the reasoning graphG⋆​\(q\)G^\{\\star\}\(q\), and serialized for answer generation\.We describe ColGraphRAG as a modification of a graph\-grounded multimodal QA pipeline: everything except the visual candidate\-ranking operator over graph\-linked image nodes is unchanged—pooled single\-vector similarity is replaced with late\-interaction MaxSim\-style multi\-vector scoring\. Figure[1](https://arxiv.org/html/2607.16208#S3.F1)sketchesGp​\(q\)→G\_\{p\}\(q\)\\rightarrowgraph\-linked visual ranking over𝒳img​\(G\)\\mathcal\{X\}\_\{\\mathrm\{img\}\}\(G\)→Gm​\(q\)→Ge​\(q\)→G⋆​\(q\)\\rightarrow G\_\{m\}\(q\)\\rightarrow G\_\{e\}\(q\)\\rightarrow G^\{\\star\}\(q\)\. Throughout, graph\-linked candidates arexxwith local visual unitsU​\(x\)U\(x\)\.

### 3\.1Problem setup

Letqqdenote a natural\-language question\. For structured retrieval and extraction we summarize clues by

κ​\(q\)=\(cent,cτ,cvis\)\.\\kappa\(q\)=\(c\_\{\\mathrm\{ent\}\},c\_\{\\tau\},c\_\{\\mathrm\{vis\}\}\)\.Herecentc\_\{\\mathrm\{ent\}\}is an entity clue,cτc\_\{\\tau\}is a temporal clue, andcvisc\_\{\\mathrm\{vis\}\}is a visual clue phrase feeding template\-based phrase construction for the image branch\. Deterministic visual phrases are

Cvis​\(q\)=ImagePhrases​\(q,κ​\(q\),G\)\.C\_\{\\mathrm\{vis\}\}\(q\)=\\mathrm\{ImagePhrases\}\(q,\\kappa\(q\),G\)\.This yields a finite phrase set induced fromκ​\(q\)\\kappa\(q\)under fixed templates onGG—not an additional generative VLM output for ranking\. Visual query tokens for late interaction are

Qv=\{qiv\}i=1m=Enc​\(Cvis​\(q\)\)\.Q^\{v\}=\\\{q\_\{i\}^\{v\}\\\}\_\{i=1\}^\{m\}=\\mathrm\{Enc\}\(C\_\{\\mathrm\{vis\}\}\(q\)\)\.Let𝒳\\mathcal\{X\}denote a finite multimodal candidate pool \(pages, passages, images, tables\) attached toGG\. We distinguish the full pool𝒳\\mathcal\{X\}from the graph\-linked image candidate pool𝒳img​\(G\)\\mathcal\{X\}\_\{\\mathrm\{img\}\}\(G\), which is the only pool rescored by late interaction in this work\. The distinction matters: late interaction only reranks the graph\-linked image pool, while text and table retrieval remain unchanged\. For the*visual*branch, each image\-linked candidatex∈𝒳img​\(G\)x\\in\\mathcal\{X\}\_\{\\mathrm\{img\}\}\(G\)is represented by local units

U​\(x\)=\{uj\}j=1nx,U\(x\)=\\\{u\_\{j\}\\\}\_\{j=1\}^\{n\_\{x\}\}\\,,where eachuju\_\{j\}may be a patch, region, or other visual fragment\. The goal is to predict an answer by retrieving a relevant evidence subset, grounding it in an evidence graph, and reasoning over that graph\.

The single\-vector*visual*baseline pools query and image to one embedding each before matching; ColGraphRAG replaces only that operator with*late\-interaction MaxSim*scoring over𝒳img​\(G\)\\mathcal\{X\}\_\{\\mathrm\{img\}\}\(G\)so alignment stays fine\-grained until scoring\. Text/table retrieval follows the baseline; promoted evidence still gates what downstream stages can use\. When only visual ranking changes whileGG,B​\(q\)B\(q\), extraction templates, and the generator are unchanged, end\-to\-end shifts trace to the visual candidate\-ranking interface—useful attribution for an isolatable bottleneck\.

### 3\.2Offline construction and inference on a pre\-built graph

Offline construction\.We separate*offline*graph construction from*online*inference\. During offline construction, the multimodal corpus is parsed into text, table, image, and entity nodes and serialized \(e\.g\., GraphML\) as a pre\-built graphGG\.

Inference on a pre\-built graph\.Algorithm[1](https://arxiv.org/html/2607.16208#alg1)summarizes the visual\-ranker comparison: graph construction, baseline non\-visual retrieval, and downstream graph\-grounded answering are unchanged, while only the operator that ranks graph\-linked image candidates is replaced with late\-interaction MaxSim scoring\. At inference time, the user query is processed against this graph: baseline text/table/entity retrievalRbaseR\_\{\\mathrm\{base\}\}is unchanged, while the image branch appliesRimgR\_\{\\mathrm\{img\}\}over graph\-linkedIMAGEnodes\. The image branch forms deterministic template\-based visual query phrases and ranks candidates with late\-interaction MaxSim scoring onGG\. This phrase construction is*deterministic and does not call a generative LLM*in the evaluated pipeline; it only supplies text tokens for alignment with image encoders\. Extraction yieldsGe​\(q\)G\_\{e\}\(q\), which is expanded onGGtoG⋆​\(q\)G^\{\\star\}\(q\)for reasoning and answer generation\.

Algorithm 1ColGraphRAG Inference with Late\-Interaction Candidate Ranking1:Question

qq, pre\-constructed graph

GG, candidate budget

kk, expansion rounds

TT
2:Baseline non\-visual retrieval

RbaseR\_\{\\mathrm\{base\}\}; late\-interaction image ranking

RimgR\_\{\\mathrm\{img\}\}
3:Predicted answer

a^\\hat\{a\}\.

4:

B​\(q\)←Rbase​\(q,G\)B\(q\)\\leftarrow R\_\{\\mathrm\{base\}\}\(q,G\)⊳\\trianglerighttext/table/entity retrieval on nodes inGG

5:

P​\(q\)←BuildPattern​\(q,κ​\(q\),G\)P\(q\)\\leftarrow\\mathrm\{BuildPattern\}\(q,\\kappa\(q\),G\)⊳\\trianglerightquery\-conditioned extraction pattern

6:

Cvis​\(q\)←ImagePhrases​\(q,κ​\(q\),G\)C\_\{\\mathrm\{vis\}\}\(q\)\\leftarrow\\mathrm\{ImagePhrases\}\(q,\\kappa\(q\),G\)⊳\\trianglerightdeterministic template\-based visual query phrases

7:

Ek​\(q\)←Rimg​\(Cvis​\(q\),Ximg​\(G\),k\)⊳ordered top\-kgraph\-linked image candidatesE\_\{k\}\(q\)\\leftarrow R\_\{\\mathrm\{img\}\}\\bigl\(C\_\{\\mathrm\{vis\}\}\(q\),X\_\{\\mathrm\{img\}\}\(G\),k\\bigr\)\\quad\\triangleright\\ \\text\{ordered top\-$k$ graph\-linked image candidates\}
8:

R​\(q\)←Extract​\(Ek​\(q\),B​\(q\),P​\(q\)\)R\(q\)\\leftarrow\\mathrm\{Extract\}\\bigl\(E\_\{k\}\(q\),B\(q\),P\(q\)\\bigr\)
9:

Ge​\(q\)←AssembleGraph​\(R​\(q\)\)G\_\{e\}\(q\)\\leftarrow\\mathrm\{AssembleGraph\}\(R\(q\)\)
10:

G⋆​\(q\)←Ge​\(q\)G^\{\\star\}\(q\)\\leftarrow G\_\{e\}\(q\)
11:for

t=1t=1to

TTdo

12:if

Sufficient​\(q,G⋆​\(q\)\)\\mathrm\{Sufficient\}\(q,G^\{\\star\}\(q\)\)then

13:break

14:endif

15:

G⋆​\(q\)←ExpandOnGraph​\(G⋆​\(q\),G\)G^\{\\star\}\(q\)\\leftarrow\\mathrm\{ExpandOnGraph\}\(G^\{\\star\}\(q\),G\)
16:endfor

17:

zser←Serialize​\(G⋆​\(q\)\)z\_\{\\mathrm\{ser\}\}\\leftarrow\\mathrm\{Serialize\}\(G^\{\\star\}\(q\)\)
18:

a^←Generate​\(q,zser\)\\hat\{a\}\\leftarrow\\mathrm\{Generate\}\(q,z\_\{\\mathrm\{ser\}\}\)
19:return

a^\\hat\{a\}

All optional augmentation modules are disabled in MultimodalQA runs; both systems sharekk,TT, the graph snapshot, non\-visual retrievers, extraction templates, and generator\.

### 3\.3Late\-interaction visual retrieval

Figure[3](https://arxiv.org/html/2607.16208#S3.F3)situates the swap along the shared routeq→Gp​\(q\)→q\\rightarrow G\_\{p\}\(q\)\\rightarrow*graph\-linked visual candidate ranking over𝒳img​\(G\)\\mathcal\{X\}\_\{\\mathrm\{img\}\}\(G\)*→Gm​\(q\)→\\rightarrow G\_\{m\}\(q\)\\rightarrowinference, then contrasts \(i\) pooled single\-vector scoring with \(ii\) late\-interaction MaxSim overQvQ^\{v\}andU​\(x\)U\(x\)\. Both sides rank the same fixed pool𝒳img​\(G\)\\mathcal\{X\}\_\{\\mathrm\{img\}\}\(G\); only the scoring operator differs\. The pooled single\-vector baseline maps deterministic visual phrases to one embedding per side:

spool​\(Cvis​\(q\),x\)=⟨f​\(Cvis​\(q\)\),g​\(x\)⟩,f​\(Cvis​\(q\)\),g​\(x\)∈ℝd\.s\_\{\\mathrm\{pool\}\}\(C\_\{\\mathrm\{vis\}\}\(q\),x\)=\\langle f\(C\_\{\\mathrm\{vis\}\}\(q\)\),g\(x\)\\rangle\\,,\\qquad f\(C\_\{\\mathrm\{vis\}\}\(q\)\),g\(x\)\\in\\mathbb\{R\}^\{d\}\\,\.The proposed operator instead keeps token vectorsQv=Enc​\(Cvis​\(q\)\)Q^\{v\}=\\mathrm\{Enc\}\(C\_\{\\mathrm\{vis\}\}\(q\)\)and local candidate unitsU​\(x\)U\(x\), scoring with a MaxSim\-style aggregate\.

For each graph\-linked image candidatex∈𝒳img​\(G\)x\\in\\mathcal\{X\}\_\{\\mathrm\{img\}\}\(G\), query\-side tokensQv=\{qiv\}i=1mQ^\{v\}=\\\{q\_\{i\}^\{v\}\\\}\_\{i=1\}^\{m\}and local unitsU​\(x\)=\{uj\}j=1nxU\(x\)=\\\{u\_\{j\}\\\}\_\{j=1\}^\{n\_\{x\}\}lie in the same retrieval space\. The graph\-linked visual*candidate\-ranking*score is

smv​\(Qv,x\)=∑i=1mmaxu∈U​\(x\)⁡ϕ​\(qiv,u\),s\_\{\\mathrm\{mv\}\}\(Q^\{v\},x\)=\\sum\_\{i=1\}^\{m\}\\max\_\{u\\in U\(x\)\}\\phi\(q\_\{i\}^\{v\},u\)\\,,\(1\)whereϕ​\(⋅,⋅\)\\phi\(\\cdot,\\cdot\)denotes cosine similarity or dot product afterℓ2\\ell\_\{2\}normalization, depending on the encoder\. This differs from pooled retrieval, which collapses each side to one vector before matching\. Deferring pooling preserves local evidence when answers hinge on small visual cues—common when graph\-linked nodes are whole pages or figures but the query targets a localized object, label, or region that pooled scores can miss\. We refer to Eq\.[1](https://arxiv.org/html/2607.16208#S3.E1)as*MaxSim\-style late\-interaction*scoring in the ColBERT/ColPali lineage\[[13](https://arxiv.org/html/2607.16208#bib.bib13),[15](https://arxiv.org/html/2607.16208#bib.bib15)\]; it is*not*applied to text or table rows in our pipeline\. Let𝒳img​\(G\)\\mathcal\{X\}\_\{\\mathrm\{img\}\}\(G\)denote graph\-linked image candidates onGG\. We define the top\-kkgraph\-linked image evidence list by late\-interaction score as

Ek​\(q\)=TopKx∈𝒳img​\(G\)ranked⁡smv​\(Qv,x\),E\_\{k\}\(q\)=\\operatorname\{TopK\}^\{\\mathrm\{ranked\}\}\_\{x\\in\\mathcal\{X\}\_\{\\mathrm\{img\}\}\(G\)\}s\_\{\\mathrm\{mv\}\}\(Q^\{v\},x\),whereEk​\(q\)E\_\{k\}\(q\)denotes the ordered top\-kkgraph\-linked image candidate list sorted bysmvs\_\{\\mathrm\{mv\}\}\. Retrieval\-stage metrics such as Hit@1, MRR@kk, and nDCG@kkare computed on this ranked order, while downstream extraction consumes the top\-kkmembers\. Ties are broken arbitrarily andkkis held constant in experiments\. ThusEk​\(q\)E\_\{k\}\(q\)draws only from𝒳img​\(G\)⊆𝒳\\mathcal\{X\}\_\{\\mathrm\{img\}\}\(G\)\\subseteq\\mathcal\{X\}; we never rank over the full multimodal pool𝒳\\mathcal\{X\}for this visual gate\. Baseline text/table retrieval outputs areB​\(q\)B\(q\); together withEk​\(q\)E\_\{k\}\(q\)they feed structured extraction\.

![Refer to caption](https://arxiv.org/html/2607.16208v1/x2.png)Figure 2:Illustrative case study of evidence linking and graph\-based reasoning\. For the query asking which sport is associated with Ben Piazza’s 1976 film,κ​\(q\)=\(cent,cτ,cvis\)\\kappa\(q\)=\(c\_\{\\mathrm\{ent\}\},c\_\{\\tau\},c\_\{\\mathrm\{vis\}\}\)summarizes entity, temporal, and visual clues\. The temporal cluecτ=1976c\_\{\\tau\}=1976guides a table lookup to a film entitym⋆m^\{\\star\}, and the visual branch retrieves an associated poster candidatep⋆∈ℐ​\(m⋆\)p^\{\\star\}\\in\\mathcal\{I\}\(m^\{\\star\}\)\. The selected visual evidence is converted into a symbolic visual cluez⋆=ψ​\(p⋆\)z^\{\\star\}=\\psi\(p^\{\\star\}\)and inserted intoGe​\(q\)G\_\{e\}\(q\); expansion yieldsG⋆​\(q\)G^\{\\star\}\(q\)for answer generation\. Edgesrctxr\_\{\\mathrm\{ctx\}\},rtabr\_\{\\mathrm\{tab\}\},ryearr\_\{\\mathrm\{year\}\}, andrimgr\_\{\\mathrm\{img\}\}denote context, table lookup, temporal, and image\-linking relations used only to illustrate the evidence path\.
### 3\.4Query\-conditioned structured extraction

Given the ordered top\-kkgraph\-linked image candidate listEk​\(q\)E\_\{k\}\(q\)*and*text\- and table\-linked nodes retrieved fromGGby the baseline graph\-grounded procedure \(not rescored by late interaction\), ColGraphRAG switches from dense scoring to structured evidence processing\. We derive a query\-conditioned extraction patternP​\(q\)P\(q\)that specifies a small schema of entity types, relation labels, and table fields to inspect; extraction is constrained to retrieved evidence and preserves source provenance\. In our evaluated pipeline, this step does not rescore candidates and does not introduce additional visual retrieval\. We run constrained extraction on the combined evidence bundle:

R​\(q\)=Extract​\(Ek​\(q\),B​\(q\),P​\(q\)\)\.R\(q\)=\\mathrm\{Extract\}\\bigl\(E\_\{k\}\(q\),B\(q\),P\(q\)\\bigr\)\.HereB​\(q\)B\(q\)denotes baseline text\- and table\-side retrieval outputs fromGG\.R​\(q\)R\(q\)is a set of structured records \(entity tuples, relation tuples, optional table\-linked rows\)\.

Extraction is query\-aware and limited to retrieved candidates \(no full\-corpus parse at inference\), keeping structured outputs compact and retrieval\-tied\.

### 3\.5Structured evidence graph construction

We mapR​\(q\)R\(q\)to a heterogeneous structured evidence graphGe​\(q\)=\(Ve​\(q\),Ae​\(q\)\)G\_\{e\}\(q\)=\(V\_\{e\}\(q\),A\_\{e\}\(q\)\)\. NodesVe​\(q\)V\_\{e\}\(q\)include normalized entities, table rows or table anchors, graph\-linked image evidence units, and provenance\-bearing source nodes; arcsAe​\(q\)A\_\{e\}\(q\)store extracted relations plus provenance links back to sources\. Construction is deterministic: records are parsed, normalized, merged by identity, and linked with rule\-based assembly rather than latent edge prediction\.

ColGraphRAG applies four steps: \(1\)*Entity normalization and merge*to unify co\-referring mentions; \(2\)*Relation insertion and aggregation*to add or strengthen edges when multiple records support the same relation; \(3\)*Table attachment*, representing tables as structured nodes linked to anchor entities; and \(4\)*Provenance preservation*, recording document, page, image, or table identifiers for each node and edge\.

The graph is*not*a vector\-valued memory: it is a structured, interpretable substrate that organizes evidence already chosen by the retriever\.

![Refer to caption](https://arxiv.org/html/2607.16208v1/x3.png)Figure 3:Single\-vector pooled ranking versus late\-interaction MaxSim candidate ranking\. Both variants follow the same routeq→Gp​\(q\)→q\\rightarrow G\_\{p\}\(q\)\\rightarrowvisual candidate ranking→Gm​\(q\)→\\rightarrow G\_\{m\}\(q\)\\rightarrowinference and rank the same fixed graph\-linked image candidate pool𝒳img​\(G\)\\mathcal\{X\}\_\{\\mathrm\{img\}\}\(G\)\. The pooled baseline maps deterministic visual phrasesCvis​\(q\)C\_\{\\mathrm\{vis\}\}\(q\)and each candidatexxto one vector per side and scores them byspool​\(Cvis​\(q\),x\)=⟨f​\(Cvis​\(q\)\),g​\(x\)⟩s\_\{\\mathrm\{pool\}\}\(C\_\{\\mathrm\{vis\}\}\(q\),x\)=\\langle f\(C\_\{\\mathrm\{vis\}\}\(q\)\),g\(x\)\\rangle\. The late\-interaction ranker instead preserves query\-side token vectorsQv=\{qiv\}i=1mQ^\{v\}=\\\{q\_\{i\}^\{v\}\\\}\_\{i=1\}^\{m\}and local visual unitsU​\(x\)=\{uj\}j=1nxU\(x\)=\\\{u\_\{j\}\\\}\_\{j=1\}^\{n\_\{x\}\}, both inℝd\\mathbb\{R\}^\{d\}, and scores candidates bysmv​\(Qv,x\)=∑i=1mmaxu∈U​\(x\)⁡ϕ​\(qiv,u\)s\_\{\\mathrm\{mv\}\}\(Q^\{v\},x\)=\\sum\_\{i=1\}^\{m\}\\max\_\{u\\in U\(x\)\}\\phi\(q\_\{i\}^\{v\},u\)\. Only this graph\-linked visual ranking operator is substituted\.
### 3\.6Graph\-grounded answering

Evidence assembly yieldsGe​\(q\)G\_\{e\}\(q\); Algorithm[1](https://arxiv.org/html/2607.16208#alg1)initializesG⋆​\(q\)←Ge​\(q\)G^\{\\star\}\(q\)\\leftarrow G\_\{e\}\(q\)and appliesExpandOnGraph​\(⋅,G\)\\mathrm\{ExpandOnGraph\}\(\\cdot,G\)for up toTTrounds until a sufficiency check passes\. Compactly,

G⋆​\(q\)=ExpandOnGraph​\(Ge​\(q\),G,T\),a^=Generate​\(q,Serialize​\(G⋆​\(q\)\)\),G^\{\\star\}\(q\)=\\mathrm\{ExpandOnGraph\}\(G\_\{e\}\(q\),G,T\)\\,,\\qquad\\hat\{a\}=\\mathrm\{Generate\}\\bigl\(q,\\mathrm\{Serialize\}\(G^\{\\star\}\(q\)\)\\bigr\)\\,,\(2\)whereExpandOnGraph​\(Ge​\(q\),G,T\)\\mathrm\{ExpandOnGraph\}\(G\_\{e\}\(q\),G,T\)denotes that iterative expansion contract \(rather than a single\-hop rewrite\)\. Equivalently, writingFqF\_\{q\}for the induced scoring functional over answer strings given serialized graph context,

a^=arg⁡maxa∈𝒜⁡Fq​\(a;G⋆​\(q\)\)\.\\hat\{a\}=\\arg\\max\_\{a\\in\\mathcal\{A\}\}F\_\{q\}\\bigl\(a;G^\{\\star\}\(q\)\\bigr\)\\,\.\(3\)We attach neighboring entities, relations, and table\-linked nodes during expansion to form a compact answering subgraph, serialized to structured text for the generator\.

Retrieval proposes evidence, graph construction organizes it, generation composes the answer; in this regime graph\-grounded QA can bottleneck on*evidence inclusion*, since downstream stages cannot invent missing image support without promotion into the evidence assembly set\. Late interaction keeps fine\-grained alignment until scoring and mainly targets*candidate omission*—the node exists inGGbut pooled ranking excludes it fromEk​\(q\)E\_\{k\}\(q\)—rather than graph\-missing assets or extraction and reasoning failures once evidence is present; empirical results below interpret along this chain\.

## 4Experiments

Table 1:Contextual end\-task results on MultimodalQA and WebQA\. MultimodalQA reports F1 and exact match \(EM\); WebQA reportsQA\-FL,QA\-Acc, and corpus\-levelQA\. Upper rows reproduce protocol\-specific literature numbers and are included only to contextualize scale; they are not treated as an apples\-to\-apples leaderboard\. The bottom two rows compare QD\-MMGraphRAG and ColGraphRAG under the same graph snapshot, candidate budget, non\-visual retrieval, extraction interface, expansion behavior, and generator, differing only in the visual candidate\-ranking operator\. Within each Type block and metric column, bold marks the highest displayed value and underline marks the second\-highest displayed value; em dashes are excluded\.TypeMultimodalQAWebQAModelUnimodalMultimodalAllModelQA\-FLQA\-AccQAF1EMF1EMF1EMSupervisedAR\[[1](https://arxiv.org/html/2607.16208#bib.bib1)\]58\.551\.740\.234\.251\.144\.7VLP\-x101fpn\[[10](https://arxiv.org/html/2607.16208#bib.bib10)\]42\.636\.722\.6ID\[[1](https://arxiv.org/html/2607.16208#bib.bib1)\]58\.451\.651\.244\.655\.548\.8VLP\-VinVL\[[10](https://arxiv.org/html/2607.16208#bib.bib10)\]44\.238\.924\.1MMHQA\-ICL\[[31](https://arxiv.org/html/2607.16208#bib.bib31)\]72\.960\.555\.546\.265\.854\.8MuRAG\[[32](https://arxiv.org/html/2607.16208#bib.bib32)\]55\.754\.636\.1SKURG\[[33](https://arxiv.org/html/2607.16208#bib.bib33)\]69\.766\.157\.252\.564\.059\.8SKURG\[[33](https://arxiv.org/html/2607.16208#bib.bib33)\]55\.457\.137\.7UnsupervisedVicuna\-7B\[[34](https://arxiv.org/html/2607.16208#bib.bib34)\]20\.317\.116\.411\.918\.614\.9Vicuna\-7B\[[34](https://arxiv.org/html/2607.16208#bib.bib34)\]20\.638\.112\.1Llama2Chat\-13b\[[35](https://arxiv.org/html/2607.16208#bib.bib35)\]24\.621\.317\.313\.021\.517\.7Llama2Chat\-13b\[[35](https://arxiv.org/html/2607.16208#bib.bib35)\]19\.538\.013\.2OpenChat\-v2\-w\-13b\[[36](https://arxiv.org/html/2607.16208#bib.bib36)\]25\.322\.018\.915\.522\.519\.2MOQAGPT\[[37](https://arxiv.org/html/2607.16208#bib.bib37)\]25\.544\.817\.1MOQAGPT\[[37](https://arxiv.org/html/2607.16208#bib.bib37)\]54\.649\.133\.830\.545\.641\.1OFA\-Cap\[[38](https://arxiv.org/html/2607.16208#bib.bib38)\]52\.855\.433\.5Binder\[[40](https://arxiv.org/html/2607.16208#bib.bib40)\]————57\.151\.0PROMPTCap\[[39](https://arxiv.org/html/2607.16208#bib.bib39)\]53\.057\.234\.5QD\-MMGraphRAG\[[9](https://arxiv.org/html/2607.16208#bib.bib9)\]66\.464\.535\.534\.254\.753\.0QD\-MMGraphRAG\[[9](https://arxiv.org/html/2607.16208#bib.bib9)\]58\.463\.143\.6ColGraphRAG \(Ours\)68\.665\.545\.642\.258\.355\.0ColGraphRAG \(Ours\)77\.173\.658\.4

### 4\.1Experimental objective and design

We test whether late\-interaction MaxSim changes which graph\-linked image evidence reachesEk​\(q\)E\_\{k\}\(q\)early enough to affect downstream reasoning and answering when visuals matter\. Only that candidate\-ranking operator swaps from pooled similarity to MaxSim;GG,B​\(q\)B\(q\), extraction, evidence\-graph construction, and generation are shared, attributing differences to the retrieval\-to\-graph gateway rather than a new stack\.

The intended causal chain is:

MaxSim⇒Ek​\(q\)⇒R​\(q\)=Extract​\(Ek​\(q\),B​\(q\),P​\(q\)\)⇒G⋆​\(q\)⇒a^\.\\mathrm\{MaxSim\}\\Rightarrow E\_\{k\}\(q\)\\Rightarrow R\(q\)=\\mathrm\{Extract\}\\bigl\(E\_\{k\}\(q\),B\(q\),P\(q\)\\bigr\)\\Rightarrow G^\{\\star\}\(q\)\\Rightarrow\\hat\{a\}\.HereEk​\(q\)E\_\{k\}\(q\)is the ordered top\-kkgraph\-linked image candidate list,R​\(q\)R\(q\)is the structured record bundle, andG⋆​\(q\)G^\{\\star\}\(q\)is the question\-specific reasoning subgraph used for generation\. We do*not*treat aggregate leaderboard rank as the scientific target; end\-task metrics support the downstream link in this chain under the same downstream machinery\.

### 4\.2Datasets and evaluation protocol

We evaluate the same visual\-ranker substitution on graph\-grounded QA, a contextual end\-task benchmark, and retrieval\-native ViDoRe v3\.

MultimodalQA\[[1](https://arxiv.org/html/2607.16208#bib.bib1)\]is primary\. Questions may blend text, tables, and images; we test whether MaxSim changes \(i\) membership inEk​\(q\)E\_\{k\}\(q\)and \(ii\) EM/F1 after unchanged extraction/generation\. Modality tags reflect dominant gold evidence type and do not isolate pure single\-modality runs when assembly still mixes assets\.

WebQA\[[10](https://arxiv.org/html/2607.16208#bib.bib10)\]appears as a paired block in Table[1](https://arxiv.org/html/2607.16208#S4.T1)\. Upper literature rows report numbers under each source paper’s protocol and are not treated here as strict apples\-to\-apples reimplementations in our stack\. Thebottom two rowscompare the two GraphRAG variants under the same graph snapshot, candidate budget, non\-visual retrieval, extraction interface, expansion behavior, and generator, differing only in the visual candidate\-ranking operator; shared settings for that pairwise comparison are summarized in the experimental setup and appendix tables\.

ViDoRe v3\[[41](https://arxiv.org/html/2607.16208#bib.bib41)\]reports NDCG@10 on visually grounded slices and evaluates the same late\-interaction visual evidence ranking interface against strong retrieval baselines\[[15](https://arxiv.org/html/2607.16208#bib.bib15),[18](https://arxiv.org/html/2607.16208#bib.bib18)\], complementing in\-stack candidate\-list metrics on MultimodalQA\.

### 4\.3Metrics and interpreting end\-task scores

*Exact match*\(EM\) is computed after the benchmark’s answer normalization;*F1*is word\-/token\-level overlap between the generated answer and references\. These scores summarize the full graph\-grounded pipeline and are not direct measures of candidate\-list accuracy or complete graph support coverage\. Before downstream extraction and generation, graph\-linked image candidates are evaluated with*Hit@1*,*MRR@kk*, and*nDCG@kk*:*Hit@1*indicates whether a supporting image ranks first;*MRR@kk*is the reciprocal rank of the first relevant candidate within the top\-kk\(zero if none\);*nDCG@kk*discounts relevant candidates by rank under shared relevance labels\. Following the WebQA protocol\[[10](https://arxiv.org/html/2607.16208#bib.bib10)\],*QA\-FL*is a BARTScore\-based fluency measure aligned with reference answers;*QA\-Acc*checks key information via keyword\-style rules with handling for closed answer categories; corpus*QA*averages the per\-example product of QA\-FL and QA\-Acc\.*NDCG@10*summarizes retrieval quality up to rank ten on ViDoRe v3\[[41](https://arxiv.org/html/2607.16208#bib.bib41)\]\.

Retrieval\-stage metrics ask whether the visual scorer promotes relevant graph\-linked images early enough under the shared downstream interface; end\-task EM/F1 ask whether the full pipeline yields reference\-matching answers\. Concordant trends support the routing narrative, but neither metric family alone proves the other\. We therefore read MultimodalQA gains as*consistent with*improved routing when modality tags align with visual mechanisms, and treat text\-tagged rows as coupling diagnostics\.

Table[2](https://arxiv.org/html/2607.16208#S4.T2)reports ViDoRe v3\[[41](https://arxiv.org/html/2607.16208#bib.bib41)\]NDCG@10 across six domain slices \(C\.S\., Phar\., Ind\., Fin\., H\.R\., Phys\.\)\. The Avg\. column averages all six slices when every slice is reported; rows with missing slices show an em dash in Avg\. and are not compared by that aggregate\. The final row evaluates the matched ColGraphRAG visual evidence ranking setting, providing retrieval\-native evidence for the same graph\-linked image ranking mechanism used in the GraphRAG comparison\. We also measure graph\-linked image ranking before extraction/reasoning on MultimodalQA using Hit@1, MRR@3/10, and nDCG@3; because non\-visual stages are unchanged, shifts isolate the visual scorer, and MaxSim improves every reported candidate\-list point estimate versus pooled ranking \(Table[3](https://arxiv.org/html/2607.16208#A1.T3)\)\.

Table 2:ViDoRe v3\[[41](https://arxiv.org/html/2607.16208#bib.bib41)\]retrieval: NDCG@10 \(%\) on six slices \(C\.S\., Phar\., Ind\., Fin\., H\.R\., Phys\.\)\. The ColGraphRAG row is the matched late\-interaction visual evidence ranking setting\. Avg\. is the six\-slice mean when every slice is numeric; otherwise —\. Within each type block,boldandunderlinemark best and second\-best scores per column \(excluding —\)\.TypeModelSizeC\.S\.Phar\.Ind\.Fin\.H\.R\.Phys\.Avg\.Textual RetrieversBGE\-M3\[[24](https://arxiv.org/html/2607.16208#bib.bib24)\]0\.6B58\.052\.028\.539\.842\.435\.942\.8Jina\-v4 \(Textual\)\[[25](https://arxiv.org/html/2607.16208#bib.bib25)\]3B64\.354\.938\.448\.452\.843\.650\.4Qwen3\[[26](https://arxiv.org/html/2607.16208#bib.bib26)\]8B71\.759\.240\.449\.447\.645\.652\.3Qwen3\.5\[[26](https://arxiv.org/html/2607.16208#bib.bib26)\]8B73\.767\.042\.6————Visual RetrieversNomic\[[28](https://arxiv.org/html/2607.16208#bib.bib28)\]7B66\.658\.937\.948\.846\.244\.250\.4ColQwen2\.5\[[15](https://arxiv.org/html/2607.16208#bib.bib15)\]3B72\.357\.941\.352\.351\.245\.953\.5ColEmbed\[[29](https://arxiv.org/html/2607.16208#bib.bib29)\]1B71\.362\.646\.658\.957\.044\.156\.8ColEmbed\[[29](https://arxiv.org/html/2607.16208#bib.bib29)\]3B75\.263\.747\.160\.958\.745\.158\.5ColNomic\[[28](https://arxiv.org/html/2607.16208#bib.bib28)\]3B72\.761\.147\.456\.357\.347\.557\.1ColNomic\[[28](https://arxiv.org/html/2607.16208#bib.bib28)\]7B76\.262\.350\.156\.658\.748\.358\.7Jina\-v4 \(Visual\)\[[25](https://arxiv.org/html/2607.16208#bib.bib25)\]3B71\.863\.150\.459\.359\.546\.658\.5ColEmbed\-v2\[[30](https://arxiv.org/html/2607.16208#bib.bib30)\]3B77\.166\.051\.764\.262\.347\.061\.4ColGraphRAG \(Ours\)3B80\.274\.861\.979\.360\.264\.970\.2
### 4\.4Aggregate and modality\-wise downstream effects

We next ask whether improved evidence routing translates to downstream answer quality; Table[1](https://arxiv.org/html/2607.16208#S4.T1)reports aggregate MultimodalQA F1/EM together with paired WebQA QA\-FL, QA\-Acc, and QA\.

Literature rows are protocol\-contextual and not a joint leaderboard with our bottom comparison rows\. The bottom comparison block shares snapshot, image budget, expansion, non\-visual retrievers, extraction templates, and generator—only visual ranking differs; cell emphasis follows Table[1](https://arxiv.org/html/2607.16208#S4.T1)’s caption \(per Type block and metric column\)\.

With the same pipeline except for the visual ranking operator, ColGraphRAG improves aggregate F1/EM point estimates relative to the baseline graph\-grounded row\[[9](https://arxiv.org/html/2607.16208#bib.bib9)\], matching the expectation that better*candidate ranking over graph\-linked image evidence*can raise end\-task scores when the unchanged pipeline can exploit richer image\-linked context\.

Table[4](https://arxiv.org/html/2607.16208#A1.T4)decomposes*MultimodalQA*F1 and EM by dominant evidence modality under the same comparison\. Image\- and table\-tagged questions improve in both F1 and EM, which aligns with a mechanism story in which graph\-linked assets tied to visuals or to visually anchored layouts are more likely to be promoted into the ordered top\-kkcandidate listEk​\(q\)E\_\{k\}\(q\)and thus participate in extraction and subgraph formation\. Thetextrows show a slight drop in both F1 and EM under the same comparison\. Noise, coupled extraction effects whenEk​\(q\)E\_\{k\}\(q\)shifts, and imperfect modality isolation can explain the dip; we treat it as diagnostic rather than a primary claim, and we expect benefits to concentrate where fine\-grained query–image alignment gates inclusion whereas text/table\-heavy items with incidental images should show neutral or mixed movement, consistent with the modality breakdown\. Definitions for the WebQA columns follow the preceding notation and the WebQA protocol\[[10](https://arxiv.org/html/2607.16208#bib.bib10)\]; literature rows in the WebQA block are contextual, while ColGraphRAG uses the same metric code paths as the bottom comparison rows\.

#### Interpreting the evidence\-inclusion chain\.

Ek​\(q\)E\_\{k\}\(q\)gates graph\-linked image evidence intoR​\(q\)R\(q\)andG⋆​\(q\)G^\{\\star\}\(q\); ranking gains matter only when promoted assets survive evidence assembly\. We read retrieval\-through\-end\-task scores as evidence\-inclusion signals between candidate lists and QA outputs, while finer tabulations \(gold retention through extraction, path coverage\) remain future work\.Ek​\(q\)E\_\{k\}\(q\)highlights the contrast between pooled ranking and MaxSim at the*scoring interface*: pooling collapses each candidate to one global vector before matching, whereas late\-interaction MaxSim preserves local units so different query tokens can align with different regions or patches, which targets omission of a graph\-linked image fromEk​\(q\)E\_\{k\}\(q\)even when it is present inGG\. MaxSim promotions yield improved downstream inputs without changingGGorB​\(q\)B\(q\), whereas text\- or table\-dominated gold evidence yields weaker or mixed movement because editing onlyEk​\(q\)E\_\{k\}\(q\)may perturb coupled extraction \(cf\. text rows above\); late interaction sharpens promotion into structured inputs rather than the full retrieval stack, and qualitative comparisons plus ablations overkk, pools, and phrase templates are deferred\.

### 4\.5Retrieval\-native diagnostics on ViDoRe v3

Table[2](https://arxiv.org/html/2607.16208#S4.T2)evaluates ColGraphRAG’s late\-interaction visual evidence ranking on ViDoRe v3\[[41](https://arxiv.org/html/2607.16208#bib.bib41)\]\. On the matched slices, this setting achieves the strongest six\-slice average NDCG@10 and large gains on Finance and Physics, while H\.R\. remains a mixed domain slice\. These results provide retrieval\-native evidence that MaxSim ranks visually grounded candidates effectively, and they align with the MultimodalQA comparison in whichGG, non\-visual retrieval, extraction, expansion, and generation are shared across the paired comparison\. Appendix[B](https://arxiv.org/html/2607.16208#A2)supplements this aggregate slice summary with top\-kkretrieval curves on Physics and Finance\.

## 5Conclusion

We isolated*graph\-linked image candidate ranking*within a graph\-grounded multimodal QA pipeline by keeping offline construction ofGG, baseline text\- and table\-side retrieval, and graph\-grounded answering unchanged while replacing only the graph\-linked image ranker with late\-interaction MaxSim scoring\[[13](https://arxiv.org/html/2607.16208#bib.bib13),[15](https://arxiv.org/html/2607.16208#bib.bib15)\]; on MultimodalQA this is associated with improved retrieval\-stage candidate\-ranking point estimates and higher aggregate EM/F1 point estimates, but yields mixed modality trends, including a small regression on text\-dominant items\. We scope claims to that substitution without statistical significance, leaderboard optimality, or broader transfer guarantees\. Improved document QA may support education, accessibility, and analyst workflows, whereas the same retrieval machinery can facilitate large\-scale extraction from sensitive corpora if deployed without safeguards\.

## References

- \[1\]A\. Talmor, O\. Yoran, A\. Catav, D\. Lahav, Y\. Wang, A\. Asai, G\. Ilharco, H\. Hajishirzi, and J\. Berant\.MultiModalQA: Complex Question Answering over Text, Tables and Images\.In*International Conference on Learning Representations \(ICLR\)*, 2021\.
- \[2\]M\. Suri, P\. Mathur, F\. Dernoncourt, K\. Goswami, R\. A\. Rossi, and D\. Manocha\.VisDoM: Multi\-Document QA with Visually Rich Elements Using Multimodal Retrieval\-Augmented Generation\.In*Proceedings of NAACL*, 2025\.
- \[3\]R\. Tanaka, T\. Iki, T\. Hasegawa, K\. Nishida, K\. Saito, and J\. Suzuki\.VDocRAG: Retrieval\-Augmented Generation over Visually\-Rich Documents\.In*Proceedings of CVPR*, 2025\.
- \[4\]Y\. Ma, Y\. Zang, L\. Chen, M\. Chen, Y\. Jiao, X\. Li, X\. Lu, Z\. Liu, Y\. Ma, X\. Dong, P\. Zhang, L\. Pan, Y\.\-G\. Jiang, J\. Wang, Y\. Cao, and A\. Sun\.MMLongBench\-Doc: Benchmarking Long\-context Document Understanding with Visualizations\.In*Advances in Neural Information Processing Systems*, 2024\.
- \[5\]C\. Deng, J\. Yuan, P\. Bu, P\. Wang, Z\.\-Z\. Li, J\. Xu, X\.\-H\. Li, Y\. Gao, J\. Song, B\. Zheng, and C\.\-L\. Liu\.LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating\.In*Proceedings of ACL \(Long Papers\)*, 2025\.
- \[6\]Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning\.HotpotQA: A Dataset for Diverse, Explainable Multi\-hop Question Answering\.In*Proceedings of EMNLP*, 2018\.
- \[7\]H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal\.MuSiQue: Multihop Questions via Single\-hop Question Composition\.*Transactions of the Association for Computational Linguistics*, 10:539–554, 2022\.
- \[8\]X\. He and X\. Wang\.Multimodal Graph Transformer for Multimodal Question Answering\.In*Proceedings of EACL*, 2023\.
- \[9\]C\. Bu, G\. Chang, Z\. Chen, C\. Dang, Z\. Wu, Y\. He, and X\. Wu\.Query\-Driven Multimodal GraphRAG: Dynamic Local Knowledge Graph Construction for Online Reasoning\.In*Findings of the Association for Computational Linguistics: ACL 2025*, pages 21360–21380, Vienna, Austria\.Association for Computational Linguistics\.DOI:[10\.18653/v1/2025\.findings\-acl\.1100](https://doi.org/10.18653/v1/2025.findings-acl.1100)\.URL:[https://aclanthology\.org/2025\.findings\-acl\.1100/](https://aclanthology.org/2025.findings-acl.1100/)\.
- \[10\]Y\. Chang, M\. Narang, H\. Suzuki, G\. Cao, J\. Gao, and Y\. Bisk\.WebQA: Multihop and Multimodal Question Answering\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 16495–16504, 2022\.
- \[11\]C\. Yang, D\.\-K\. Vu, M\.\-T\. Nguyen, X\.\-Q\. Nguyen, L\. Nguyen, and H\. Le\.SuperRAG: Beyond RAG with Layout\-Aware Graph Modeling\.In*Proceedings of NAACL \(Industry Track\)*, 2025\.
- \[12\]A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. Sutskever\.Learning Transferable Visual Models From Natural Language Supervision\.In*Proceedings of ICML*, 2021\.
- \[13\]O\. Khattab and M\. Zaharia\.ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT\.In*Proceedings of SIGIR*, 2020\.
- \[14\]K\. Santhanam, O\. Khattab, J\. Saad\-Falcon, C\. Potts, and M\. Zaharia\.ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction\.In*Proceedings of NAACL*, 2022\.
- \[15\]M\. Faysse, H\. Sibille, T\. Wu, B\. Omrani, G\. Viaud, C\. Hudelot, and P\. Colombo\.ColPali: Efficient Document Retrieval with Vision Language Models\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.
- \[16\]W\. Lin, J\. Chen, J\. Mei, A\. Coca, and B\. Byrne\.Fine\-grained Late\-interaction Multi\-modal Retrieval for Retrieval Augmented Visual Question Answering\.In*Advances in Neural Information Processing Systems*, 2023\.
- \[17\]W\. Lin, J\. Mei, J\. Chen, and B\. Byrne\.PreFLMR: Scaling Up Fine\-Grained Late\-Interaction Multi\-modal Retrievers\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, 2024\.
- \[18\]K\. Dong, Y\. Chang, D\. Goh, D\. Li, R\. Tang, and Y\. Liu\.MMDocIR: Benchmarking Multimodal Retrieval for Long Documents\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, 2025\.
- \[19\]N\. Wasserman, R\. Pony, O\. Naparstek, A\. R\. Goldfarb, E\. Schwartz, U\. Barzelay, and L\. Karlinsky\.REAL\-MM\-RAG: A Real\-World Multi\-Modal Retrieval Benchmark\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, 2025\.
- \[20\]Y\. K\. Chia, L\. Cheng, H\. P\. Chan, M\. Song, C\. Liu, M\. Aljunied, S\. Poria, and L\. Bing\.M\-LongDoc: A Benchmark For Multimodal Super\-Long Document Understanding And A Retrieval\-Aware Tuning Framework\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, 2025\.
- \[21\]C\. Jain, Y\. Wu, Y\. Zeng, J\. Liu, S\. Dai, Z\. Shao, Q\. Wu, and H\. Wang\.SimpleDoc: Multi\-Modal Document Understanding with Dual\-Cue Page Retrieval and Iterative Refinement\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, 2025\.
- \[22\]J\. Van Landeghem, R\. Tito, Ł\. Borchmann, M\. Pietruszka, P\. Joziak, R\. Powalski, D\. Jurkiewicz, M\. Coustaty, B\. Anckaert, E\. Valveny, M\. Blaschko, S\. Moens, and T\. Stanislawek\.Document Understanding Dataset and Evaluation \(DUDE\)\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)*, 2023\.
- \[23\]T\. Blau, S\. Fogel, R\. Ronen, A\. Golts, R\. Ganz, E\. Ben Avraham, A\. Aberdam, S\. Tsiper, and R\. Litman\.GRAM: Global Reasoning for Multi\-Page VQA\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2024\.
- \[24\]J\. Chen, S\. Xiao, P\. Zhang, K\. Luo, D\. Lian, and Z\. Liu\.BGE M3\-Embedding: Multi\-lingual, multi\-functionality, multi\-granularity text embeddings through self\-knowledge distillation\.*arXiv preprint arXiv:2402\.03216*, 2024\.
- \[25\]M\. Günther, S\. Sturua, M\. K\. Akram, I\. Mohr, A\. Ungureanu, S\. Eslami, S\. Martens, B\. Wang, N\. Wang, and H\. Xiao\.jina\-embeddings\-v4: Universal embeddings for multimodal multilingual retrieval\.*arXiv preprint arXiv:2506\.18902*, 2025\.
- \[26\]Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin, F\. Huang, and J\. Zhou\.Qwen3 Embedding: Advancing text embedding and reranking through foundation models\.*arXiv preprint arXiv:2506\.05176*, 2025\.
- \[27\]S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang, et al\.Qwen2\.5\-VL technical report\.*arXiv preprint arXiv:2502\.13923*, 2025\.
- \[28\]Nomic Team\.Nomic Embed Multimodal: Interleaved text, image, and screenshots for visual document retrieval\.Nomic AI blog, 2025\.Available at[https://nomic\.ai/blog/posts/nomic\-embed\-multimodal](https://nomic.ai/blog/posts/nomic-embed-multimodal)\.
- \[29\]M\. Xu, G\. Moreira, R\. Ak, R\. Osmulski, Y\. Babakhin, Z\. Yu, B\. Schifferer, and E\. Oldridge\.Llama Nemoretriever Colembed: Top\-performing text\-image retrieval model\.*arXiv preprint arXiv:2507\.05513*, 2025\.
- \[30\]G\. d\. S\. P\. Moreira, R\. Ak, M\. Xu, O\. Holworthy, B\. Schifferer, Z\. Yu, Y\. Babakhin, R\. Osmulski, J\. Cai, R\. Chesler, B\. Liu, and E\. Oldridge\.Nemotron ColEmbed V2: Top\-performing late interaction embedding models for visual document retrieval\.*arXiv preprint arXiv:2602\.03992*, 2026\.
- \[31\]W\. Liu, F\. Lei, T\. Luo, J\. Lei, S\. He, J\. Zhao, and K\. Liu\.MMHQA\-ICL: Multimodal in\-context learning for hybrid question answering over text, tables and images\.*arXiv preprint arXiv:2309\.04790*, 2023\.
- \[32\]W\. Chen, H\. Hu, X\. Chen, P\. Verga, and W\. Cohen\.MuRAG: Multimodal retrieval\-augmented generator for open question answering over images and text\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing*, pages 5558–5570, 2022\.
- \[33\]Q\. Yang, Q\. Chen, W\. Wang, B\. Hu, and M\. Zhang\.Enhancing multi\-modal multi\-hop question answering via structured knowledge and unified retrieval\-generation\.In*Proceedings of the 31st ACM International Conference on Multimedia*, pages 5223–5234, 2023\.
- \[34\]W\.\-L\. Chiang, Z\. Li, Z\. Lin, Y\. Sheng, Z\. Wu, H\. Zhang, L\. Zheng, S\. Zhuang, Y\. Zhuang, J\. E\. Gonzalez, I\. Stoica, and E\. P\. Xing\.Vicuna: An open\-source chatbot impressing GPT\-4 with 90%\* ChatGPT quality\.LMSYS Blog, 2023\.Available at[https://lmsys\.org/blog/2023\-03\-30\-vicuna/](https://lmsys.org/blog/2023-03-30-vicuna/)\.
- \[35\]H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale, et al\.Llama 2: Open foundation and fine\-tuned chat models\.*arXiv preprint arXiv:2307\.09288*, 2023\.
- \[36\]G\. Wang, S\. Cheng, X\. Zhan, X\. Li, S\. Song, and Y\. Liu\.OpenChat: Advancing open\-source language models with mixed\-quality data\.In*International Conference on Learning Representations*, 2024\.
- \[37\]L\. Zhang, Y\. Wu, F\. Mo, J\.\-Y\. Nie, and A\. Agrawal\.MoqaGPT: Zero\-shot multi\-modal open\-domain question answering with large language model\.In*Findings of the Association for Computational Linguistics: EMNLP*, pages 1195–1210, 2023\.
- \[38\]P\. Wang, A\. Yang, R\. Men, J\. Lin, S\. Bai, Z\. Li, J\. Ma, C\. Zhou, J\. Zhou, and H\. Yang\.OFA: Unifying architectures, tasks, and modalities through a simple sequence\-to\-sequence learning framework\.In*Proceedings of the 39th International Conference on Machine Learning*, pages 23318–23340, 2022\.
- \[39\]Y\. Hu, H\. Hua, Z\. Yang, W\. Shi, N\. A\. Smith, and J\. Luo\.PromptCap: Prompt\-guided image captioning for VQA with GPT\-3\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)*, pages 2963–2975, 2023\.
- \[40\]Z\. Cheng, T\. Xie, P\. Shi, C\. Li, R\. Nadkarni, Y\. Hu, C\. Xiong, D\. Radev, M\. Ostendorf, L\. Zettlemoyer, N\. A\. Smith, and T\. Yu\.Binding language models in symbolic languages\.In*International Conference on Learning Representations*, 2023\.
- \[41\]A\. Loison, Q\. Macé, A\. Edy, V\. Xing, T\. Balough, G\. Moreira, B\. Liu, M\. Faysse, C\. Hudelot, and G\. Viaud\.ViDoRe V3: A comprehensive evaluation of retrieval augmented generation in complex real\-world scenarios\.*arXiv preprint arXiv:2601\.08620*, 2026\.

## Appendix ACandidate\-ranking and end\-task breakdowns

This section collects evaluation results on*MultimodalQA*for the main\-text visual\-ranker comparison\. Table[3](https://arxiv.org/html/2607.16208#A1.T3)reports retrieval\-stage quality of graph\-linked image candidate ranking; Table[4](https://arxiv.org/html/2607.16208#A1.T4)reports end\-to\-end F1 and EM by dominant evidence modality on the same benchmark\.

Table 3:MultimodalQA\.Retrieval\-stage quality when replacing pooled visual ranking with late\-interaction MaxSim ranking, measured on ranked candidates from𝒳img​\(G\)\\mathcal\{X\}\_\{\\mathrm\{img\}\}\(G\)*before*downstream graph extraction and generation\. The pooled single\-vector variant uses global image representations, whereas the late\-interaction MaxSim variant ranks candidates in𝒳img​\(G\)\\mathcal\{X\}\_\{\\mathrm\{img\}\}\(G\)with token/patch\-level matching\.Table 4:MultimodalQA\.End\-to\-end diagnostic breakdown when replacing pooled visual ranking with late\-interaction MaxSim ranking\. F1 and EM are reported by dominant gold evidence modality \(image, table, text\), matching Table[1](https://arxiv.org/html/2607.16208#S4.T1)\.
## Appendix BViDoRe v3 subsets: retrieval performance across top\-kk

Figure[4](https://arxiv.org/html/2607.16208#A2.F4)complements the aggregate ViDoRe v3 slice summary in Table[2](https://arxiv.org/html/2607.16208#S4.T2)\[[41](https://arxiv.org/html/2607.16208#bib.bib41)\]\. As the candidate budget increases, Recall@kkimproves, while nDCG@kkand MRR@kkindicate whether relevant evidence remains concentrated near the top of the ranked list\. The*Physics*subset exhibits a gradual gain askkgrows, while the*Finance*subset reaches strong retrieval quality with a smaller candidate budget and shows earlier saturation\.

![Refer to caption](https://arxiv.org/html/2607.16208v1/x4.png)

![Refer to caption](https://arxiv.org/html/2607.16208v1/x5.png)

Figure 4:ViDoRe v3 subsets: retrieval performance across top\-kk\. We report nDCG@kk, Recall@kk, and MRR@kkfor the Physics \(left\) and Finance \(right\) subsets\. Larger candidate budgets increase recall\-oriented coverage; nDCG@kkand MRR@kkadditionally indicate whether promoted candidates remain ranked early enough to be useful, with stronger early saturation on Finance and more gradual movement on Physics\.
## Appendix CQualitative MultimodalQA examples

#### Qualitative examples\.

Figure[5](https://arxiv.org/html/2607.16208#A3.F5)illustrates how to read the qualitative MultimodalQA cases: each panel shows the query, gold evidence and answer, a compact retrieved evidence graph, and the prediction after MaxSim visual ranking\. The examples cover two image\-grounded questions and one table–image compositional question, where the visual cue helps identify the table row used to answer\.

![Refer to caption](https://arxiv.org/html/2607.16208v1/x6.png)Figure 5:Qualitative MultimodalQA examples\. Each panel shows a question, gold evidence, a compact retrieved evidence graph, and the final prediction\. The examples illustrate how graph\-linked visual evidence ranking supports image\-grounded and table–image compositional reasoning in ColGraphRAG\.