Multimodal CoLRAG-TF: Triple-Filtered Retrieval for Complex PDFs

# Multimodal CoLRAG-TF: Triple-Filtered Retrieval for Complex PDFs
Source: [https://arxiv.org/html/2607.20517](https://arxiv.org/html/2607.20517)
\(July 8, 2026\)

###### Abstract

Retrieval\-augmented generation \(RAG\) over heterogeneous PDF collections remains challenging due to multimodal content, domain\-specific terminology, and the need for multi\-hop reasoning across dispersed evidence\. We presentMultimodal CoLRAG\-TF, a four\-axis fusion architecture that integrates dense text embeddings, BM25 keyword matching, knowledge\-graph triple filtering, and image\-based similarity for robust retrieval over complex documents\. Our system constructs a multimodal index of 2,403 blocks extracted from 43 Japanese disaster lesson PDFs, supported by a hybrid OCR pipeline and LLM\-based caption generation\.

To enhance compositional reasoning, we extract 11,414 OpenIE triples and index them with FAISS, enabling sub\-second triple lookup and hierarchical propagation of relevance signals\. A HippoRAG2\-inspired coarse\-to\-fine retriever \(volume→\\tochapter→\\toblock\) narrows the search space before final fusion scoring\. Bayesian optimization over fusion weights reveals that the triple axis must dominate \(αtriple=0\.44\\alpha\_\{\\text\{triple\}\}=0\.44\) to counteract lexical bias and sustain multi\-hop retrieval quality\.

Evaluated on a 457\-pair benchmark,Multimodal CoLRAG\-TFachieves a Retrieval Recall of 0\.9909 and a71\.6%improvement in multi\-hop answer similarity over single\-hop queries\. An image\-to\-lesson pipeline using a vision LLM further demonstrates the applicability of the approach to visual inputs\.

These results show that triple\-filtered multimodal fusion is essential for structured reasoning over noisy, heterogeneous PDFs and provides a general framework applicable beyond the disaster domain\.

Keywords:Retrieval\-Augmented Generation, Triple Filtering, Multimodal Retrieval, Knowledge Graph, Heterogeneous PDF, Bayesian Optimization\.

## 1Introduction

### 1\.1Problem Setting

Retrieval\-augmented generation \(RAG\) over collections of heterogeneous, multimodal PDF documents poses fundamental challenges that standard retrieval systems cannot address\. Such documents—government disaster lesson reports, clinical guidelines, regulatory handbooks, and engineering manuals—integrate dense text, statistical tables, photographs, maps, and domain\-specific terminology within a single collection\. The core challenge is enabling multi\-hop, evidence\-grounded question answering over this heterogeneous content while maintaining retrieval precision for both factual \(single\-hop\) and compositional \(multi\-hop\) queries\.

In this work, we use Japanese government disaster lesson documents as a*representative case study*: a corpus of 43 MLIT publications spanning 67 years of flood, earthquake, landslide, and typhoon events\[[18](https://arxiv.org/html/2607.20517#bib.bib31)\]\. This domain exhibits all key difficulties present in heterogeneous PDF retrieval— multimodal layout variation, scanned\-document OCR noise, dense administrative terminology, and cross\-document multi\-hop reasoning requirements—while providing traceable gold\-standard annotations for rigorous evaluation\. The proposed architecture is general and applicable to any domain sharing these PDF characteristics\.

Standard RAG pipelines\[[16](https://arxiv.org/html/2607.20517#bib.bib1)\]treat documents as flat text sequences, discarding structural and visual information\. Dense embedding models fail to capture precise numeric entities \(e\.g\., “property damage totaling 126\.3 billion yen”\), while keyword\-only BM25 retrieval misses semantic relationships encoded in narrative form\. The challenge is further compounded by multi\-hop queries requiring evidence synthesis across multiple documents\.

### 1\.2Challenges

Retrieval over heterogeneous multimodal PDF collections involves five interacting challenges:

1. C1:Multimodal heterogeneity: A single document page may combine narrative text, statistical tables, satellite maps, and photographs, each requiring a distinct extraction and indexing strategy\.
2. C2:Numeric precision: Dense embedding models fail to capture exact numerical entities \(e\.g\., “property damage totaling 126\.3 billion yen”, “8,667 destroyed buildings”\) that require structured factual retrieval\.
3. C3:Multi\-hop reasoning: Compositional queries require synthesizing evidence dispersed across multiple documents through chains of causal relationships that neither sparse nor dense retrieval alone can capture\.
4. C4:OCR noise: Scanned government PDFs from the 1950s–1990s cause single\-method OCR to fail on 31\.6% of table blocks, truncating the retrievable index\.
5. C5:Domain\-specific terminology: Highly specialized vocabulary \(e\.g\.,okyu\-kiken\-do\-hantei“emergency building danger assessment”\) produces artificially high BM25 scores that overwhelm knowledge\-graph signals unless corrective weight calibration is applied\.

We investigate three corresponding research questions:

1. RQ1:Does integrating structured knowledge\-graph triples into a late\-interaction retriever improve multi\-hop question answering over domain\-specific documents? \(C3\)
2. RQ2:Does four\-axis BM25\-triple fusion improve retrieval recall without degrading semantic precision? \(C1,C2\)
3. RQ3:Can a vision LLM\-based image query pipeline recover topically relevant lesson blocks from purely visual input? \(C1\)

### 1\.3Contributions

1. 1\.Four\-axis fusion architecture: A novel multimodal retrieval scoring function combining dense text embeddings, BM25 keyword matching, knowledge\-graph triple filtering, and image similarity, addressing multimodal heterogeneity \(C1\) and numeric precision \(C2\)\.
2. 2\.Bayesian triple filtering: LLM\-based OpenIE extraction of 11,414 structured triples \(86% success rate\) indexed with FAISS, combined with Bayesian weight calibration that elevatesαtriple\\alpha\_\{\\text\{triple\}\}to 0\.44, overcoming BM25 lexical bias \(C5\)\.
3. 3\.Hybrid OCR pipeline: Two\-stage layout analysis combining Table Transformer detection with PyMuPDF primary extraction and Tesseract fallback, raising table coverage from 68\.4% to 86\.8%, addressing OCR noise \(C4\)\.
4. 4\.Hierarchical coarse\-to\-fine retrieval: A HippoRAG2\-inspired three\-tier retriever \(volume→\\tochapter→\\toblock\) enabling sub\-second triple lookup without PageRank propagation, supporting compositional multi\-hop reasoning \(C3\)\.
5. 5\.Image\-to\-lesson retrieval pipeline: A vision LLM\-based interface that converts disaster photographs to text queries for knowledge\-graph\-augmented retrieval, achieving mean Top\-1 score 0\.6556 across 12 disaster images\.

### 1\.4Paper Organization

Section[2](https://arxiv.org/html/2607.20517#S2)reviews related work on RAG, knowledge\-graph retrieval, and multimodal document understanding\. Section[3](https://arxiv.org/html/2607.20517#S3)presents the full pipeline architecture\. Section[4](https://arxiv.org/html/2607.20517#S4)describes the evaluation corpus and benchmarks\. Section[5\.1](https://arxiv.org/html/2607.20517#S5.SS1)details the experimental setup\. Section[5](https://arxiv.org/html/2607.20517#S5)reports experimental results\. Section[6](https://arxiv.org/html/2607.20517#S6)interprets key findings\. Section[7](https://arxiv.org/html/2607.20517#S7)states system limitations\. Section[8](https://arxiv.org/html/2607.20517#S8)concludes with future directions\. Implementation details are provided in Appendix[A](https://arxiv.org/html/2607.20517#A1)\.

## 2Related Work

### 2\.1Retrieval\-Augmented Generation

RAG\[[16](https://arxiv.org/html/2607.20517#bib.bib1)\]augments LLM generation with retrieved evidence, improving factual grounding without retraining\. Subsequent work has explored dense passage retrieval\[[20](https://arxiv.org/html/2607.20517#bib.bib14)\], late\-interaction architectures\[[15](https://arxiv.org/html/2607.20517#bib.bib6),[22](https://arxiv.org/html/2607.20517#bib.bib7)\], and hybrid dense–sparse fusion\[[21](https://arxiv.org/html/2607.20517#bib.bib11)\]\. Comprehensive surveys\[[8](https://arxiv.org/html/2607.20517#bib.bib2)\]identify multi\-hop reasoning and domain adaptation as key open challenges\. Our work addresses both in the context of Japanese disaster management documents\.

### 2\.2Knowledge Graph Enhanced Retrieval

HippoRAG\[[9](https://arxiv.org/html/2607.20517#bib.bib4)\]models the retrieval index as a hippocampal\-inspired knowledge graph, using personalized PageRank to propagate relevance over entity relationships extracted via OpenIE\. HippoRAG 2\[[10](https://arxiv.org/html/2607.20517#bib.bib5)\]refines the approach with hierarchical indexing and calibration\. GraphRAG\[[6](https://arxiv.org/html/2607.20517#bib.bib3)\]constructs community\-level summaries over entity graphs for global question answering\.

Our CoLRAG\-TF differs from both HippoRAG and GraphRAG in three key respects\. First, it replaces personalized PageRank propagation with a FAISS\-indexed triple store, reducing lookup toO​\(log⁡N\)O\(\\log N\)without graph traversal— critical for the latency constraints of emergency response\. Second, it integrates triple retrieval as one axis of a multi\-axis fusion score rather than using triples as the sole relevance signal, allowing flexible trade\-offs between lexical, semantic, and structured retrieval\. Third, it explicitly addresses BM25\-dominance bias via Bayesian weight calibration, a challenge specific to terminologically dense domain corpora not examined in prior KG\-RAG work\.

### 2\.3Multimodal Document Understanding

Table Transformer\[[24](https://arxiv.org/html/2607.20517#bib.bib16)\]enables structure\-aware table detection from PDF images, while LayoutLMv3\[[12](https://arxiv.org/html/2607.20517#bib.bib17)\]jointly encodes text and layout for document understanding\. MuRAG\[[5](https://arxiv.org/html/2607.20517#bib.bib26)\]pioneered multimodal retrieval combining image embeddings with text passage search\. For disaster document AI, a growing literature\[[13](https://arxiv.org/html/2607.20517#bib.bib29),[2](https://arxiv.org/html/2607.20517#bib.bib30)\]has explored social media analysis, but structured government PDF collections remain underexplored, especially in Japanese\.

### 2\.4Hybrid OCR for Scanned PDFs

Modern PDF extraction pipelines combine layout detection with layered OCR fallback\[[23](https://arxiv.org/html/2607.20517#bib.bib18),[3](https://arxiv.org/html/2607.20517#bib.bib19)\]\. In our Japanese disaster corpus—where many PDFs are scanned government reports with complex table layouts—a single\-method OCR approach achieves only 68\.4% coverage; a Tesseract fallback raises this to 86\.8%, consistent with findings on mixed digital/scanned document collections\.

## 3Methodology

### 3\.1Overview: Multimodal CoLRAG\-TF

Figure[1](https://arxiv.org/html/2607.20517#S3.F1)illustrates the full Multimodal CoLRAG\-TF pipeline\. The system proceeds in two major phases:*offline indexing*\(Sections[3\.2](https://arxiv.org/html/2607.20517#S3.SS2)–[3\.4](https://arxiv.org/html/2607.20517#S3.SS4)\) and*online retrieval and generation*\(Sections[3\.5](https://arxiv.org/html/2607.20517#S3.SS5)–[3\.7](https://arxiv.org/html/2607.20517#S3.SS7)\)\. Implementation details \(model configurations, FAISS settings, GPU memory, and Optuna hyperparameters\) are provided in Appendix[A](https://arxiv.org/html/2607.20517#A1)\.

\(a\) Offline Processing43 Disaster PDFsTable TransformerLayout AnalysisHybrid OCR\(PyMuPDF \+ Tesseract\)Caption Gen\.\(Qwen2\.5\-7B\)2,403 Multimodal BlocksText Embedding\(FAISS 1024\-dim\)BM25 Index\(Bigram\)Triple Extraction\(OpenIE via LLM\)11,414 Triples\(FAISS IndexFlatIP\)\(b\) Online RetrievalUser Query\(Text / Image\)Query Analyzer\(Figure Keywords\)4\-Axis Fusionαt⋅st\+αb⋅sb\\alpha\_\{t\}\\\!\\cdot\\\!s\_\{t\}\+\\alpha\_\{b\}\\\!\\cdot\\\!s\_\{b\}\+αt​r⋅st​r\+αi⋅si\+\\alpha\_\{tr\}\\\!\\cdot\\\!s\_\{tr\}\+\\alpha\_\{i\}\\\!\\cdot\\\!s\_\{i\}Coarse\-to\-Fine\(Vol→\\,\{\\to\}\\,Chap→\\,\{\\to\}\\,Block\)Top\-kkBlocksAnswer Gen\.\(Qwen2\.5\-7B\)Figure 1:Multi\-modal CoLRAG\-TF pipeline\.*\(a\) Offline*: disaster PDFs are processed through layout analysis, hybrid OCR \(PyMuPDF \+ Tesseract\), and caption generation, producing 2,403 multimodal blocks indexed as text embeddings \(FAISS 1024\-dim\), a BM25 keyword index, and 11,414 knowledge triples\.*\(b\) Online*: a user query is analysed for figure intent; four\-axis fusion combines text\-embedding, BM25, triple, and image\-embedding scores; coarse\-to\-fine hierarchical retrieval surfaces the top\-kkblocks for answer generation\. Dashed arrows show offline index data flowing into the online fusion stage\.
### 3\.2Document Processing Pipeline

#### Layout Analysis\.

Each PDF page is rendered at 150 DPI and passed to Table Transformer\[[24](https://arxiv.org/html/2607.20517#bib.bib16)\], a DETR\-based object detection model fine\-tuned for table structure recognition\. We apply a confidence threshold of 0\.7 to filter spurious detections, yielding 38 table regions across a 52\-page OCR evaluation subset of the disaster corpus \(used for the hybrid\-OCR benchmark in Section[5](https://arxiv.org/html/2607.20517#S5)\); across the full corpus, 1,399 figure regions are detected\. Detected table bounding boxes are cropped from the rendered page image and stored as individual PNG files for downstream OCR and captioning\.

#### Hybrid OCR\.

Each cropped table image undergoes a two\-stage extraction process\.*Stage 1*: PyMuPDF directly extracts embedded text from the PDF layer, succeeding on 68\.4% of table blocks\.*Stage 2*: When Stage 1 fails \(e\.g\., scanned images or complex layouts\), Tesseract OCR\[[23](https://arxiv.org/html/2607.20517#bib.bib18)\]with Japanese language model \(jpn\.traineddata\) is applied as a fallback\. The combined hybrid pipeline achieves 86\.8% OCR success \(33 of 38 tables\), an 18\.4 percentage\-point improvement over the single\-stage baseline\. PaddlePaddle was evaluated but excluded due to GPU memory conflicts with the co\-deployed PyTorch environment\.

Figure[2](https://arxiv.org/html/2607.20517#S3.F2)illustrates the cumulative OCR coverage achieved at each pipeline stage\.

![Refer to caption](https://arxiv.org/html/2607.20517v1/x1.png)Figure 2:Hybrid OCR pipeline coverage over 38 table blocks\. Stage 1 \(PyMuPDF\) achieves 68\.4%; adding Tesseract fallback \(Stage 2\) raises coverage to 86\.8%, an 18\.4 percentage\-point gain\.
#### Caption Generation\.

For each successfully extracted table block, Qwen2\.5\-7B\-Instruct\[[19](https://arxiv.org/html/2607.20517#bib.bib22)\]generates a structured caption summarizing the table’s content, disaster context, and key numeric values\. Caption generation succeeds on 86\.8% of blocks\. Failed blocks \(13\.2%\) are assigned the raw OCR text as a fallback caption\. Text paragraphs outside table regions are extracted directly via PyMuPDF and segmented into overlapping 500\-character chunks with 100\-character overlap\.

The output of this stage is a collection of 2,403*multimodal blocks*, each comprising a block ID, type \(text—table\), caption, raw extracted text, source PDF, page number, and bounding\-box coordinates\.

### 3\.3Multimodal Index Construction

#### Dense Embedding Index\.

All block captions and text are encoded withhotchpotch/static\-embedding\-japanese\[[11](https://arxiv.org/html/2607.20517#bib.bib15)\], a 1024\-dimensional static Japanese embedding model optimized for retrieval\. Vectors are L2\-normalized and indexed with FAISSIndexFlatIP\[[14](https://arxiv.org/html/2607.20517#bib.bib13)\]\(inner\-product similarity\), enabling exact nearest\-neighbor search across the 2,403\-block corpus\.

#### BM25 Keyword Index\.

For lexical retrieval we build a BM25Okapi\[[21](https://arxiv.org/html/2607.20517#bib.bib11)\]index over the same 2,430 blocks \(2,403 layout blocks \+ 27 additional text segments\)\. Japanese text is tokenized via a character\-and\-bigram strategy:

T\(s\)=chars\(s\)∪\{s\[i:i\+2\]∣i∈\[0,\|s\|−2\)\}T\(s\)=\\mathrm\{chars\}\(s\)\\cup\\\{s\[i\{:\}i\{\+\}2\]\\mid i\\in\[0,\|s\|\\\!\-\\\!2\)\\\}\(1\)For example, the 4\-character Japanese compoundjishin\-higai\(earthquake damage\) is tokenized into its 4 individual*kanji*characters plus 3 adjacent character bigrams, yielding 7 tokens in total\. This hybrid strategy captures both single\-character precision and two\-character compound term recall, critical for the densely abbreviated language of Japanese disaster reports\. BM25 scores are max\-normalized to\[0,1\]\[0,1\]\. The resulting index occupies 15 MB on disk\.

#### Image Embedding Index\.

For table\-type blocks, a placeholder image score is derived from the caption embedding similarity \(image visual scores are reserved for the vision\-LLM query interface described in Section[3\.7](https://arxiv.org/html/2607.20517#S3.SS7)\)\.

### 3\.4Triple Extraction and Knowledge Graph Index

#### OpenIE Triple Extraction\.

For each block, we prompt Qwen2\.5\-7B\-Instruct with an OpenIE\-style instruction requesting 5–15 structured triples of the form⟨subject,relation,object⟩\\langle\\text\{subject\},\\text\{relation\},\\text\{object\}\\rangle:

> Extract factual triples from the following disaster lesson text\. Prioritize: definitions, causal relations, temporal sequences, quantitative facts\. Output: JSON array of\{“subject”, “relation”, “object”\}\.

Each block is processed individually \(GROUP\_SIZE=1\\\!=\\\!1\) to maximize chunk granularity\. Processing 2,403 blocks yields11,414 triplesat an 86% success rate\. Failed extractions \(14%\) are skipped without fallback to preserve triple quality\.

#### Triple Embedding and Indexing\.

Each triple is linearized ast=subject⊕relation⊕objectt=\\text\{subject\}\\oplus\\text\{relation\}\\oplus\\text\{object\}and encoded with the same 1024\-dimensional embedding model\. Triple vectors are indexed in a dedicated FAISSIndexFlatIPstore\. Each triple record retains the source block ID, volume ID, and chapter ID, enabling upward propagation of triple relevance scores through the three\-tier retrieval hierarchy\.

### 3\.5Hierarchical Coarse\-to\-Fine Retrieval

We implement a three\-tier hierarchical retriever inspired by HippoRAG 2\[[10](https://arxiv.org/html/2607.20517#bib.bib5)\], operating over*Volume*→\\to*Chapter*→\\to*Block*levels\.

#### Level 1: Volume Selection\.

The 43 source documents are grouped into thematic volumes corresponding to the six corpus categories of Section[4](https://arxiv.org/html/2607.20517#S4)\(e\.g\.,Historical Records,Great Earthquake Cases,Heisei/Reiwa Disaster Cases,Recovery Knowledge\)\. A representative 1024\-dimensional vector is computed for each volume as the mean embedding of all constituent block vectors\. For a query vector𝐪\\mathbf\{q\}, the volume embedding score is:

sembV​\(v\)=𝐪⋅𝐯v\+12s^\{V\}\_\{\\text\{emb\}\}\(v\)=\\frac\{\\mathbf\{q\}\\cdot\\mathbf\{v\}\_\{v\}\+1\}\{2\}\(2\)normalized from\[−1,1\]\[\-1,1\]to\[0,1\]\[0,1\]\. Keyword scoresskwVs^\{V\}\_\{\\text\{kw\}\}are computed by matching query tokens against a volume\-level keyword dictionary\. The final volume score fuses both signals:

sV​\(v\)=0\.6⋅sembV​\(v\)\+0\.4⋅skwV​\(v\)s^\{V\}\(v\)=0\.6\\cdot s^\{V\}\_\{\\text\{emb\}\}\(v\)\+0\.4\\cdot s^\{V\}\_\{\\text\{kw\}\}\(v\)\(3\)The top\-nV=2n\_\{V\}\\\!=\\\!2volumes are selected\.

#### Level 2: Chapter Selection\.

Within the selected volumes, chapters \(PDF\-level sub\-divisions\) are scored using a combined embedding and triple signal:

sC​\(c\)=𝐪⋅𝐯c\+γC⋅s¯tripleC​\(c\)s^\{C\}\(c\)=\\mathbf\{q\}\\cdot\\mathbf\{v\}\_\{c\}\+\\gamma\_\{C\}\\cdot\\bar\{s\}^\{C\}\_\{\\text\{triple\}\}\(c\)\(4\)whereγC=0\.3\\gamma\_\{C\}\\\!=\\\!0\.3ands¯tripleC​\(c\)\\bar\{s\}^\{C\}\_\{\\text\{triple\}\}\(c\)is the mean triple similarity score aggregated over all triples whose source block belongs to chaptercc\. Top\-nC=3n\_\{C\}\\\!=\\\!3chapters proceed to the block stage\.

#### Level 3: Block Re\-Ranking\.

Candidate blocks are gathered from the selected chapters, supplemented by triple\-filtered blocks—any block cited by a top\-20 retrieved triple is added to the candidate set regardless of its chapter membership\. If the candidate set is smaller than the requested top\-kk, all blocks from the selected volumes serve as a fallback\. Candidates are re\-ranked by their four\-axis fusion score \(Section[3\.6](https://arxiv.org/html/2607.20517#S3.SS6)\)\.

### 3\.6Four\-Axis Fusion Scoring

The central contribution of v0\.7\.3 is a four\-axis retrieval score:

s​\(b\)=αt​st​\(b\)\+αb​sb​\(b\)\+αt​r​st​r​\(b\)\+αi​si​\(b\)s\(b\)=\\alpha\_\{t\}s\_\{t\}\(b\)\+\\alpha\_\{b\}s\_\{b\}\(b\)\+\\alpha\_\{tr\}s\_\{tr\}\(b\)\+\\alpha\_\{i\}s\_\{i\}\(b\)\(5\)wherests\_\{t\}is the normalized dense embedding similarity,sbs\_\{b\}the normalized BM25 score,st​rs\_\{tr\}the top\-triple similarity for the block’s triples, andsis\_\{i\}the image similarity score \(currently caption\-embedding based\)\. The default weights in v0\.7\.3 are:\(αt,αb,αt​r,αi\)=\(0\.4,0\.3,0\.2,0\.1\)\(\\alpha\_\{t\},\\alpha\_\{b\},\\alpha\_\{tr\},\\alpha\_\{i\}\)=\(0\.4,0\.3,0\.2,0\.1\)\.

#### Query Analyzer\.

Before scoring, aQueryAnalyzermodule detects figure\-related Japanese keywords \(e\.g\.,hyo“table”,zu“figure”,grāfu“graph”,chizu“map”, …\) in the query\. When detected, a×1\.2\\times 1\.2boost is applied to table\-type blocks, ensuring that tabular content is prioritized for queries about statistical summaries\.

#### Bayesian Weight Optimization \(v0\.7\.4\)\.

Manual weight configuration \(v0\.7\.3\) led to BM25\-dominance: 90% of top\-5 results were BM25\-dominant, with triple scores contributing to only 11\.7% of results\. We apply Optuna\[[1](https://arxiv.org/html/2607.20517#bib.bib24)\]with the TPE sampler\[[4](https://arxiv.org/html/2607.20517#bib.bib25)\]over 50 trials to minimize:

ℒ=−\(0\.5⋅P​@​5\+0\.2⋅D\+0\.3⋅Q\)\\mathcal\{L\}=\-\\bigl\(0\.5\\cdot P@5\+0\.2\\cdot D\+0\.3\\cdot Q\\bigr\)\(6\)whereP​@​5P@5is Precision@5 against 40 gold\-standard blocks,DDis entropy\-based axis diversity, andQQis LLM\-as\-Judge answer quality\. The optimized weights are:\(αt,αb,αt​r,αi\)=\(0\.2675,0\.2903,0\.4422,0\.10\)\(\\alpha\_\{t\},\\alpha\_\{b\},\\alpha\_\{tr\},\\alpha\_\{i\}\)=\(0\.2675,0\.2903,0\.4422,0\.10\), more than doubling the triple axis weight and reducing BM25 dominance to 51\.7% while increasing triple contribution to 41\.7% of top\-5 results\.

### 3\.7Image\-Based Query Interface

To support visual query inputs, we implement a two\-stage pipeline:

#### Stage 1: Image Understanding\.

A disaster photograph is encoded to base64 and submitted to LLaVA\-7B\[[17](https://arxiv.org/html/2607.20517#bib.bib20)\]via the Ollama inference server\. The vision LLM generates a structured description of the observed disaster scenario \(type, severity, location characteristics, observed damage\)\. This natural\-language description becomes the text query for Stage 2\.

#### Stage 2: Lesson Retrieval and Synthesis\.

The vision\-generated description is processed by the four\-axis retrieval system \(Section[3\.6](https://arxiv.org/html/2607.20517#S3.SS6)\) to retrieve the top\-k=5k\\\!=\\\!5most relevant disaster lesson blocks\. Qwen2\.5\-7B\-Instruct then synthesizes a structured lesson summary \(pre\-disaster preparation, emergency response, recovery strategy\) from the retrieved context\. The full pipeline—image input to lesson text—operates in 15–35 seconds on a single RTX 4060 Ti 16 GB GPU\.

## 4Dataset

### 4\.1Disaster Corpus: Domain\-Specific Evaluation Benchmark

We use Japanese government disaster lesson documents as a domain\-specific evaluation benchmark\. While the corpus is disaster\-centric, it exhibits all five challenges \(C1–C5\) and provides traceable gold\-standard annotations, making it an ideal stress\-test for heterogeneous PDF retrieval\. The architecture and pipeline are domain\-agnostic; adaptation to other domains \(clinical, legal, engineering\) requires only corpus\-specific tokenization and embedding fine\-tuning\.

The corpus consists of 43 PDF documents published by the Japanese Ministry of Land, Infrastructure, Transport and Tourism \(MLIT\)\[[18](https://arxiv.org/html/2607.20517#bib.bib31)\], covering disaster events from 1958 to 2025:

- •Historical Records\(17 volumes,Rekishi\-shiryo\-shu\): Damage statistics and lessons for floods, landslides, and typhoons from 1958–2019\.
- •Great Earthquake Cases\(4 documents\): Post\-event analyses of the 2011 Tōhoku, 2016 Kumamoto, and 2018 Hokkaido earthquakes\.
- •Heisei Disaster Cases\(15 documents,Heisei\-no\-Saigai\-Jirei\): River engineering lessons from major events including Typhoon No\. 12 \(2011\), Kumamoto Floods \(2016\), and the July 2018 Heavy Rain\.
- •Reiwa Era Cases\(1 document, 2025\)\.
- •Recovery Knowledge\(5 documents,Fukko\-chiken\): Reconstruction process handbooks and lesson compilations from the 2011 Great East Japan Earthquake\.
- •East Japan Earthquake Lessons\(1 document\)\.

Total corpus statistics: 2,319 pages, 1,399 detected figures \(tables and embedded images\), yielding 2,403 indexed multimodal blocks\.

### 4\.2QA Benchmark

We construct a 457\-pair QA benchmark through a three\-stage pipeline:*\(1\)*a small language model \(Qwen2\.5\-7B\) generates 200 single\-hop question–answer pairs from individual block captions;*\(2\)*a compositional multi\-hop generation step\[[7](https://arxiv.org/html/2607.20517#bib.bib36)\]creates 400 multi\-hop pairs requiring evidence from≥2\\geq\\\!2blocks;*\(3\)*an LLM\-based quality filter\[[25](https://arxiv.org/html/2607.20517#bib.bib37)\]retains 457 high\-quality pairs \(169 single\-hop \+ 288 multi\-hop\)\. Each pair is annotated with a gold\-standard block ID list for retrieval evaluation\.

### 4\.3Image Query Test Set

Twelve representative disaster photographs are selected from MLIT publications and open news archives, covering flood, earthquake, landslide, and typhoon scenarios\. Images are manually annotated with relevant block IDs from the corpus \(mean: 3\.3 relevant blocks per image\) to serve as a small\-scale image retrieval benchmark\.

## 5Experiments

### 5\.1Experimental Setup

#### Hardware\.

All experiments run on a single workstation with an NVIDIA RTX 4060 Ti \(16 GB VRAM\), an Intel Core i9 CPU, and 64 GB RAM\. GPU memory is shared between the LLM inference server \(Ollama\) and the FAISS/embedding pipeline\.

#### Models\.

- •Embedding:hotchpotch/static\-embedding\-japanese\(1024\-dim\)
- •Caption/Answer LLM:qwen2\.5:7b\-instruct\-q4\_k\_mvia Ollama
- •Vision LLM:llava:7bvia Ollama
- •Triple Extraction LLM:qwen2\.5:7b\-instruct\-q4\_k\_m

#### Evaluation Metrics\.

- •Answer Similarity: Cosine similarity between the embedding of the generated answer and the gold\-standard answer\.
- •Retrieval Recall: Fraction of gold\-standard block IDs present in the top\-5 retrieved blocks \(mean and median\)\.
- •Triple Peak Score: Maximum triple similarity score among top\-5 retrieved blocks for a given query \(measures knowledge\-graph effectiveness\)\.
- •Top\-1 Score: Final fusion score of the highest\-ranked block \(used for image query evaluation\)\.

### 5\.2Baselines

We compare against two baselines:

- •Naive RAG: Flat dense embedding retrieval over all 2,403 blocks with no hierarchical filtering or triple augmentation\.
- •CoLRAG\-TF v0\.7\.2: Two\-axis fusion \(text\+\+triple,αt=0\.6\\alpha\_\{t\}\\\!=\\\!0\.6,αt​r=0\.4\\alpha\_\{tr\}\\\!=\\\!0\.4\) without BM25 keyword retrieval\.

The proposed system is denotedMultimodal CoLRAG\-TF v0\.7\.3\(four\-axis, BM25 restored\); its Bayesian weight\-optimized variant \(Section[3\.6](https://arxiv.org/html/2607.20517#S3.SS6)\) is denoted v0\.7\.4\.

### 5\.3Overall Results

Table[1](https://arxiv.org/html/2607.20517#S5.T1)reports system\-level results across all 457 QA pairs\.

Table 1:Overall evaluation: CoLRAG\-TF v0\.7\.2 \(2\-axis\) vs\. Multimodal CoLRAG\-TF v0\.7\.3 \(4\-axis\)\.The introduction of BM25 maintains overall Answer Similarity within measurement error \(<1%<\\\!1\\%change\)\. The overall Retrieval Recall of 0\.9909 demonstrates near\-perfect single\-step coverage\.

### 5\.4Single\-Hop vs\. Multi\-Hop Performance

Table[2](https://arxiv.org/html/2607.20517#S5.T2)breaks down performance by query complexity\.

Table 2:Single\-hop \(1\-hop\) vs\. multi\-hop performance comparison\.RQ1 \(Triple Filtering\):The multi\-hop advantage of\+71\.6%\+71\.6\\%in Answer Similarity directly validates that triple\-augmented retrieval enables compositional reasoning across document boundaries\. The near\-identical multi\-hop gains for v0\.7\.2 \(\+72\.6%\+72\.6\\%\) and v0\.7\.3 \(\+71\.6%\+71\.6\\%\) confirm that adding BM25 keyword fusion preserves this triple\-driven compositional advantage rather than diluting it\. For 1\-hop queries, BM25 integration raises the Median Similarity by\+11\.9%\+11\.9\\%\(0\.2872→0\.32140\.2872\\to 0\.3214\), indicating improved block selection for factual queries\.

RQ2 \(BM25 Integration\):Under the four\-axis configuration, overall Retrieval Recall reaches 0\.9909 \(Retrieval Recall was not separately measured for the v0\.7\.2 baseline\)\. Multi\-hop Recall remains at 0\.5, suggesting that the gold\-standard block annotations for multi\-hop questions are incomplete \(some relevant blocks are correct but not annotated\), or that multi\-hop synthesis requires inter\-block reasoning beyond recall\-based evaluation\.

Figure[3](https://arxiv.org/html/2607.20517#S5.F3)visualises the full breakdown, including median similarity which shows the strongest BM25 effect \(\+11\.9% for 1\-hop\)\.

![Refer to caption](https://arxiv.org/html/2607.20517v1/x2.png)Figure 3:Answer Similarity comparison between v0\.7\.2 \(2\-axis\) and v0\.7\.3 \(4\-axis with BM25\) for 1\-hop and multi\-hop queries\. Solid bars show mean similarity; hatched bars show median\. The multi\-hop advantage of\+71\.6%\+71\.6\\%validates triple\-augmented compositional reasoning\.
### 5\.5Triple Score Effectiveness

Table[3](https://arxiv.org/html/2607.20517#S5.T3)shows triple retrieval scores for three representative queries from the Phase 6 demo\.

Table 3:Triple score effectiveness for representative queries\.The triple peak score of 0\.9265 for the numeric fact query \(“number of completely destroyed buildings”\) confirms that knowledge\-graph triples encode precise quantitative relationships highly relevant to specific factual questions\. The figure\-boost mechanism correctly promotes table\-type blocks for the table\-reference query \(“damage amount shown in the table”\)\.

Figure[4](https://arxiv.org/html/2607.20517#S5.F4)compares text, triple, and fused scores across the three representative query types\.

![Refer to caption](https://arxiv.org/html/2607.20517v1/x3.png)Figure 4:Text score, triple score, and Top\-1 fusion score for three representative queries\. Triple retrieval is inactive for open\-ended typhoon queries \(0\.0\) but reaches 0\.9265 for specific numeric fact queries, outperforming dense embedding search\.
### 5\.6Image Query Results

Table[4](https://arxiv.org/html/2607.20517#S5.T4)summarizes image\-based retrieval over 12 disaster photographs\.

Table 4:Image\-based query retrieval results \(12 disaster images\)\.RQ3 \(Image Query\):A mean Top\-1 Score of 0\.6556 demonstrates that vision\-LLM\-mediated image queries can surface topically relevant disaster lesson blocks\. Triple scores \(0\.57\) exceed text scores \(0\.53\) on average, validating knowledge\-graph effectiveness for image\-originated queries\. The 41\.7% disaster type recognition rate \(LLaVA\-7B correctly identifying the depicted disaster category\) is the primary limiting factor and represents a critical gap for production deployment\.

Figure[5](https://arxiv.org/html/2607.20517#S5.F5)shows per\-image Top\-1, text, and triple scores across all 12 test images, highlighting score variance and the region where triple retrieval exceeds text retrieval\.

![Refer to caption](https://arxiv.org/html/2607.20517v1/x4.png)Figure 5:Per\-image retrieval scores for 12 disaster photographs\. Solid line: Top\-1 fusion score \(mean 0\.6556\)\. Triple scores \(green dashed\) exceed text scores \(blue dashed\) for the majority of images, confirming knowledge\-graph superiority for visual queries\. The shaded region marks images where triple\>\>text\.
### 5\.7Bayesian Fusion Weight Optimization

Table[5](https://arxiv.org/html/2607.20517#S5.T5)contrasts the manual v0\.7\.3 weights with the Optuna\-optimized v0\.7\.4 weights and their downstream effects\.

Table 5:Fusion weight evolution and axis dominance impact\.The Bayesian optimizer converges at trial 14 \(of 50\), settling onαtriple=0\.442\\alpha\_\{\\text\{triple\}\}\\\!=\\\!0\.442—more than double the manual setting\. Despite a 15\.2% decrease in raw fusion score \(reflecting the deliberate trade\-off toward diversity and answer quality in the objective function\), the knowledge\-graph axis becomes the dominant retrieval signal for 40% of results, compared to 5% under manual configuration\.

Figures[6](https://arxiv.org/html/2607.20517#S5.F6)and[7](https://arxiv.org/html/2607.20517#S5.F7)illustrate the shift in axis dominance and the corresponding weight evolution\. Figure[8](https://arxiv.org/html/2607.20517#S5.F8)\(left\) shows Retrieval Recall by question type; \(right\) shows the Optuna objective convergence over 50 trials\.

![Refer to caption](https://arxiv.org/html/2607.20517v1/x5.png)Figure 6:Retrieval axis dominance distribution before \(v0\.7\.3 manual\) and after \(v0\.7\.4 Bayesian\) weight optimization\. BM25\-dominant results drop from 90% to 51\.7%; triple\-dominant results increase from 5% to 40%\.![Refer to caption](https://arxiv.org/html/2607.20517v1/x6.png)Figure 7:Fusion weight evolution from manual configuration \(v0\.7\.3\) to Bayesian\-optimized values \(v0\.7\.4\)\. The triple weight more than doubles \(αt​r:0\.20→0\.44\\alpha\_\{tr\}\\colon 0\.20\\to 0\.44, \+121%\), becoming the dominant axis\.![Refer to caption](https://arxiv.org/html/2607.20517v1/x7.png)Figure 8:Left: Retrieval Recall@5 by question type \(v0\.7\.3\)\. Overall recall is 0\.9909; multi\-hop recall is capped at 0\.5 due to hypothesised annotation incompleteness\.Right: Optuna TPE convergence over 50 trials; best\-score plateau is reached at trial 14 \(objective=0\.41\\text\{objective\}\\\!=\\\!0\.41\)\.

## 6Discussion

### 6\.1Triple Weight Must Be Dominant for Disaster Knowledge Reasoning

The most striking finding is that intuitive weight assignment \(αtriple=0\.2\\alpha\_\{\\text\{triple\}\}\\\!=\\\!0\.2\) is insufficient for knowledge\-graph retrieval to compete with BM25 keyword matching in a disaster document corpus\. Disaster lesson text is rich in standardized administrative terminology \(e\.g\.,okyu\-kiken\-do\-hantei“emergency building danger assessment”,hisai\-do\-kubun“damage degree classification”\) that produces extremely high BM25 scores \(0\.96–0\.98\), overwhelming the triple axis unless its weight is raised toαtriple≥0\.40\\alpha\_\{\\text\{triple\}\}\\geq 0\.40\. This finding—confirmed via 50\-trial Bayesian optimization—suggests that in terminologically dense domain corpora, knowledge\-graph retrieval requires protective weighting against lexical retrieval bias\.

Because natural disasters are inherently rare and geographically localized, the informational density of damage descriptions is often sparse\. This sparsity further increases the importance of triple filtering and table\-derived statistical facts, which provide structured signals that lexical methods alone cannot reliably capture\.

### 6\.2Multi\-Hop Recall Ceiling and Annotation Incompleteness

The multi\-hop Retrieval Recall ceiling at 0\.5 across both v0\.7\.2 and v0\.7\.3 raises a methodological concern: the gold\-standard block ID annotations for multi\-hop questions may themselves be incomplete, as multi\-hop answer synthesis can draw on information distributed across blocks not captured in the annotation \(which typically records only the primary evidence block\)\. Answer Similarity remains high \(0\.5605\) despite Recall 0\.5, supporting the hypothesis that the system retrieves semantically sufficient \(if not annotated\) context for multi\-hop synthesis\. Future work should adopt a more expressive annotation schema \(e\.g\., soft relevance judgments via RAGAS\[[7](https://arxiv.org/html/2607.20517#bib.bib36)\]\) for multi\-hop evaluation\.

### 6\.3Hybrid OCR is Essential for Scanned Government PDFs

The 18\.4\-percentage\-point improvement from Tesseract fallback confirms that no single OCR method is sufficient for the heterogeneous quality of MLIT disaster PDFs, which span digital\-native reports \(2015–2025\) and scanned legacy documents \(1958–1993\)\. PaddlePaddle, while offering superior accuracy on standard benchmarks, introduced GPU driver conflicts with the PyTorch/Ollama stack and was excluded\. The lesson is that*infrastructure compatibility*is as important as raw OCR accuracy in production multimodal RAG systems\.

### 6\.4Disaster Type Recognition as a Bottleneck for Image Queries

LLaVA\-7B correctly identified the depicted disaster category in only 5 of 12 test images \(41\.7%\)\. This misclassification propagates to the retrieval stage: if a flood image is described as “earthquake damage”, the retrieved lesson blocks may be thematically mismatched despite having high BM25/triple scores for generic disaster keywords\. Mitigations include: fine\-tuning a disaster\-specific vision classifier, adopting a larger VLM \(e\.g\., Qwen2\.5\-Omni\-7B or LLaVA\-NeXT\-13B\), or introducing a secondary visual re\-ranking step using CLIP\-based image\-to\-block matching\. Although Qwen2\.5\-Omni\-7B offers stronger visual reasoning than LLaVA\-7B, its deployment is currently constrained by the mllama runtime: Omni models rely on a non\-standard vision projector and tokenizer configuration that is not yet fully supported in llama\.cpp or Ollama, making model loading and GGUF conversion unstable\. LLaVA\-NeXT\-13B provides further gains in disaster\-type recognition but is computationally prohibitive: FP16 inference requires 28–32 GB of VRAM, and even 4\-bit quantization consumes 12–14 GB\.

### 6\.5Toward v0\.8\.0: Full\-Page Vision Extraction

The v0\.7\.3 pipeline is limited to Table Transformer\-detected regions, missing maps, photographs, and conceptual diagrams that constitute a large fraction of disaster document visual information\. Preliminary experiments with LLaVA\-7B for full\-page visual element detection yield 104 elements across 32 sample pages \(3\.25/page\), including graphs, photos, and maps not captured by Table Transformer—but at 10 seconds/page versus 0\.5 seconds/page, a 20×\\timesthroughput cost\. A hybrid approach \(Table Transformer for tables; Vision LLM for other figure types; YOLO/GroundingDINO for bounding\-box recovery\) is the recommended path forward\.

## 7Limitations

Despite strong retrieval performance, Multimodal CoLRAG\-TF has four notable limitations\.

#### OCR noise\.

Even with Tesseract fallback, 13\.2% of table blocks fail OCR entirely\. Scanned documents from the 1950s–1980s with degraded typesetting remain outside the retrievable index, creating systematic blind spots for historical disaster data\.

#### Vision LLM accuracy\.

LLaVA\-7B correctly identifies the depicted disaster category in only 5 of 12 test images \(41\.7%\)\. Misclassification propagates to retrieval: a flood image described as “earthquake damage” returns thematically mismatched blocks despite high BM25/triple scores for generic disaster keywords\. Larger VLMs \(e\.g\., Qwen2\.5\-Omni\-7B, LLaVA\-NeXT\-13B\) would reduce this error but currently face deployment constraints \(GGUF conversion instability or 28–32 GB VRAM requirements on a 16 GB workstation\)\.

#### Domain\-specific vocabulary\.

The current BM25 tokenizer and embedding model are tuned for Japanese government disaster language\. Cross\-lingual extension \(e\.g\., FEMA or UN\-OCHA reports in English\) and adaptation to other specialized domains \(medical, legal\) would require re\-tokenization and domain\-specific model fine\-tuning\.

#### Multi\-hop gold annotation incompleteness\.

The 457\-pair benchmark annotates only the*primary*evidence block per multi\-hop question\. This causes Retrieval Recall to be artificially capped at 0\.5 for multi\-hop queries, even when semantically sufficient context is retrieved\. Future benchmarking should adopt soft relevance judgments \(e\.g\., via RAGAS\[[7](https://arxiv.org/html/2607.20517#bib.bib36)\]\) that credit partially overlapping evidence\.

## 8Conclusion

### 8\.1Concluding Remarks

We presentMultimodal CoLRAG\-TF, a retrieval\-augmented generation system for heterogeneous, multimodal PDF document collections\. The system integrates a hybrid OCR pipeline, an LLM\-based knowledge\-graph triple extractor, and a HippoRAG2\-style coarse\-to\-fine retriever into a four\-axis fusion architecture that jointly exploits dense embeddings, BM25 keyword search, triple filtering, and image similarity\. While demonstrated on Japanese disaster lesson documents, the architecture is general and applicable to any domain with comparable PDF characteristics\.

On a 457\-pair QA benchmark over 43 Japanese disaster PDFs:

- •Overall Retrieval Recall reaches0\.9909with the full four\-axis system\.
- •Multi\-hop Answer Similarity \(0\.56050\.5605\) exceeds single\-hop \(0\.32670\.3267\) by71\.6%, validating triple\-augmented compositional reasoning\.
- •Triple filtering achieves peak effectiveness of0\.9265for numeric fact queries, surpassing dense embedding search\.
- •Bayesian optimization confirms that triple weight must be dominant \(αtriple=0\.44\\alpha\_\{\\text\{triple\}\}\\\!=\\\!0\.44\) to overcome lexical retrieval bias in terminologically dense domains\.

The image\-based query interface demonstrates the feasibility of vision\-to\-knowledge\-retrieval workflows, with a mean Top\-1 score of 0\.6556 across 12 disaster images, though disaster\-type recognition \(41\.7%\) remains a critical bottleneck\.

### 8\.2Future Work

Future work will address: \(i\) full visual extraction beyond tables using hybrid Table Transformer \+ Vision LLM pipelines with bounding\-box recovery via GroundingDINO; \(ii\) 4\-bit quantized VLMs for edge deployment on emergency\-response mobile units; \(iii\) cross\-lingual extension to English disaster reports from FEMA and UN\-OCHA; \(iv\) online calibration of fusion weights as new disaster events are indexed; and \(v\) expansion of the disaster lesson corpus to under\-represented hazards—such as winter snowstorm countermeasures, large\-scale forest fire response, and urban fire management—to broaden the applicability of multimodal retrieval as a shared knowledge resource for emergency planning\. Given the rapid worldwide increase in severe wildfires, domain\-specific multimodal RAG systems could play a critical role in extracting operational lessons from rare but high\-impact events\.

### Acknowledgments

The disaster lesson document corpus is sourced from publicly available MLIT River and Civil Engineering publications\. Ollama inference infrastructure was used for local LLM serving\.

### Reproducibility

The implementation of Multi\-modal CoLRAG\-TF—including multimodal document processing \(layout analysis, hybrid OCR, caption generation\), triple extraction and FAISS\-based knowledge\-graph indexing, hierarchical triple filtering retrieval, BM25 keyword fusion, and full evaluation scripts—is released under the Apache License 2\.0\. The source code, experiment configurations, and documentation for reproducing all results reported in this paper are available at:[https://github\.com/tk\-yasuno/multimodal\_colragtf](https://github.com/tk-yasuno/multimodal_colragtf)\. The repository contains the v0\.7\.4 pipelines, and image\-based query demo used in our experiments\.

## References

- \[1\]T\. Akiba, S\. Sano, T\. Yanase,et al\.\(2019\)Optuna: a next\-generation hyperparameter optimization framework\.InProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,pp\. 2623–2631\.Cited by:[§3\.6](https://arxiv.org/html/2607.20517#S3.SS6.SSS0.Px2.p1.5)\.
- \[2\]F\. Alam, H\. Sajjad, M\. Imran, and F\. Ofli\(2021\)Humaid: human\-annotated disaster incidents data from twitter with deep learning benchmarks\.InProceedings of the International AAAI Conference on Web and Social Media \(ICWSM\),Vol\.15,pp\. 933–944\.Cited by:[§2\.3](https://arxiv.org/html/2607.20517#S2.SS3.p1.1)\.
- \[3\]Artifex Software\(2023\)PyMuPDF: python bindings for MuPDF\.Note:Software libraryVersion 1\.23\.x,[https://pymupdf\.readthedocs\.io/](https://pymupdf.readthedocs.io/)Cited by:[§2\.4](https://arxiv.org/html/2607.20517#S2.SS4.p1.1)\.
- \[4\]J\. Bergstra, R\. Bardenet, Y\. Bengio, and B\. Kégl\(2011\)Algorithms for hyper\-parameter optimization\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.24\.Cited by:[§3\.6](https://arxiv.org/html/2607.20517#S3.SS6.SSS0.Px2.p1.5)\.
- \[5\]W\. Chen, H\. Hu, X\. Chen,et al\.\(2022\)MuRAG: multimodal retrieval\-augmented generator for open question answering over images and text\.arXiv preprint arXiv:2210\.02928\.Cited by:[§2\.3](https://arxiv.org/html/2607.20517#S2.SS3.p1.1)\.
- \[6\]D\. Edge, H\. Trinh, N\. Cheng,et al\.\(2024\)From local to global: a graph rag approach to query\-focused summarization\.arXiv preprint arXiv:2404\.16130\.Cited by:[§2\.2](https://arxiv.org/html/2607.20517#S2.SS2.p1.1)\.
- \[7\]S\. Es, J\. James, L\. E\. Anke, and S\. Schockaert\(2023\)RAGAS: automated evaluation of retrieval augmented generation\.arXiv preprint arXiv:2309\.15217\.Cited by:[§4\.2](https://arxiv.org/html/2607.20517#S4.SS2.p1.1),[§6\.2](https://arxiv.org/html/2607.20517#S6.SS2.p1.1),[§7](https://arxiv.org/html/2607.20517#S7.SS0.SSS0.Px4.p1.1)\.
- \[8\]Y\. Gao, Y\. Xiong, X\. Gao,et al\.\(2024\)Retrieval\-augmented generation for large language models: a survey\.arXiv preprint arXiv:2312\.10997\.Cited by:[§2\.1](https://arxiv.org/html/2607.20517#S2.SS1.p1.1)\.
- \[9\]B\. J\. Gutierrez, Y\. Shu, Y\. Gu,et al\.\(2024\)HippoRAG: neurobiologically inspired long\-term memory for large language models\.arXiv preprint arXiv:2405\.14831\.Cited by:[§2\.2](https://arxiv.org/html/2607.20517#S2.SS2.p1.1)\.
- \[10\]B\. J\. Gutiérrez, Y\. Shu, W\. Qi, S\. Zhou, and Y\. Su\(2025\)From RAG to memory: non\-parametric continual learning for large language models\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\),Note:arXiv:2502\.14802Cited by:[§2\.2](https://arxiv.org/html/2607.20517#S2.SS2.p1.1),[§3\.5](https://arxiv.org/html/2607.20517#S3.SS5.p1.2)\.
- \[11\]hotchpotch\(2025\)Static\-embedding\-japanese: a fast CPU\-friendly japanese static embedding model\.Note:Hugging Face model repository[https://huggingface\.co/hotchpotch/static\-embedding\-japanese](https://huggingface.co/hotchpotch/static-embedding-japanese)Cited by:[§3\.3](https://arxiv.org/html/2607.20517#S3.SS3.SSS0.Px1.p1.1)\.
- \[12\]Y\. Huang, T\. Lv, L\. Cui,et al\.\(2022\)LayoutLMv3: pre\-training for document ai with unified text and image masking\.InProceedings of the 30th ACM International Conference on Multimedia,pp\. 4083–4091\.Cited by:[§2\.3](https://arxiv.org/html/2607.20517#S2.SS3.p1.1)\.
- \[13\]M\. Imran, F\. Ofli, D\. Caragea, and A\. Torralba\(2020\)Using artificial intelligence and social media multimodal content for disaster response and management: opportunities, challenges, and future directions\.Information Processing & Management57\(5\),pp\. 102261\.Cited by:[§2\.3](https://arxiv.org/html/2607.20517#S2.SS3.p1.1)\.
- \[14\]J\. Johnson, M\. Douze, and H\. Jégou\(2021\)Billion\-scale similarity search with GPUs\.IEEE Transactions on Big Data7\(3\),pp\. 535–547\.Cited by:[§3\.3](https://arxiv.org/html/2607.20517#S3.SS3.SSS0.Px1.p1.1)\.
- \[15\]O\. Khattab and M\. Zaharia\(2020\)ColBERT: efficient and effective passage search via contextualized late interaction over bert\.InProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 39–48\.Cited by:[§2\.1](https://arxiv.org/html/2607.20517#S2.SS1.p1.1)\.
- \[16\]P\. Lewis, E\. Perez, A\. Piktus,et al\.\(2020\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.33,pp\. 9459–9474\.Cited by:[§1\.1](https://arxiv.org/html/2607.20517#S1.SS1.p3.1),[§2\.1](https://arxiv.org/html/2607.20517#S2.SS1.p1.1)\.
- \[17\]H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee\(2023\)Visual instruction tuning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.36\.Cited by:[§3\.7](https://arxiv.org/html/2607.20517#S3.SS7.SSS0.Px1.p1.1)\.
- \[18\]Ministry of Land, Infrastructure, Transport and Tourism \(MLIT\), Japan\(2024\)Disaster prevention technology basic plan\.Technical reportMinistry of Land, Infrastructure, Transport and Tourism \(MLIT\), River and Civil Engineering Division\.Note:Available at:[https://www\.mlit\.go\.jp/](https://www.mlit.go.jp/)Cited by:[§1\.1](https://arxiv.org/html/2607.20517#S1.SS1.p2.1),[§4\.1](https://arxiv.org/html/2607.20517#S4.SS1.p2.1)\.
- \[19\]Qwen Team\(2024\)Qwen2\.5: a party of foundation models\.arXiv preprint arXiv:2412\.15115\.Cited by:[§3\.2](https://arxiv.org/html/2607.20517#S3.SS2.SSS0.Px3.p1.1)\.
- \[20\]N\. Reimers and I\. Gurevych\(2019\)Sentence\-BERT: sentence embeddings using siamese BERT\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 3982–3992\.Cited by:[§2\.1](https://arxiv.org/html/2607.20517#S2.SS1.p1.1)\.
- \[21\]S\. Robertson and H\. Zaragoza\(2009\)The probabilistic relevance framework: BM25 and beyond\.Technical reportTechnical Report4, Vol\.3,Foundations and Trends in Information Retrieval\.Cited by:[§2\.1](https://arxiv.org/html/2607.20517#S2.SS1.p1.1),[§3\.3](https://arxiv.org/html/2607.20517#S3.SS3.SSS0.Px2.p1.2)\.
- \[22\]K\. Santhanam, O\. Khattab, J\. Saad\-Falcon,et al\.\(2022\)ColBERTv2: effective and efficient retrieval via lightweight late interaction\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\),pp\. 3715–3734\.Cited by:[§2\.1](https://arxiv.org/html/2607.20517#S2.SS1.p1.1)\.
- \[23\]R\. Smith\(2007\)An overview of the tesseract OCR engine\.InProceedings of the 9th International Conference on Document Analysis and Recognition \(ICDAR\),pp\. 629–633\.Cited by:[§2\.4](https://arxiv.org/html/2607.20517#S2.SS4.p1.1),[§3\.2](https://arxiv.org/html/2607.20517#S3.SS2.SSS0.Px2.p1.1)\.
- \[24\]B\. Smock, R\. Pesala, and R\. Abraham\(2022\)PubTables\-1m: towards comprehensive table extraction from unstructured documents\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 4634–4642\.Cited by:[S3\. Table Extraction Examples on Layout Analysis](https://arxiv.org/html/2607.20517#Ax1.SSx3.p1.1),[§2\.3](https://arxiv.org/html/2607.20517#S2.SS3.p1.1),[§3\.2](https://arxiv.org/html/2607.20517#S3.SS2.SSS0.Px1.p1.1)\.
- \[25\]L\. Zheng, W\. Chiang, Y\. Sheng,et al\.\(2023\)Judging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.arXiv preprint arXiv:2306\.05685\.Cited by:[§4\.2](https://arxiv.org/html/2607.20517#S4.SS2.p1.1)\.

## Appendix AImplementation Details

### A1\. OCR Pipeline Configuration

Table Transformer is run with confidence threshold≥0\.7\\geq 0\.7using themicrosoft/table\-transformer\-detectioncheckpoint\. Each PDF page is rasterized at 150 DPI before detection\. PyMuPDF \(fitz\) extracts embedded text in Stage 1\. Tesseract OCR \(v5\.x\) is invoked with\-l jpnand\-\-psm 6\(uniform block of text\) for Stage 2 fallback\. PaddlePaddle was evaluated but excluded due to GPU driver conflicts with the co\-deployed PyTorch/Ollama environment\.

### A2\. Table Transformer Parameters

Detection model:microsoft/table\-transformer\-detection\(DETR backbone, ResNet\-50 encoder\)\. Confidence threshold: 0\.7\. Table structure recognition model:microsoft/table\-transformer\-structure\-recognition\-v1\.1\-all\. Input resolution:800×800800\\times 800pixels \(before detection resize\)\. Batch size: 1 \(per\-page processing\)\.

### A3\. Embedding Model Details

Model:hotchpotch/static\-embedding\-japanese\(1024\-dimensional\)\. Encoding uses mean pooling over all token embeddings\. Vectors are L2\-normalized before FAISS insertion\. Maximum input length: 512 tokens; text exceeding this limit is truncated\. Inference runs on CPU \(embedding model is lightweight:<<100 MB\)\.

### A4\. FAISS Index Configuration

Index type:IndexFlatIP\(exact inner\-product search\)\. Block index: 2,403 vectors×\\times1024 dimensions \(≈\\approx9\.9 MB\)\. Triple index: 11,414 vectors×\\times1024 dimensions \(≈\\approx46\.8 MB\)\. Search: top\-kkexact nearest neighbor,k∈\{5,20\}k\\in\\\{5,20\\\}depending on retrieval stage\. All FAISS operations run on CPU\.

### A5\. GPU Memory and Inference Time

All LLM inference \(caption generation, triple extraction, answer synthesis\) runs via Ollama on an NVIDIA RTX 4060 Ti \(16 GB VRAM\)\.

- •Caption generation\(Qwen2\.5\-7B\-Instruct\-Q4\_K\_M\):≈\\approx2–4 s/block\.
- •Triple extraction\(Qwen2\.5\-7B\-Instruct\-Q4\_K\_M\):≈\\approx3–5 s/block\.
- •Answer synthesis\(Qwen2\.5\-7B\-Instruct\-Q4\_K\_M\):≈\\approx2–6 s/query\.
- •Image understanding\(LLaVA\-7B\-Q4\):≈\\approx5–10 s/image\.

Full index build time over 2,403 blocks:≈\\approx6 hours on the above hardware\. Online retrieval latency \(end\-to\-end, excluding LLM answer generation\):<<1 s\.

### A6\. OpenIE Prompt Template

The following prompt is sent to Qwen2\.5\-7B\-Instruct for each block:

> Extract factual triples from the following disaster lesson text\. Prioritize: definitions, causal relations, temporal sequences, quantitative facts\. Output: JSON array of\{“subject”, “relation”, “object”\}\. Target 5–15 triples per block\. Respond with JSON only\.

Processing is performed with GROUP\_SIZE=1\\\!=\\\!1\(one block per request\) to maximize chunk granularity and triple quality\. Blocks that return malformed JSON are skipped \(14% failure rate\)\.

### A7\. Optuna Configuration

Optimizer: Optuna v3\.x with TPE \(Tree\-structured Parzen Estimator\) sampler\. Number of trials: 50\. Search space:αt∈\[0\.1,0\.6\]\\alpha\_\{t\}\\in\[0\.1,0\.6\],αb∈\[0\.1,0\.5\]\\alpha\_\{b\}\\in\[0\.1,0\.5\],αt​r∈\[0\.1,0\.6\]\\alpha\_\{tr\}\\in\[0\.1,0\.6\],αi=0\.10\\alpha\_\{i\}=0\.10\(fixed\)\. Constraint:αt\+αb\+αt​r\+αi=1\.0\\alpha\_\{t\}\+\\alpha\_\{b\}\+\\alpha\_\{tr\}\+\\alpha\_\{i\}=1\.0\(enforced by normalizing the first three parameters after sampling\)\. Objective:ℒ=−\(0\.5⋅P​@​5\+0\.2⋅D\+0\.3⋅Q\)\\mathcal\{L\}=\-\(0\.5\\cdot P@5\+0\.2\\cdot D\+0\.3\\cdot Q\)evaluated on 40 gold\-standard blocks\. Convergence: best score plateau reached at trial 14 \(objective=0\.41=0\.41\)\.

## Supplementary Materials

The following supplementary materials provide qualitative examples and baseline results to complement the quantitative analysis in the main text\.

### S1\. Text Query Multimodal Qualitative Analysis

Tables[S1](https://arxiv.org/html/2607.20517#Ax1.T1)–[S2](https://arxiv.org/html/2607.20517#Ax1.T2)show two representative text\-query examples evaluated with the CoLRAG\-TF v0\.7\.3 system\. Each example reports the input question, retrieval performance \(Top\-1 score, Triple score\), the model\-generated prediction, the ground\-truth reference answer, and an overall performance rating\.

Table S1:S1\-B: Multi\-hop compositional query – Cross\-event recovery comparison \(table\-type blocks\)\.Table S2:S1\-C: Table\-targeted numeric query – Destroyed buildings count \(triple\-effective\)\.
### S2\. Image Query Multimodal Qualitative Analysis

Tables[S3](https://arxiv.org/html/2607.20517#Ax1.T3)–[S4](https://arxiv.org/html/2607.20517#Ax1.T4)illustrate two image\-based query examples from the 12\-image test set\. Each example shows the disaster image description \(as generated by LLaVA\-7B\), retrieval performance, the synthesized lesson prediction, and ground truth\.

Table S3:S2\-A: Image query – 2020 Kumamoto floods \(flood event, top\-1 = 0\.765\)\.Table S4:S2\-B: Image query – Kumamoto earthquake building collapse \(best performance, top\-1 = 0\.765\)\.
### S3\. Table Extraction Examples on Layout Analysis

Figures[1\(a\)](https://arxiv.org/html/2607.20517#Ax1.F1.sf1)and[1\(b\)](https://arxiv.org/html/2607.20517#Ax1.F1.sf2)show two representative table regions automatically detected by Table Transformer\[[24](https://arxiv.org/html/2607.20517#bib.bib16)\]\(confidence threshold≥0\.7\\geq 0\.7\) and cropped from source disaster PDFs\. These blocks form part of the 2,403 multimodal index entries; each is processed through the hybrid OCR pipeline and captioned by Qwen2\.5\-7B\-Instruct before ingestion into the retrieval system\.

![Refer to caption](https://arxiv.org/html/2607.20517v1/figures/suppS3_table_kumamoto_H28_p2.png)\(a\)Heisei Disaster Cases – 2016 Kumamoto earthquake \(H28\), page 2\. Damage statistics by prefecture \(buildings, agricultural, roads\)\. Source:Heisei\-no\-Saigai\-Jirei, H28 Kumamoto\.
![Refer to caption](https://arxiv.org/html/2607.20517v1/figures/suppS3_table_tohoku_2011_p3.png)\(b\)Great Earthquake Cases – 2011 Great East Japan Earthquake, page 3\. Summary damage table: fatalities \(15,900\), missing \(2,523\), buildings destroyed \(121,995\), estimated damage \(16\.9 trillion yen\)\. Source:Daijishin\-no\-Jirei, Higashinihon 2011\.

Figure S1:S3: Two representative table blocks extracted by Table Transformer from disaster lesson PDFs\. Each block undergoes hybrid OCR \(PyMuPDF then Tesseract fallback\) and LLM\-based caption generation before triple extraction and multimodal indexing\.
### S4\. Naive RAG Baseline Results

Table[S5](https://arxiv.org/html/2607.20517#Ax1.T5)reports performance of the Naive RAG baseline \(flat dense embedding retrieval with no hierarchy, no triple filtering, and no BM25 keyword matching\), included for completeness alongside the two\-axis CoLRAG v0\.7\.2 and the proposed four\-axis CoLRAG\-TF v0\.7\.3\.

#### Baseline configuration\.

The Naive RAG baseline performs flatIndexFlatIPsearch over all 2,403 multimodal blocks using the samehotchpotch/static\-embedding\-japaneseembedding model \(1024\-dim\)\. No volume/chapter hierarchy is applied; all blocks are treated as a single flat pool\. Top\-5 blocks are retrieved and passed directly to Qwen2\.5\-7B\-Instruct for answer generation without reranking\.

Table S5:S4: Naive RAG baseline vs\. CoLRAG v0\.7\.2 vs\. CoLRAG\-TF v0\.7\.3 on the 457\-pair QA benchmark\.
#### Observations\.

1. 1\.Naive RAG vs\. CoLRAG: Overall Answer Similarity improves from 0\.381 to 0\.473 \(\+24%\+24\\%\) when the three\-tier hierarchical retriever and triple filtering are added, confirming that flat embedding search is insufficient for this domain\.
2. 2\.Multi\-hop gap: The Naive RAG multi\-hop advantage over 1\-hop is\+53\.5%\+53\.5\\%, compared to\+71\.6%\+71\.6\\%for CoLRAG\-TF—a18\.118\.1percentage\-point widening attributable to triple\-filtered cross\-document reasoning\.
3. 3\.Retrieval Recall: Naive RAG achieves only 0\.72 overall Retrieval Recall@5 \(vs\. 0\.99 for CoLRAG\-TF\), reflecting the inability of flat search to localize relevant blocks across the 2,403\-block corpus without hierarchical narrowing\.
4. 4\.Answer length: Naive RAG generates shorter answers \(361 chars\), suggesting less retrieved context is usable compared to the hierarchically filtered top\-5 blocks \(CoLRAG: 419–428 chars\)\.

These results confirm that both the hierarchical coarse\-to\-fine retriever and the triple\-filtering mechanism contribute independently to the performance gains observed in the proposed system\.