LFRAG: Layout-oriented Fine-grained Retrieval-Augmented Generation on Multimodal Document Understanding

arXiv cs.AI Papers
retrieval-augmented-generation multimodal document-understanding layout-segmentation fine-grained-retrieval benchmark

Summary

LFRAG proposes a layout-oriented fine-grained retrieval-augmented generation framework that moves from page-level to block-level retrieval in multimodal documents, achieving state-of-the-art performance and 73% token reduction on the new LFDocQA benchmark.

arXiv:2605.22829v1 Announce Type: cross Abstract: Multimodal Retrieval-Augmented Generation (RAG) has emerged as an effective paradigm for enhancing Large Language Models (LLMs) with external knowledge. However, existing multimodal RAG systems predominantly rely on coarse-grained page-level retrieval, which fails to capture fine-grained semantic and layout structures in visually rich documents, thereby compromising retrieval accuracy and leading to redundant context in downstream tasks. To address these issues, we propose Layout-oriented Fine-grained Retrieval-Augmented Generation (LFRAG), a novel framework that advances multimodal RAG from page-level to block-level retrieval. We perform layout segmentation to construct semantically coherent fine-grained retrieval units and design a semantic-layout fusion encoder that integrates local semantics with global context via cross-attention. With block-level late interaction retrieval, LFRAG enables precise query-content alignment and reduces irrelevant content for downstream generation. To enable rigorous evaluation, we construct LFDocQA, a large-scale benchmark with block-level annotations spanning diverse document types, designed to assess both multimodal document retrieval and question answering with greater granularity than existing datasets. Extensive experiments on LFDocQA demonstrate that LFRAG achieves state-of-the-art performance on retrieval tasks, outperforms the best baseline by 7.20% in answer accuracy, and reduces token consumption by 73.07% in generation tasks, confirming LFRAG as an accurate and efficient framework for multimodal RAG over visually rich documents. Our code and datasets will be released soon.
Original Article
View Cached Full Text

Cached at: 05/25/26, 09:00 AM 

# LFRAG: Layout-oriented Fine-grained Retrieval-Augmented Generation on Multimodal Document Understanding
Source: [https://arxiv.org/abs/2605.22829](https://arxiv.org/abs/2605.22829)
[View PDF](https://arxiv.org/pdf/2605.22829)

> Abstract:Multimodal Retrieval\-Augmented Generation \(RAG\) has emerged as an effective paradigm for enhancing Large Language Models \(LLMs\) with external knowledge\. However, existing multimodal RAG systems predominantly rely on coarse\-grained page\-level retrieval, which fails to capture fine\-grained semantic and layout structures in visually rich documents, thereby compromising retrieval accuracy and leading to redundant context in downstream tasks\. To address these issues, we propose Layout\-oriented Fine\-grained Retrieval\-Augmented Generation \(LFRAG\), a novel framework that advances multimodal RAG from page\-level to block\-level retrieval\. We perform layout segmentation to construct semantically coherent fine\-grained retrieval units and design a semantic\-layout fusion encoder that integrates local semantics with global context via cross\-attention\. With block\-level late interaction retrieval, LFRAG enables precise query\-content alignment and reduces irrelevant content for downstream generation\. To enable rigorous evaluation, we construct LFDocQA, a large\-scale benchmark with block\-level annotations spanning diverse document types, designed to assess both multimodal document retrieval and question answering with greater granularity than existing datasets\. Extensive experiments on LFDocQA demonstrate that LFRAG achieves state\-of\-the\-art performance on retrieval tasks, outperforms the best baseline by 7\.20% in answer accuracy, and reduces token consumption by 73\.07% in generation tasks, confirming LFRAG as an accurate and efficient framework for multimodal RAG over visually rich documents\. Our code and datasets will be released soon\.

## Submission history

From: Yifan Zhu \[[view email](https://arxiv.org/show-email/c8c7d9b4/2605.22829)\] **\[v1\]**Sat, 18 Apr 2026 05:04:49 UTC \(1,038 KB\)

Similar Articles

LightRAG: Simple and Fast Retrieval-Augmented Generation

Papers with Code Trending

The article introduces LightRAG, an open-source framework that enhances Retrieval-Augmented Generation by integrating graph structures for improved contextual awareness and efficient information retrieval.

Fine-grained Fragment Retrieval in Multi-modal Long-form Dialogues

arXiv cs.CL

This paper introduces Fine-grained Fragment Retrieval (FFR), a new task for locating semantically coherent multi-modal fragments (text and images) within long-form dialogues. The authors propose F2RVLM, a generation-based retrieval model trained with reinforcement learning, and FFRS, a two-stage retrieval system, along with a new dataset MLDR for evaluation.

Multimodal CoLRAG-TF: Triple-Filtered Retrieval for Complex PDFs

arXiv cs.LG

Introduces Multimodal CoLRAG-TF, a four-axis fusion architecture for retrieval-augmented generation over complex PDFs, integrating dense text, BM25, knowledge-graph triple filtering, and image similarity. Achieves significant improvements in multi-hop reasoning on a benchmark of Japanese disaster lesson PDFs.