Disco-RAG: Discourse-Aware Retrieval-Augmented Generation

# Disco-RAG: Discourse-Aware Retrieval-Augmented Generation

Source: https://arxiv.org/html/2601.04377

Dongqi Liu^Ω,Θ, Hang Ding^Δ, Qiming Feng^Γ, Xurong Xie^Ψ, Zhucun Xue^Ψ, Chengjie Wang^Θ, Jian Li^Θ, Jiangning Zhang^Θ, Yabiao Wang^Ψ,Θ

^Ω Saarland University, ^Δ Shanghai Jiaotong University, ^Γ Fudan University, ^Ψ Zhejiang University, ^Θ Tencent YouTu Lab

###### Abstract

Retrieval-Augmented Generation (RAG) has emerged as an important means of enhancing the performance of large language models (LLMs) in knowledge-intensive tasks. However, most existing RAG strategies treat retrieved passages in a flat and unstructured way, which prevents the model from capturing structural cues and constrains its ability to synthesize knowledge from dispersed evidence across documents. To overcome these limitations, we propose Disco-RAG, a discourse-aware framework that explicitly injects discourse signals into the generation process. Our method constructs intra-chunk discourse trees to capture local hierarchies and builds inter-chunk rhetorical graphs to model cross-passage coherence. These structures are jointly integrated into a planning blueprint that conditions the generation. Experiments on question answering and long-document summarization benchmarks show the efficacy of our approach. Disco-RAG achieves state-of-the-art results on the benchmarks without fine-tuning. These findings underscore the important role of discourse structure in advancing RAG systems. The project information is available at https://dongqi.me/projects/Disco-RAG.

## 1 Introduction

The advent of large language models (LLMs; Touvrov et al. 2023, Yang et al. 2025, Achiam et al. 2023) has advanced research progress in natural language processing (NLP), achieving competitive performance across a wide range of tasks, including question answering, document summarization, and text generation. However, due to the reliance on static training corpora, LLMs can be inadequate for knowledge-intensive scenarios, such as handling domain-specific knowledge, proprietary data, or information requiring real-time updates. Retrieval-Augmented Generation (RAG) has been proposed as a suitable strategy by integrating an external knowledge component through retrieval-based mechanisms.

![Figure 1: Comparison between standard RAG and Disco-RAG. While standard RAG retrieves isolated chunks without structural links, Disco-RAG organizes evidence into discourse structures (trees & graphs). Here, S denotes Satellite (the supplementary part), and N denotes Nucleus (the core part).](https://arxiv.org/html/2601.04377#S1.F1)

In standard RAG pipelines, external documents are segmented into chunks, which are then encoded into vectors and stored in a database. At query time, relevant chunks are retrieved to provide contextual grounding for the LLM. One important but insufficiently addressed limitation of existing RAG systems concerns the mismatch between retrieval granularity and generative understanding. While retrieval modules return relevant chunks, these chunks are often fragmented in discourse, resembling scattered pieces of evidence without clear logical connections. This manifests at two levels. First, *intra-chunk structural blindness*: within each chunk, RAG systems may fail to capture internal discourse. Second, *inter-chunk coherence gaps*: across multiple chunks, RAG systems may struggle to identify rhetorical connections.

As depicted in Figure 1 (left), Chunk A mentions a 12% lower incidence, while Chunk B notes no significant overall effect. Without recognizing that the former is a conditional finding (e.g., among deficient adults in winter), standard RAG tends to overgeneralize and incorrectly conclude that vitamin D reduces flu risk. These deficiencies prevent effective resolution of conflicting claims, as standard RAG approaches lack the capacity to organize retrieved evidence through a higher-level causal flow. This leaves the final LLM generator to grapple with a "bag of facts" rather than a coherent line of reasoning.

Recent investigations have revealed that integrating discourse knowledge into LLMs can improve downstream performance and alleviate hallucinations. These findings highlight the drawback of relying solely on flat sequential representations and underline the benefits of discourse for context engineering. Building on these insights, the present work aims to investigate whether explicitly modeling and providing discourse information to the LLM can improve generation quality in the context of RAG.

To answer this, we propose Disco-RAG, a framework that constructs local discourse trees for each retrieved chunk and infers inter-chunk coherence relations across chunks to form a rhetorical graph. To synthesize information, rather than merely concatenating it, the text generator needs not only to understand the relations between evidence but also to strategize how to present them. This requires a high-level plan to orchestrate the narrative flow. We thus introduce a discourse-aware planning module that enables the model to dynamically generate a plan to guide the generation.

As shown in Figure 1 (right), the discourse-aware process enables the model to infer that vitamin D is not broadly effective but may benefit deficient adults under specific conditions, producing more faithful answers and aligning with the underlying evidence.

In our experiments, we evaluate Disco-RAG on three benchmarks: Loong, ASQA, and SciNews. Consistent improvements are observed compared with standard RAG systems and state-of-the-art (SOTA) methods. On the Loong benchmark, our approach delivers an overall gain of 12.74 points in LLM Score. On the ASQA dataset, our method exceeds the best existing systems on Exact Match and ROUGE-L Score by clear margins. On the SciNews benchmark, Disco-RAG establishes new SOTA performance across most evaluation metrics.

In summary, our contributions are as follows:

- We present Disco-RAG, an inference-time strategy that explicitly injects discourse knowledge into the RAG pipeline to alleviate the discrepancy between chunk-level evidence and discourse-level reasoning.
- We propose a modeling method that combines intra-chunk discourse trees, inter-chunk rhetorical graphs, and discourse-driven plans to capture local hierarchies, cross-passage coherence, and argumentative flow.
- We conduct experiments on knowledge-intensive QA and summarization tasks, demonstrating consistent gains over strong RAG baselines. Analysis studies further confirm the efficacy of discourse-aware guidance in enhancing generation correctness, coherence, and factuality.

## 2 Related Work

### 2.1 Structure-Aware Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) enhances LLMs in knowledge-intensive tasks by retrieving external evidence. However, conventional RAG methods typically treat retrieved chunks as isolated and flat sequences, overlooking their structural interconnections. To mitigate this, recent research has explored structure-aware variants of RAG. Graph-based methods such as GraphRAG and KG-RAG organize evidence into knowledge graphs, while subsequent work has improved retrieval by simulating human memory mechanisms or enriching graph semantics. Other approaches construct structured subgraphs for coherence, or employ alternative formats like hierarchical graphs, semantic chunking, trees, and tables. More adaptive strategies dynamically select structures based on context.

Despite these advances, most efforts emphasize surface-level associations (e.g., linking entities) while largely overlooking the rhetorical structure that governs causal flow, evidence presentation, and conclusion formulation. This hinders logical depth and discourse coherence, which our work seeks to address.

### 2.2 Rhetorical Structure Theory for Text Generation

Rhetorical Structure Theory (RST; Mann and Thompson 1987, 1988) is a discourse framework that models hierarchical dependencies and rhetorical relations among Elementary Discourse Units (EDUs). It distinguishes between *nucleus* and *satellite* units, connected by relations such as Elaboration, Causality, and Contrast, forming tree structures that reflect communicative intent. Foundational work has established strong correlations between rhetorical structure and human text planning. Later studies have leveraged RST by converting trees into dependency graphs or imposing structural constraints to improve coherence and consistency in neural generation models. More recent efforts have integrated RST into LLMs to improve cross-sentence reasoning and enhance both structural integrity and interpretability of generated outputs.

Compared with shallow discourse markers or sentence-level connectives, the present work extends RST modeling to the RAG setting by explicitly encoding the deeper structure of retrieved passages and highlighting the importance of hierarchical structure.

## 3 Proposed Method

![Figure 2: The Disco-RAG pipeline: Starting from passage retrieval (providing context), then intra-chunk RST tree parsing (capturing local discourse), inter-chunk rhetorical graph construction (modeling global discourse), rhetorical planning (blueprint generation), and answer generation (producing the final output).](https://arxiv.org/html/2601.04377#S3.F2)

#### Method Overview

We formalize standard RAG as a conditional generation problem. Given a query *q* and a set of Top-*k* retrieved chunks $\mathcal{C}(q;\mathcal{D})=\{c_1,c_2,\ldots,c_k\}$ from a corpus $\mathcal{D}$, the output is $y=\arg\max_{y'} P(y'|q,\mathcal{C}(q;\mathcal{D}))$, where $P(\cdot)$ denotes the conditional distribution of the answer generator.

To overcome the limitations of the retrieval-and-concatenation paradigm, we propose Disco-RAG to augment standard RAG with rhetorical parsing and discourse-aware planning. As illustrated in Figure 2, our pipeline consists of three main stages. (1) We delve into each chunk $c_i$ to uncover its internal logical hierarchy by constructing an intra-chunk RST tree $t_i$. (2) We zoom out to map the relational landscape across all chunks $\mathcal{C}$ via an inter-chunk rhetorical graph $\mathcal{G}$. (3) We apply a discourse-driven planning module that devises a blueprint $\mathcal{B}$ based on $\mathcal{T}=\{t_i\}_{i=1}^k$ and $\mathcal{G}$ to guide the final generation process.

We hypothesize that under identical retriever and decoding conditions, explicitly injecting discourse knowledge improves the correctness, coherence, and factual consistency of generated text. Here, rhetorical modeling serves as a *knowledge-level prior*, while