Rethinking the Necessity of Adaptive Retrieval-Augmented Generation through the Lens of Adaptive Listwise Ranking

# Rethinking the Necessity of Adaptive Retrieval-Augmented Generation through the Lens of Adaptive Listwise Ranking
Source: https://arxiv.org/html/2604.15621
Jun Feng1, Jiahui Tang2, Zhicheng He2, Hang Lv2, Hongchao Gu2, Hao Wang2, Xuezhi Yang1, Shuai Fang1 1Hefei University of Technology, 2University of Science and Technology of China

###### Abstract

Adaptive Retrieval-Augmented Generation aims to mitigate the interference of extraneous noise by dynamically determining the necessity of retrieving supplementary passages. However, as Large Language Models evolve with increasing robustness to noise, the necessity of adaptive retrieval warrants re-evaluation. In this paper, we rethink this necessity and propose AdaRankLLM, a novel adaptive retrieval framework. To effectively verify the necessity of adaptive listwise reranking, we first develop an adaptive ranker employing a zero-shot prompt with a passage dropout mechanism, and compare its generation outcomes against static fixed-depth retrieval strategies. Furthermore, to endow smaller open-source LLMs with this precise listwise ranking and adaptive filtering capability, we introduce a two-stage progressive distillation paradigm enhanced by data sampling and augmentation techniques. Extensive experiments across three datasets and eight LLMs demonstrate that AdaRankLLM consistently achieves optimal performance in most scenarios with significantly reduced context overhead. Crucially, our analysis reveals a role shift in adaptive retrieval: it functions as a critical noise filter for weaker models to overcome their limitations, while serving as a cost-effective efficiency optimizer for stronger reasoning models. Our code is available at USTC-StarTeam/adaptive-listwise-ranking-rag (https://github.com/USTC-StarTeam/adaptive-listwise-ranking-rag).

## I. Introduction

Refer to captionFigure 1: The framework of AdaRankLLM. The left part shows two examples to demonstrate how the Adaptive Ranker works. The right part outlines the process of data collection and model training in order to distill an efficient ranker.

Retrieval-Augmented Generation (RAG) stands as a pivotal paradigm that augments Large Language Models (LLMs) by incorporating external knowledge[18,27,25,12]. By grounding generation in retrieved passages, RAG effectively mitigates hallucinations[53] and facilitates continuous knowledge updates[16]. However, its efficacy is not guaranteed; recent studies[34,37] show that indiscriminate retrieval often introduces extraneous noise. While standard RAG systems typically retrieve a fixed number of passages, this static strategy struggles to balance information coverage against noise intrusion: retrieving too few passages risks missing relevant information, while retrieving too many introduces noise that degrades model performance.

To address this dilemma, adaptive retrieval[2] has emerged as a dominant solution. By dynamically determining when and what to retrieve, it aims to shield models from irrelevant contexts and has historically yielded significant gains. However, the landscape of LLMs is rapidly evolving. Recent work[41] reveals that modern foundation models are becoming increasingly robust to retrieval noise, capable of ignoring irrelevant information via strong internal attention mechanisms. This evolution raises a fundamental question: Is adaptive retrieval still essential for these increasingly capable RAG systems?

To investigate this question, we propose AdaRankLLM, a framework that synergizes Adaptive Ranking with Instruction Distillation. Our approach hinges on a rigorous mechanism to determine retrieval necessity: an adaptive ranker equipped with a Passage Dropout mechanism. Utilizing a zero-shot listwise prompt, this module evaluates passage relevance, explicitly discarding irrelevant candidates or outputting a termination token when no passages are relevant. To ensure this verification capability is not confined to cost-prohibitive large models, we further integrate a two-stage progressive distillation paradigm. By leveraging clustering-based sampling and data augmentation, we transfer listwise ranking and adaptive filtering skills from teacher models to smaller open-source backbones, allowing us to verify the utility of adaptive retrieval across varying model capabilities and deployment constraints.

We conduct extensive experiments on three benchmarks across eight LLMs, spanning models from smaller open-source backbones to advanced reasoning-enhanced architectures. The results demonstrate that AdaRankLLM consistently achieves optimal performance in most scenarios while significantly reducing context overhead. Crucially, our analysis reveals a clear distinction: for weaker models, adaptive retrieval acts as a critical noise filter to ensure generation quality; for advanced models, while it offers limited performance gains, it serves as a vital efficiency optimizer by significantly reducing inference costs.

Our main contributions can be summarized as follows:

- We examine the necessity of adaptive RAG for modern LLMs, moving beyond the traditional view to evaluate its distinct utility across models with varying capabilities.
- We propose AdaRankLLM, a framework that employs an adaptive ranker with a passage dropout mechanism and instruction distillation to endow smaller models with precise noise-filtering capabilities.
- Extensive experiments across eight LLMs reveal a functional distinction: adaptive retrieval acts as a critical noise filter for weaker models to ensure quality, while serving as a vital efficiency optimizer for advanced models.

## II. Related Work

**Passage Ranking.** Traditional ranking methods mainly rely on pointwise[35] or pairwise[33] scorers, yet they often overlook passage interrelations, prompting the shift to LLM-based listwise approaches[28,22,24]. While early distillation attempts like RankVicuna[31] and RankZephyr[32] successfully transferred listwise capabilities to 7B models, they remained constrained by the quadratic complexity of decoder-only architectures. Recent advancements address these limitations: LiT5[40] reintroduces the Fusion-in-Decoder (FiD) architecture for linear scaling, while Rank-R1[55] integrates Chain-of-Thought (CoT) reasoning via Group Relative Policy Optimization (GRPO) to explicitly justify relevance. Further reducing latency, non-generative approaches utilize passage embeddings[23] or attention patterns[4] directly. Our method aligns with this trajectory, enhancing zero-shot listwise distillation with adaptive capabilities.

**Adaptive Retrieval.** Recent studies[34,37,13,20] indicate that RAG performance is easily compromised by extraneous information. Early approaches like FLARE[15] and Self-RAG[2] relied on token probabilities or learned reflection tokens, often resulting in high inference or fine-tuning costs. Recent research pivots toward internal state monitoring: CTRLA[21] and TARG[46] leverage "honesty" directions and prefix logits to gate retrieval without auxiliary classifiers. For complex reasoning, FAIR-RAG[3], REPAIR[17] and RAPID[8] introduce structured, plan-aware loops to trigger "mid-course" retrieval based on intermediate steps rather than the initial query. Additionally, cost-aware reinforcement learning strategies[10] dynamically adjust retrieval depth. However, their complexity hampers zero-shot generalization and deployment efficiency, motivating our lightweight framework.

## III. Method

### III-A. Problem Definition

Given a user query x and a set of candidate passages P={p₁,p₂,...,pₘ} retrieved from a massive corpus 𝒞 (where m is the number of candidates), the goal of Adaptive Retrieval-Augmented Generation is to identify an optimal ordered subset A⊆P. This process aims to maximize the relevance of the selected passages to x while filtering out noise. Crucially, the size of the result set |A| is dynamic and dependent on the query complexity. The set A can range from the empty set ∅ (when retrieval is unnecessary) to the full candidate set. Finally, the selected passages A and query x serve as input to the generator to produce the response y=LLM(A;x).

### III-B. Adaptive Ranker with Passage Dropout

To transform the coarse candidate set P into the refined adaptive set A, we propose an **Adaptive Ranker** that advances the paradigm of listwise reranking. Unlike conventional rerankers that merely permute the fixed candidate list, our module integrates a passage dropout mechanism to dynamically filter extraneous noise while capturing the interrelations among passages.

Specifically, we design a zero-shot conversational prompt to instruct the LLM to perform relevance assessment and selection simultaneously. Each passage in the candidate set P is assigned a unique identifier (e.g., '[1]'). Instead of forcing a full permutation, the LLM is directed to select only the passages that contribute to answering the query and rank them by relevance. Crucially, the passage dropout mechanism enables a rejection option: if the model determines that a passage (or even the entire set) is irrelevant, it explicitly excludes it from the output list A. In extreme cases where no relevant information is found, the model outputs a special termination token '[0]'. The specific prompt structure is illustrated in Figure 2. Following this adaptive ranking process, we parse the generated identifier list to construct the final context set A={pr₁,...,prₙ}, where the subset size n is dynamically determined by the model's relevance judgment.

### III-C. Instruction Distillation

While proprietary LLMs like GPT-4 demonstrate exceptional zero-shot adaptive ranking capabilities, their prohibitive costs hinder direct large-scale deployment. Consequently, we aim to distill these capabilities into a compact open-source model using a progressive two-stage paradigm to balance distillation cost and data quality.

**Stage 1: Structural Alignment.** We utilize GPT-3.5 to generate a large-scale dataset from MSMARCO V1. This cost-effective initialization teaches the student model the fundamental listwise output schema and basic relevance ordering, establishing a solid structural foundation beforehand.

**Stage 2: Adaptive Refinement.** To inject precise passage dropout capabilities, we rely on GPT-4's superior reasoning. To minimize costs, we apply K-means clustering to query embeddings to sample a representative subset. GPT-4 annotates these selected queries to fine-tune the student's decision boundaries for distinguishing valid evidence from noise.

Finally, by integrating the adaptive ranker with passage dropout and instruction distillation, we obtain an AdaRankLLM that is capable of selecting necessary passages while maintaining efficiency and accessibility.

## IV. Experimental Setup

TABLE I: The three datasets employed in the experiments.

### IV-A. Datasets

We validate our method on three diverse public datasets: ASQA[38], QAMPARI[1], and ELI5[6]. Consistent with standard protocols[7], we employ Exact Match (EM) for ASQA, F1 score for QAMPARI, and Claim Recall for ELI5. To evaluate comprehensive capabilities, we also report an **Overall*** metric computed as the average of these scores. Detailed dataset information is provided in Table I.

### IV-B. Baselines

We compare AdaRankLLM with several baselines categorized into three groups: (1) **Vanilla-k**: Models utilizing retrieved passages, where k denotes the number of passages provided to the LLM. (2) **Rerank-k**: Models employing reranked passages, with k indicating the number of reranked passages supplied to the LLM. RankGPT4[39] is used as the reranker. (3) **AdaRank-GPT4**: Models utilizing our proposed adaptive ranking prompt (Section III-B) with GPT-4, serving as a baseline to verify the necessity of adaptive RAG. Additionally, we report the **Oracle** values for each backbone, representing the theoretical optimal performance when selecting the best number of reranked passages (k∈[0,10]).

### IV-C. Models

To comprehensively investigate the necessity of adaptive retrieval across varying model capabilities, we conduct experiments on seven backbone models: Alpaca-7b, Mistral-7b, GPT-3.5, GPT-4o, Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, and Qwen3-8B (specifically evaluated with enable_thinking=False and True). These models span a diverse spectrum of open-source and proprietary architectures.

Refer to captionFigure 2: Illustration of the prompt template used in AdaRankLLM for relevance-based passage selection and reranking.

### IV-D. Implementation Details

#### Training Details

We fine-tune AdaRankLLM on Mistral-7B-v0.1[14] using a subset of the MSMARCO V1[30] dataset. The data comprises 100K queries with the top-20 BM25-retrieved passages, truncated to 10 to align with typical RAG settings. We use all 100K queries for the first stage and sample 5K queries for the second stage using ADA2 embeddings[29] with k-means clustering (k=20). Training runs for three epochs per stage with a learning rate of 5e-6 (cosine decay) and a batch size of 64.

#### Evaluation Details

For all three datasets, we utilize the pre-processed first-stage retrieval results from ALCE[7] and apply official chat templates for all backbone models. Regarding metric implementation, we strictly follow the ALCE codebase: employing STR-EM for ASQA and the T5-11B based TRUE model[11] to verify claim entailment for ELI5.

TABLE II: Results for Alpaca-7B, Mistral-Instruct, GPT-3.5, GPT-4o, Llama, Qwen2.5-