Less can be More: Relieving RAG Bottlenecks via Evidence Frontloading and Pressure-Adaptive Budgeting

# Relieving RAG Bottlenecksvia Evidence Frontloading and Pressure-Adaptive Budgeting
Source: [https://arxiv.org/html/2608.25115](https://arxiv.org/html/2608.25115)
## Less can be More: Relieving RAG Bottlenecks via Evidence Frontloading and Pressure\-Adaptive Budgeting

DOI:[XXXXXXX\.XXXXXXX](https://doi.org/XXXXXXX.XXXXXXX)Conference:Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NYISBN:978\-1\-4503\-XXXX\-X/2018/06CCS:Information systems Retrieval models and rankingCCS:Information systems Evaluation of retrieval resultsWeibin Caiemail:[weibin44@data\.syr\.edu](mailto:[email protected])Affiliation:Data Lab, EECS Department,Syracuse University,Syracuse,NY,USAReza Zafaraniemail:[reza@data\.syr\.edu](mailto:[email protected])Affiliation:Data Lab, EECS Department,Syracuse University,Syracuse,NY,USA

2018

###### Abstract\.

Existing methods for improving Retrieval\-Augmented Generation \(RAG\) efficiency mainly optimize downstream LLM generation, such as context compression or serving optimization\. However, RAG is an end\-to\-end system, and its bottleneck can shift between upstream reranking and downstream generation under different serving loads and reranking budgets\. In this paper, we first empirically characterize this shifting\-bottleneck behavior and show that upstream reranking can become the dominant bottleneck under high query rates or large reranking budgets\. Reducing the reranking budget can relieve this bottleneck, but it may also drop supporting evidence and degrade recall\. To address this problem, we proposePACE\(PrioritizedAdaptiveCoverage ofEvidence\), a training\-free framework that combinesevidence frontloadingwithpressure\-adaptive budgeting\.PACEfirst reorders candidates by marginal evidence coverage, prioritizing documents that are query\-relevant, complementary, and useful for forming multi\-hop evidence chains\. We show that this objective is monotone submodular, giving greedy selection a\(1−1/e\)\(1\-1/e\)approximation guarantee\.PACEthen dynamically adjusts the reranking budget according to the relative pressure of the reranker and the LLM\. Experiments on three multi\-hop QA datasets and online serving simulations show thatPACEimproves evidence recall, reduces p95 latency under ranking\-heavy workloads\. More importantly, the two components together reveal thatless can be more: an evidence\-dense top\-ranked candidates enable higher final recall with fewer reranked documents\.

###### Keywords:

Retrieval\-augmented generation, RAG bottleneck mitigation, evidence frontloading, pressure\-adaptive budgeting, evidence recall, multi\-hop question answering

## 1\.Introduction

![Refer to caption](https://arxiv.org/html/2608.25115v1/intro_figure_v3.png)Figure 1\.HowPACEimproves RAG effectiveness and efficiency\. \(1\) Evidence frontloading makes top\-ranked candidates more evidence\-dense, achieving higher evidence recall at a smaller reranking budgetDD\. \(2\) Pressure\-adaptive budgeting then reducesDDunder ranking\-heavy workloads to lower latency\. \(3\) Together,PACEenables denser reranker input and higher final recall with fewer reranked documents\.RAG has become a widely used paradigm for equipping Large Language Models \(LLMs\) with external knowledge across diverse downstream tasks\([Shuster et al\., 2021](https://arxiv.org/html/2608.25115#bib.bib10);[Lewis et al\., 2020](https://arxiv.org/html/2608.25115#bib.bib11);[Borgeaud et al\., 2022](https://arxiv.org/html/2608.25115#bib.bib12)\)\. A typical RAG pipeline first retrieves query\-relevant chunks from a corpus using a dense retriever\([Karpukhin et al\., 2020](https://arxiv.org/html/2608.25115#bib.bib15);[Xiong et al\., 2020a](https://arxiv.org/html/2608.25115#bib.bib16);[Khattab and Zaharia, 2020](https://arxiv.org/html/2608.25115#bib.bib17)\), then applies a reranker\([Nogueira and Cho, 2019](https://arxiv.org/html/2608.25115#bib.bib6);[Sun et al\., 2023](https://arxiv.org/html/2608.25115#bib.bib13);[Nogueira et al\., 2020](https://arxiv.org/html/2608.25115#bib.bib14)\)to refine their order and feeds the top\-kkchunks to the LLM as context\. The quality of the generated answer therefore largely depends on whether the context \(top\-kkchunks\) covers the evidence needed to answer the query\. Increasingkkcan improve evidence recall, but longer contexts also make RAG systems harder to process effectively and efficiently: useful information may be lost in the middle\([Liu et al\., 2024](https://arxiv.org/html/2608.25115#bib.bib3)\), noisy sentences can degrade generation\([Shi et al\., 2023](https://arxiv.org/html/2608.25115#bib.bib4)\), and long inputs introduce substantial inference overhead\([Déjean and Clinchant, 2026](https://arxiv.org/html/2608.25115#bib.bib7);[Yu et al\., 2024](https://arxiv.org/html/2608.25115#bib.bib9)\)\. Existing work addresses these issues from two main directions\. At the system level, techniques such as KV\-cache reuse\([Zheng et al\., 2024](https://arxiv.org/html/2608.25115#bib.bib24)\), scheduling\([Yu et al\., 2022](https://arxiv.org/html/2608.25115#bib.bib25)\), and optimized prefilling\([Agrawal et al\., 2024](https://arxiv.org/html/2608.25115#bib.bib26)\)accelerate LLM inference\. At the data and algorithm level, context compressors\([Chirkova et al\., 2025](https://arxiv.org/html/2608.25115#bib.bib1);[Xu et al\., 2023](https://arxiv.org/html/2608.25115#bib.bib2);[Jiang et al\., 2023](https://arxiv.org/html/2608.25115#bib.bib5)\)aim to preserve core information while removing irrelevant tokens from the retrieved context\. These methods reduce input length or accelerate generation, thereby alleviating downstream bottlenecks \(i\.e\., generation stage bottleneck\) while largely maintaining question\-answering performance\.

However, RAG is an end\-to\-end system, and improving it requires coordinating upstream retrieval/reranking with downstream generation for both effectiveness and efficiency\. For effectiveness, if necessary evidence is not recalled upstream, removing noisy context downstream cannot recover the missing information or improve answer accuracy\. For efficiency, If queries are already queued at the reranker, optimizing only the generator may not improve end\-to\-end latency\. Consequently, approaches that mainly optimize downstream generation may not generalize across different RAG configurations and serving loads\. In particular, as the reranking budget \(i\.e\., the number of retrieved candidates sent to the reranker\) and QPS \(queries per second\) vary, the system bottleneck can shift between ranking and generation\. When queries arrive frequently and many documents must be reranked, the ranking stage can become the bottleneck; otherwise, latency may be dominated by generation\. We refer to these two scenarios asranking\-heavyandgeneration\-heavy\. Although using a small fixed reranking budget can relieve ranking\-heavy workloads, this fixed budget is difficult to choose under dynamic serving loads, and an overly small budget can miss necessary evidence and degrade answer quality\. Therefore, how todynamically relieve upstream ranking bottlenecks while preserving evidence recallremains an open problem\. A promising solution requires two key properties, as illustrated in Figure[1](https://arxiv.org/html/2608.25115#S1.F1):

Evidence frontloading\.This is especially important for multi\-hop questions, where answering a query often requires multiple supporting documents rather than a single highly relevant chunk\. If these supporting documents can be frontloaded into the first few candidates, the system can use a smaller reranking budget, reducing upstream workload while preserving evidence recall\. Existing retrieval\-side methods partially address this goal through multi\-hop retrieval or diversity\-aware ranking\. Multi\-hop retrievers usually acquire supporting evidence through iterative retrieval or reasoning\-guided search\([Xiong et al\., 2020b](https://arxiv.org/html/2608.25115#bib.bib38);[Khattab et al\., 2021](https://arxiv.org/html/2608.25115#bib.bib40);[Trivedi et al\., 2023](https://arxiv.org/html/2608.25115#bib.bib41)\), but they require specialized retriever training or introduce additional LLM\-retrieval iterations, which can increase serving cost\. In contrast, the goal in this scenario isnot to acquire new evidence through extra retrieval steps, but to prioritize evidence within an existing candidate poolso that a smaller reranking budget can still preserve recall under serving pressure\. Diversity\-aware methods are closer to this setting because they usually reorder existing candidates by reducing redundancy\. For example, maximal marginal relevance \(MMR\)\([Carbonell and Goldstein, 1998](https://arxiv.org/html/2608.25115#bib.bib21)\)penalizes similarity to previously selected documents, while Dartboard\([Pickett et al\., 2024](https://arxiv.org/html/2608.25115#bib.bib23)\)optimizes relevant information gain for diverse RAG retrieval\. However, these methods remain limited for multi\-hop evidence frontloading due to three issues: \(i\)coarse\-grained relevance modeling, as they typically rely on document\-level similarity rather than fine\-grained semantic dimensions; \(ii\)missing inter\-document dependencies, as a supporting document may be weakly related to the query but strongly connected to another evidence document; and \(iii\)limited robustness, as their performance often depends on dataset\-sensitive hyperparameters and lacks principled performance guaranties\.

Pressure\-adaptive budgeting\.Instead of relying on a fixed budget, it should dynamically choose the largest budget that does not make reranking a bottleneck relative to generation\. This requires jointly considering upstream reranking pressure and downstream generation pressure, so that the system can preserve as much evidence as possible while relieving ranking\-heavy workloads\. Together, these two properties enable the system to usea smaller reranking budget with higher evidence recall, thereby reducing upstream bottlenecks while maintaining or even improving evidence recall\.

In this work\.We first characterize RAG bottlenecks under online serving simulation with varying QPS and reranking budgets\. Our analysis shows that the bottleneck is not determined solely by the relative size of the reranker and the LLM:increasing QPS or the reranking budget can shift the dominant bottleneck from generation to upstream reranking\. This finding motivates us to focus on the ranking\-heavy scenario, where reducing downstream generation cost alone is insufficient\. To address this problem, we proposePACE, short forPrioritizedAdaptiveCoverage ofEvidence, to relieve upstream reranking bottlenecks while preserving evidence recall\.PACEconsists of two components\. First,evidence frontloadingreorders candidates so that the top\-ranked candidates contain evidence that is directly relevant to the query, complementary to already selected documents, and useful for forming a complete evidence chain\. We formulate this as a marginal evidence coverage problem: using fewer documents to cover more query semantics\. Each document’s coverage is weighted by its direct query relevance and its potential to bridge evidence chains\. This objective is monotone submodular, giving greedy frontloading a\(1−1/e\)\(1\-1/e\)approximation guarantee\. Second,pressure\-adaptive budgetingdynamically adjusts the reranking budget according to the relative queueing pressure of the reranker and the LLM, reducing reranker workload when upstream ranking becomes the bottleneck\. Together, the two components enable a key effect:less can be more when the top\-ranked candidates are evidence\-dense\. Although adaptive budgeting sends fewer candidates to the reranker, evidence frontloading makes these candidates more useful and less noisy, increasing the chance that the reranker promotes complete evidence into the final top\-KKcontext\. Figure[1](https://arxiv.org/html/2608.25115#S1.F1)summarizes how these two components jointly improve RAG effectiveness and efficiency, leading to the “less can be more” effect\. Our contributions are as follows:

- •We empirically identify shifting bottlenecks in RAG serving, showing that the dominant bottleneck can move between reranking and generation as the number of queries per second and the reranking budget change\.
- •We proposePACE, a training\-free framework that combines marginal evidence frontloading with pressure\-adaptive budgeting to reduce upstream reranking workload while preserving evidence recall\. We further show that the proposed evidence coverage objective ofPACEis monotone submodular, which leads to its greedy selection to provide a\(1−1/e\)\(1\-1/e\)approximation guarantee under a cardinality constraint\.
- •Experiments on three multi\-hop QA datasets and online serving simulations show thatPACEimproves evidence recall, substantially reduces latency under ranking\-heavy workloads, and demonstrate thatless can be more when top\-ranked candidates are evidence\-dense: achieve higher final recall with a smaller reranking budget\.

## 2\.RAG Bottlenecks Shift Across Configurations and Loads

\(a\)Bottleneck tendency across reranker–LLM model pairs under increasing QPS\.\(b\)Ranking\-heavy workloads\(c\)Generation\-heavy workloads
Figure 2\.Shifting RAG bottlenecks under different configurations and serving loads\. Here,DDdenotes reranking budget andKKdenotes the number of documents sent to the LLM\. In \(a\), we vary QPS across two rerankers:trecdl22\-crossencoder\-debertav3, andms\-marco\-MiniLM\-L12\-v2, and two LLMs:Qwen2\.5\-3B\-Instruct, andQwen2\.5\-7B\-Instruct\. We reportlog2⁡\(P95 reranker queue timeP95 LLM queue time\)\\log\_\{2\}\\left\(\\frac\{\\text\{P95 reranker queue time\}\}\{\\text\{P95 LLM queue time\}\}\\right\), where positive values indicate ranking\-heavy serving and negative values indicate generation\-heavy serving\. In \(b\) and \(c\) we useQwen2\.5\-3B\-Instructandtrecdl22\-crossencoder\-debertav3pair and varyDDwith and without context compressor\.In a RAG system, the LLM contains most of the parameters and computational complexity, so it is often assumed to be the main serving bottleneck\. However, the bottleneck is not always fixed at generation\. It can shift between upstream reranking and downstream generation under different model choices, configurations \(e\.g\., reranking budgetDD\), and serving loads \(e\.g\., queries per second\)\. We characterize this shift by comparing how long requests wait in the reranker queue and the LLM queue: if requests wait longer in the reranker queue than in the LLM queue, reranking is the bottleneck; otherwise, generation is the bottleneck\. We report p95 queueing latency, i\.e\., the latency below which 95% of requests fall, because serving bottlenecks often appear as tail latency and indicate system saturation under high load\. This paper focuses on the reranking and generation stages, and measures their queueing delays to identify which stage limits system throughput\. To study RAG bottlenecks under realistic serving conditions, we use an open\-loop workload, where requests are issued according to a fixed arrival process independent of previous request completion\. The detailed datasets, models, and serving setup are described in Section[4\.1](https://arxiv.org/html/2608.25115#S4.SS1); here we focus on the observed bottleneck behavior\.

1\. Model size affects bottleneck tendency, while load and reranking budget determine the actual bottleneck\.Figure[2\(a\)](https://arxiv.org/html/2608.25115#S2.F2.sf1)shows that, across different reranker–LLM model pairs, increasing queries per second generally increases upstream reranking pressure\. Relative model size affects the bottleneck, but high load can still make reranking the bottleneck even with a much larger LLM\. For example, the parameter ratio between Qwen\-3B and MiniLM\-L12 is 92\.5, and this pair behaves as generation\-heavy under low QPS but shifts to ranking\-heavy when QPS reaches 2\.5\. Reranking budget also changes the bottleneck\. For the Qwen\-3B and DeBERTa pair, Figure[2\(b\)](https://arxiv.org/html/2608.25115#S2.F2.sf2)shows that with reranking budgetD=100D=100, end\-to\-end latency becomes dominated by reranker queueing delay once QPS reaches 1, whereas Figure[2\(c\)](https://arxiv.org/html/2608.25115#S2.F2.sf3)shows that withD=50D=50, latency remains dominated by LLM queueing delay\.

2\. Relieving downstream bottlenecks cannot necessarily reduce upstream reranking pressure\.Figures[2\(b\)](https://arxiv.org/html/2608.25115#S2.F2.sf2)and[2\(c\)](https://arxiv.org/html/2608.25115#S2.F2.sf3)show that context compression reduces LLM queueing delay by shortening the input context\. As a result, it improves end\-to\-end latency when generation is the dominant bottleneck\. However, this gain becomes limited when the system is ranking\-heavy\. In Figure[2\(b\)](https://arxiv.org/html/2608.25115#S2.F2.sf2), once QPS reaches 1, most latency comes from the reranker queue\. Although compression accelerates LLM processing, it does not reduce reranker queueing delay; moreover, the compressor consumes additional compute, which can further increase reranker pressure and queueing time\. As a result, its end\-to\-end benefit remains limited, especially under high QPS and largeDD\.

## 3\.PACE: Prioritized Adaptive Coverage of Evidence

The observations in Section[2](https://arxiv.org/html/2608.25115#S2)show that RAG bottlenecks shift with serving load and system configuration\. To relieve shifting bottlenecks, the system should adaptively allocate the reranking budgetDDrather than using a fixed value\. However, simply reducingDDmay miss necessary evidence and degrade answer quality\. We therefore proposePACE\(Prioritized Adaptive Coverage of Evidence\), to pace the RAG system by addressing two requirements: \(1\) adapting the reranking budget to system pressure, and \(2\) preserving evidence recall by moving useful evidence into top\-ranked candidates\.

### 3\.1\.Evidence Frontloading

To preserve evidence recall under a small reranking budget, we first frontload useful evidence to the top of the candidate ranking\. Given a queryqqand a candidate document set𝒞=\{di\}i=1Dmax\\mathcal\{C\}=\\\{d\_\{i\}\\\}\_\{i=1\}^\{D\_\{\\max\}\}returned by the dense retriever, whereDmaxD\_\{\\max\}denotes the maximum of reranking budget, e\.g\., top\-100100, our goal is to reorder𝒞\\mathcal\{C\}so that the first few documents cover as much necessary and non\-redundant evidence as possible\. To make the top\-ranked documents useful, each selected document should satisfy two conditions\. First, it should be relevant to the query and cover part of the query’s information need, such as a specific entity, event, or location\. Second, it should provide complementary evidence rather than repeat dimensions that have already been covered by previously selected documents\. Thus, documents should be prioritized not only by their individual query relevance, but also by their marginal contribution to the uncovered evidence space\. As more documents are selected, the remaining uncovered dimensions become fewer, and the gain of adding redundant documents naturally decreases\. We formalize this intuition as a marginal evidence coverage problem\.

#### Marginal evidence covergae\.

Letq⁡\(v\)≥0q\(v\)\\geq 0anddi​\(v\)≥0d\_\{i\}\(v\)\\geq 0denote the representation values of queryqqand documentdid\_\{i\}on semantic dimensionvv, respectively\. The value on each dimension reflects the amount of information carried by the query or document along that dimension\. For a selected document setS⊆𝒞S\\subseteq\\mathcal\{C\}, we define the coverage objective as

\(1\)F⁡\(S\)=∑vq⁡\(v\)​maxdi∈S​\[ρi​di​\(v\)\]\.F\(S\)=\\sum\_\{v\}q\(v\)\\max\_\{d\_\{i\}\\in S\}\\left\[\\sqrt\{\\rho\_\{i\}\}d\_\{i\}\(v\)\\right\]\.whereρi\\rho\_\{i\}denotes the query\-document relevance weight, which can be instantiated by the dense retriever score\. The current coverage of dimensionvvis

\(2\)mS​\(v\)=maxdi∈S⁡\[ρi​di​\(v\)\]\.m\_\{S\}\(v\)=\\max\_\{d\_\{i\}\\in S\}\\left\[\\sqrt\{\\rho\_\{i\}\}d\_\{i\}\(v\)\\right\]\.Thus, the marginal gain of adding documentdid\_\{i\}toSSis

\(3\)Δ⁡\(di∣S\)=∑vq⁡\(v\)​max⁡\[ρi​di​\(v\)−mS​\(v\),0\]\.\\Delta\(d\_\{i\}\\mid S\)=\\sum\_\{v\}q\(v\)\\max\\left\[\\sqrt\{\\rho\_\{i\}\}d\_\{i\}\(v\)\-m\_\{S\}\(v\),0\\right\]\.At each step, we greedily select the document with the largest marginal gain:

\(4\)d⋆=arg⁡maxdi∈𝒞∖S⁡Δ⁡\(di∣S\)\.d^\{\\star\}=\\arg\\max\_\{d\_\{i\}\\in\\mathcal\{C\}\\setminus S\}\\Delta\(d\_\{i\}\\mid S\)\.This produces a ranking prefix that prioritizes relevant and complementary evidence\.

#### Soft\-anchor relevance refinement\.

However, using only the dense retriever score asρi\\rho\_\{i\}can miss supporting documents that are weakly related to the query but strongly connected to other evidence documents\. This is common in multi\-hop QA, where one document may provide an intermediate fact or entity bridge rather than directly matching the query\. To capture such dependencies, we refine the relevance weight using soft anchors, i\.e\., candidate documents with high query relevance that serve as evidence seeds\. Documents similar to these anchors receive higher weights, allowing the ranking prefix to include not only directly query\-relevant documents but also bridge documents that connect multiple supporting evidence into a more complete evidence chain\. Together withmarginal evidence coverage, this refined relevance weight enables the system to frontload evidence that is both comprehensive and useful\. We next describe how to compute this refinement\.

Letρi\\rho\_\{i\}denote the initial query\-document relevance score from the dense retriever, and letsim⁡\(di,dj\)\\operatorname\{sim\}\(d\_\{i\},d\_\{j\}\)denote the similarity between two candidate documents\. We first robustly standardize the relevance scores using the median absolute deviation:

\(5\)zi=ρi−median⁡\(ρ\)MAD⁡\(ρ\)\+ϵ,z\_\{i\}=\\frac\{\\rho\_\{i\}\-\\operatorname\{median\}\(\\rho\)\}\{\\operatorname\{MAD\}\(\\rho\)\+\\epsilon\},whereϵ=10−12\\epsilon=10^\{\-12\}is used for numerical stability, and

\(6\)MAD⁡\(ρ\)=mediani⁡\|ρi−median⁡\(ρ\)\|\.\\operatorname\{MAD\}\(\\rho\)=\\operatorname\{median\}\_\{i\}\\left\|\\rho\_\{i\}\-\\operatorname\{median\}\(\\rho\)\\right\|\.We then convert the standardized scores into soft\-anchor weights:

\(7\)wi=exp⁡\(zi\)∑jexp⁡\(zj\)\.w\_\{i\}=\\frac\{\\exp\(z\_\{i\}\)\}\{\\sum\_\{j\}\\exp\(z\_\{j\}\)\}\.Thus, candidates with higher query relevance receive larger weights and act as soft evidence anchors\. We estimate each document’s soft\-anchor relevance by aggregating its similarity to these anchors:

\(8\)aj=∑i≠jwi​sim⁡\(di,dj\)∑i≠jwi\.a\_\{j\}=\\frac\{\\sum\_\{i\\neq j\}w\_\{i\}\\operatorname\{sim\}\(d\_\{i\},d\_\{j\}\)\}\{\\sum\_\{i\\neq j\}w\_\{i\}\}\.Finally, we combine the direct query\-document relevance and the anchor\-based relevance as

\(9\)ρ~j=1−\(1−bj\)​\(1−a¯j\),\\tilde\{\\rho\}\_\{j\}=1\-\(1\-b\_\{j\}\)\(1\-\\bar\{a\}\_\{j\}\),wherebj=norm⁡\(ρj\)b\_\{j\}=\\operatorname\{norm\}\(\\rho\_\{j\}\),a¯j=norm⁡\(aj\)\\bar\{a\}\_\{j\}=\\operatorname\{norm\}\(a\_\{j\}\), andnorm⁡\(⋅\)\\operatorname\{norm\}\(\\cdot\)denotes min\-max normalization\. The refined weightρ~j\\tilde\{\\rho\}\_\{j\}is then used as the relevance weight in Eq\.[1](https://arxiv.org/html/2608.25115#S3.E1), allowing the coverage objective to favor documents that are directly relevant to the query while also recovering documents connected to query\-relevant evidence through document\-document dependencies\. This refinement can be applied either before or after reranking to improve recall; the only difference is the source of the query\-document relevance score in Eq\.[5](https://arxiv.org/html/2608.25115#S3.E5)\. Before reranking,ρi\\rho\_\{i\}is obtained from the dense retriever, while after reranking, it is obtained from the reranker\. We evaluate this combination of usage in Section[4\.3](https://arxiv.org/html/2608.25115#S4.SS3), Figure[7](https://arxiv.org/html/2608.25115#S4.F7)\.

#### Theoretical property\.

Beyond its empirical effectiveness, the proposedmarginal evidence coverageobjectiveF⁡\(S\)F\(S\)is monotone submodular, which gives greedy selection a standard approximation guarantee to the optimal evidence coverage under a cardinality constraint\.

###### Theorem 3\.1\.

Assumeq⁡\(v\)≥0q\(v\)\\geq 0,ρ~i≥0\\tilde\{\\rho\}\_\{i\}\\geq 0, anddi​\(v\)≥0d\_\{i\}\(v\)\\geq 0for allvvanddid\_\{i\}\. ThenF⁡\(S\)F\(S\)is monotone submodular\.

###### Proof\.

First, we prove monotonicity\. For anyA⊆B⊆𝒞A\\subseteq B\\subseteq\\mathcal\{C\}and any dimensionvv, we have

\(10\)maxdi∈A⁡\[ρ~i​di​\(v\)\]≤maxdi∈B⁡\[ρ~i​di​\(v\)\]\.\\max\_\{d\_\{i\}\\in A\}\\left\[\\sqrt\{\\tilde\{\\rho\}\_\{i\}\}d\_\{i\}\(v\)\\right\]\\leq\\max\_\{d\_\{i\}\\in B\}\\left\[\\sqrt\{\\tilde\{\\rho\}\_\{i\}\}d\_\{i\}\(v\)\\right\]\.Sinceq⁡\(v\)≥0q\(v\)\\geq 0, summing over all dimensions gives

\(11\)F⁡\(A\)≤F⁡\(B\)\.F\(A\)\\leq F\(B\)\.Therefore,F⁡\(⋅\)F\(\\cdot\)is monotone\.

Next, we prove submodularity by showing diminishing returns\. For anyA⊆B⊆𝒞A\\subseteq B\\subseteq\\mathcal\{C\}and anydj∉Bd\_\{j\}\\notin B, define

\(12\)mA​\(v\)=maxdi∈A⁡\[ρ~i​di​\(v\)\],mB​\(v\)=maxdi∈B⁡\[ρ~i​di​\(v\)\]\.m\_\{A\}\(v\)=\\max\_\{d\_\{i\}\\in A\}\\left\[\\sqrt\{\\tilde\{\\rho\}\_\{i\}\}d\_\{i\}\(v\)\\right\],\\quad m\_\{B\}\(v\)=\\max\_\{d\_\{i\}\\in B\}\\left\[\\sqrt\{\\tilde\{\\rho\}\_\{i\}\}d\_\{i\}\(v\)\\right\]\.BecauseA⊆BA\\subseteq B, we havemA​\(v\)≤mB​\(v\)m\_\{A\}\(v\)\\leq m\_\{B\}\(v\)for everyvv\. Therefore,

\(13\)max⁡\[ρ~j​dj​\(v\)−mA​\(v\),0\]≥max⁡\[ρ~j​dj​\(v\)−mB​\(v\),0\]\.\\max\\left\[\\sqrt\{\\tilde\{\\rho\}\_\{j\}\}d\_\{j\}\(v\)\-m\_\{A\}\(v\),0\\right\]\\geq\\max\\left\[\\sqrt\{\\tilde\{\\rho\}\_\{j\}\}d\_\{j\}\(v\)\-m\_\{B\}\(v\),0\\right\]\.Multiplying byq⁡\(v\)≥0q\(v\)\\geq 0and summing overvvyields

\(14\)Δ⁡\(dj∣A\)≥Δ⁡\(dj∣B\)\.\\Delta\(d\_\{j\}\\mid A\)\\geq\\Delta\(d\_\{j\}\\mid B\)\.Thus,F⁡\(⋅\)F\(\\cdot\)is monotone submodular\.

∎

###### Corollary 3\.2\.

Under a cardinality constraint\|S\|≤K\|S\|\\leq K, letSgreedyS\_\{\\mathrm\{greedy\}\}be the set selected by the greedy algorithm that iteratively adds the document with the largest marginal gain\. Then,

\(15\)F⁡\(Sgreedy\)≥\(1−1/e\)​F​\(S⋆\),F\(S\_\{\\mathrm\{greedy\}\}\)\\geq\(1\-1/e\)F\(S^\{\\star\}\),where

\(16\)S⋆=arg⁡max\|S\|≤K⁡F⁡\(S\)\.S^\{\\star\}=\\arg\\max\_\{\|S\|\\leq K\}F\(S\)\.is the optimal document set under the same constraint\.

###### Proof\.

SinceF⁡\(S\)F\(S\)is non\-negative, monotone, and submodular, the result follows from the classical greedy approximation guarantee for monotone submodular maximization under a cardinality constraint\([Nemhauser et al\., 1978](https://arxiv.org/html/2608.25115#bib.bib27)\)\. ∎

This result shows that greedy evidence frontloading achieves at least a\(1−1/e\)\(1\-1/e\)approximation to the optimal evidence coverage, while remaining efficient enough for reranking\-time use\.

### 3\.2\.Pressure\-adaptive Budgeting

Table 1\.Notation for pressure\-adaptive budgeting\.Evidence frontloading makes the ranking prefix more evidence\-dense, but the system still needs to decide how many candidates should be processed by the reranker under changing serving pressure\. A fixed reranking budgetDDis suboptimal: a largeDDpreserves recall but can create a reranker bottleneck, while a smallDDreduces latency but may miss necessary evidence\. We therefore introduce pressure\-adaptive budgeting, which dynamically selects the largest affordable reranking budget for each query based on the relative pressure of the reranker and the LLM\. Together with evidence frontloading, this policy balances the pressure between the two stages while preserving high evidence recall\. Table[1](https://arxiv.org/html/2608.25115#S3.T1)summarizes the notation used in this section\.

We estimate reranker pressure as the time needed to clear the current reranker backlog:

\(17\)WR=PRμ^R\.W\_\{R\}=\\frac\{P\_\{R\}\}\{\\hat\{\\mu\}\_\{R\}\}\.For the LLM, we estimate the remaining service time by combining the current batch and the queued batches:

\(18\)Tremain=max⁡\(T^L−\(tnow−tstart\),0\),T\_\{\\mathrm\{remain\}\}=\\max\\left\(\\hat\{T\}\_\{L\}\-\(t\_\{\\mathrm\{now\}\}\-t\_\{\\mathrm\{start\}\}\),0\\right\),\(19\)WL=Tremain\+⌈\|QL\|BL⌉​T^L\.W\_\{L\}=T\_\{\\mathrm\{remain\}\}\+\\left\\lceil\\frac\{\|Q\_\{L\}\|\}\{B\_\{L\}\}\\right\\rceil\\hat\{T\}\_\{L\}\.WhenWR≤WLW\_\{R\}\\leq W\_\{L\}, reranking is not more congested than generation, so the system usesDmaxD\_\{\\max\}\. WhenWR\>WLW\_\{R\}\>W\_\{L\}, we estimate the excess reranker backlog as

\(20\)ER=μ^R​\(WR−WL\)\.E\_\{R\}=\\hat\{\\mu\}\_\{R\}\(W\_\{R\}\-W\_\{L\}\)\.Given reranker batch sizeBRB\_\{R\}, we reduce the reranking budget by full reranker batches:

\(21\)D⁡\(q\)=max⁡\(Dmin,Dmax−BR​⌊ERBR⌋\)\.D\(q\)=\\max\\left\(D\_\{\\min\},D\_\{\\max\}\-B\_\{R\}\\left\\lfloor\\frac\{E\_\{R\}\}\{B\_\{R\}\}\\right\\rfloor\\right\)\.This rule keeps the largest possible reranking budget when reranking is not the bottleneck, and decreasesD⁡\(q\)D\(q\)only when reranker pressure exceeds LLM pressure\. Together with evidence frontloading, it reduces upstream workload under ranking\-heavy conditions while keeping useful evidence concentrated in the processed prefix\.

## 4\.Experiments

We evaluatePACEfrom two perspectives: whether evidence frontloading improves evidence recall among the top\-ranked candidates, and whether pressure\-adaptive budgeting reduces end\-to\-end latency under shifting bottlenecks while preserving recall\. All experiments are conducted on Quadro RTX 6000 GPUs with approximately 22 GB of available memory\.

### 4\.1\.Experimental Settings

Model selection\.Following prior work\([Chirkova et al\., 2025](https://arxiv.org/html/2608.25115#bib.bib1)\), we usesplade\-v3\([Lassance et al\., 2024](https://arxiv.org/html/2608.25115#bib.bib32)\)as the retriever\. It produces non\-negative query and document representations, satisfying the assumptions in Section[3\.1](https://arxiv.org/html/2608.25115#S3.SS1)\. We usetrecdl22\-crossencoder\-debertav3\([Déjean et al\., 2024](https://arxiv.org/html/2608.25115#bib.bib33)\)as the reranker,Provence\([Chirkova et al\., 2025](https://arxiv.org/html/2608.25115#bib.bib1)\)as the context compressor, andQwen2\.5\-3B\-Instruct\([Qwen et al\., 2025](https://arxiv.org/html/2608.25115#bib.bib34)\)as the generator under a resource\-limited serving setting\. All generation experiments use prefilling\.

Datasets\.We evaluate on the dev splits of three multi\-hop QA datasets with gold evidence annotations: HotpotQA\([Yang et al\., 2018](https://arxiv.org/html/2608.25115#bib.bib29)\)with 1,087 queries, MuSiQue\([Trivedi et al\., 2022](https://arxiv.org/html/2608.25115#bib.bib30)\)with 2,317 queries, and 2WikiMultiHopQA\([Ho et al\., 2020](https://arxiv.org/html/2608.25115#bib.bib31)\)with 2,861 queries\. For each dataset, we reserve an additional 100 queries to tune baselines that require hyperparameter calibration; these queries are excluded from evaluation\. Our method does not use this calibration set\. For HotpotQA and 2WikiMultiHopQA, we retain queries whose complete supporting evidence appears in the top\-100 retrieved documents\. This controls for initial retrieval failures and lets us focus on how effectively each method ranks supporting evidence toward earlier positions at different document budgets\. Following Provence\([Chirkova et al\., 2025](https://arxiv.org/html/2608.25115#bib.bib1)\), we use the Wikipedia corpus preprocessing and passage segmentation provided through BERGEN\([Rau et al\., 2024](https://arxiv.org/html/2608.25115#bib.bib28)\)\. For MuSiQue, we use the official closed\-context candidate paragraphs; therefore, all queries are retained, the maximum reranking budget is 20, and no external Wikipedia retrieval or additional passage segmentation is performed\.

Metrics\.To evaluateevidence frontloading, we measure evidence recall among top\-ranked candidates\. We reportcomplete evidence recall@D, the percentage of queries whose top\-DDdocuments contain all gold supporting evidence, andsupporting evidence recall@D, the average fraction of gold supporting evidence covered by the top\-DDdocuments\. We evaluate document selection forD=1,…,100D=1,\\ldots,100on HotpotQA and 2WikiMultiHopQA, andD=1,…,20D=1,\\ldots,20on MuSiQue, whose closed\-context setting provides at most 20 candidate paragraphs per query\. To evaluatepressure\-adaptive budgeting, we reportp95 end\-to\-end latency,p95 reranker queue time, andp95 LLM queue time\. Here, p95 latency is the latency below which 95% of requests complete\.

Batch size calibration\.To ensure that both reranker and LLM achieve maximum processing speed on the local device, thereby yielding reliable efficiency results\. We profile reranker and LLM on our hardware before the main experiments\. For the reranker, batch size denotes the number of query\-document pairs scored together, and we sweepBr∈\{2,4,8,16,32,64\}B\_\{r\}\\in\\left\\\{2,4,8,16,32,64\\right\\\}\. For the LLM, batch size denotes the maximum number of concurrent generation sequences, and we sweepBl∈\{6,7,8,9,10,11,12\}B\_\{l\}\\in\\left\\\{6,7,8,9,10,11,12\\right\\\}with a maximum of 128 new tokens\. Each candidate batch size is evaluated on the same 100 randomly sampled TydiQA queries\([Clark et al\., 2020](https://arxiv.org/html/2608.25115#bib.bib19)\)\. In this profiling run, the reranker processes the top\-50 retrieved documents and returns the top\-5 documents to the LLM\. We measure throughput, p95 latency, and GPU memory usage, repeat each run three times with a fixed random seed\. We select the smallest batch size whose throughput is within 95% of the best observed throughput while satisfying latency and memory constraints\. The selected batch sizes,BR=8B\_\{R\}=8for DeBERTa andBL=10B\_\{L\}=10for Qwen2\.5\-3B, are fixed in all main experiments so that bottleneck shifts are caused by workload changes rather than per\-setting batch\-size tuning\.

Online serving simulation\.To study RAG bottlenecks under realistic serving conditions, we use an open\-loop workload, where requests are issued according to a fixed arrival process independent of previous request completion\. Queries arrive following a Poisson process\([Ray et al\., 2025](https://arxiv.org/html/2608.25115#bib.bib20)\), with QPS varying from 0\.5 to 2\.5\. We deploy the reranker and the LLM on separate GPUs and connect them with independent asynchronous queues to enable pipeline parallelism\. To examine whether downstream context reduction translates into system\-level efficiency gains, we insert Provence\([Chirkova et al\., 2025](https://arxiv.org/html/2608.25115#bib.bib1)\), a context compressor, between the reranker and the LLM and run it on a separate GPU\. For analyzing RAG bottlenecks in Figure[2](https://arxiv.org/html/2608.25115#S2.F2), we use the first 60 seconds as warm\-up and record per\-stage latency over the following 300 seconds\. For the end\-to\-end evaluation ofPACEin Figure[7](https://arxiv.org/html/2608.25115#S4.F7)and[6](https://arxiv.org/html/2608.25115#S4.F6), we setDmin=20D\_\{\\min\}=20, andDmax=100D\_\{\\max\}=100, issue all evaluation queries according to the target QPS and continue running until all queries are completed\. This allows us to measure evidence recall under dynamically selectedDDin an online setting\. We fix the query order and document order across workloads for fair comparison\. Generation uses natural end\-of\-sequence termination with a maximum of 128 output tokens\.

#### Use of AI tools\.

We used AI tools to assist with the implementation of experimental scripts for online serving simulation\. All generated code was reviewed, tested, and validated by the authors\. The AI tool was not used to generate datasets, labels or conclusions\.

Baselines\.To evaluate the effectiveness of evidence frontloading, we compare with training\-free reordering methods that do not require additional model training\. All methods operate on the same candidate set returned by the retriever\. For baselines with tunable hyperparameters, we select the best configuration on the 100 query calibration set of each dataset and apply it unchanged to the evaluation split\.

- •Standard Dense: It uses the original ranking returned by the dense retriever without any reordering\.
- •Rocchio Pseudo\-Relevance Feedback \(PRF\)\([Rocchio Jr, 1971](https://arxiv.org/html/2608.25115#bib.bib22)\): It assumes the top retrieved documents are pseudo\-relevant, updates the query representation by combining the original query with these documents, and then reorders candidates using the expanded query\.
- •The Maximal Marginal Relevance \(MMR\)\([Carbonell and Goldstein, 1998](https://arxiv.org/html/2608.25115#bib.bib21)\): It greedily selects documents by balancing query relevance and novelty, explicitly penalizing candidates that are similar to previously selected documents\.
- •Dartboard\([Pickett et al\., 2024](https://arxiv.org/html/2608.25115#bib.bib23)\): It selects documents by maximizing relevant information gain, encouraging the selected context to contain information that is both useful for the query and diverse from previously selected documents\.
- •Adaptive\-K\([Taguchi et al\., 2025](https://arxiv.org/html/2608.25115#bib.bib18)\): It heuristically selectsDDby stopping where query\-document similarity has the largest drop between two consecutive ranked documents\. SinceDDis determined by the method rather than fixed externally, we report the mean selectedDDand its corresponding recall as a single point in the plots\.

To further analyze the role of the relevance weight in Eq\.[1](https://arxiv.org/html/2608.25115#S3.E1), we conduct ablation studies with three variants:

- •w/o query & anchor relevance: removes the relevance weightρi\\rho\_\{i\}from the coverage objective: \(22\)F⁡\(S\)=∑vq⁡\(v\)​maxdi∈S​\[di​\(v\)\]\.F\(S\)=\\sum\_\{v\}q\(v\)\\max\_\{d\_\{i\}\\in S\}\\left\[d\_\{i\}\(v\)\\right\]\.
- •w/o query relevance: uses only soft\-anchor relevance by settingbj=0b\_\{j\}=0in Eq\.[9](https://arxiv.org/html/2608.25115#S3.E9)\.
- •w/o anchor relevance: uses only direct query relevance by settinga¯j=0\\bar\{a\}\_\{j\}=0in Eq\.[9](https://arxiv.org/html/2608.25115#S3.E9)\.

### 4\.2\.Evidence Frontloading Improves Recall

Figure 3\.Complete evidence recall@D across three multi\-hop QA datasets\.PACEtends to cover all required evidence with smaller reranking budgets than training\-free baselines\.Figure 4\.Supporting evidence recall@D across three multi\-hop QA datasets\.PACEretrieves a larger fraction of supporting evidence with small reranking budgets\.Figure 5\.Ablation study of the marginal evidence coverage objective in Eq\.[1](https://arxiv.org/html/2608.25115#S3.E1)\. Removing query relevance, anchor relevance, or both weakens complete evidence recall, showing that direct query relevance and soft\-anchor document dependencies jointly contribute to effective evidence frontloading\.We first evaluate the effectivenes ofPACEevidence frontloading under a fixed budget ofD=100D=100, focusing on whether it can move supporting evidence into top\-ranked candidates\. Figure[3](https://arxiv.org/html/2608.25115#S4.F3)and[4](https://arxiv.org/html/2608.25115#S4.F4)report evidence recall at different reranking budgets\. Across all three datasets,PACEconsistently improves both complete evidence recall@D and supporting evidence recall@D over training\-free baselines, especially at smallDD, which is the key regime for reducing reranker workload\. For example, on HotpotQA,PACEatD=20D=20achieves comparable evidence recall to the best baseline atD=40D=40, using only half of the reranking budget\.

The baselines show that heuristic diversity does not necessarily improve evidence coverage\. MMR and Dartboard often underperform the original dense ranking because they penalize similar documents, while multi\-hop supporting documents can be semantically related and jointly necessary for answering\. PRF can benefit from expanding the query with top\-ranked documents, but it is sensitive to noisy initial results and may suffer from query drift\. Adaptive\-K is shown as a single point because it selectsDDautomatically; its recall is bounded by the original dense ranking at the selected position since it does not reorder documents\. These results show thatPACEmoves necessary evidence into top\-ranked candidates, rather than only improving recall at large budgets\.

Figure[5](https://arxiv.org/html/2608.25115#S4.F5)further analyzes the relevance weightρ\\rhoin the marginal evidence coverage objective in Eq\.[1](https://arxiv.org/html/2608.25115#S3.E1)\. Removing both query and anchor relevance usually leads to the worst performance, showing that pure coverage without relevance guidance cannot reliably identify useful evidence\. Using only query relevance prioritizes documents directly related to the question, but can miss evidence that is weakly related to the query but instead connected to its one\-hop evidence\. Using only anchor relevance can recover such evidence, but may also promote irrelevant documents when the anchors are noisy\. As a result, using either source alone is generally suboptimal, especially at smallDD, wherePACEconsistently achieves higher recall\. MuSiQue shows a slightly different pattern, where anchor relevance is relatively more effective than query relevance\. This is likely because MuSiQue uses a closed\-context setting with at most 20 candidates, which reduces noise in anchor estimation, and its questions are constructed by composing single\-hop questions through bridge entities\. This also explains why PRF performs relatively better on MuSiQue than on the open\-domain datasets, since its pseudo\-relevance feedback is less affected by noisy top\-ranked documents in the closed\-context candidate set\. Overall, combining query and anchor relevance gives the strongest and most stable evidence frontloading across datasets\.

### 4\.3\.PACEImproves Efficiency while Preserving Recall

Figure 6\.Latency comparison under ranking\-heavy workloads\.PACEsubstantially reduces p95 end\-to\-end latency by lowering reranker queueing time with a smaller adaptive reranking budget\. Although LLM queueing time may increase as more requests pass through the reranker, but the dominant upstream bottleneck is greatly relieved\. The fixed baseline usesD=100D=100throughout\.Figure 7\.Evidence recall under pressure\-adaptive budgeting on HotpotQA\. The first two subfigures report recall@D before reranking, and the last two report recall@5 after reranking, whereDDis dynamically selected andK=5K=5documents are sent to the LLM\.Less can be more when the top\-ranked candidates are evidence\-dense: as QPS increases,PACEreduces the reranking budget but maintains higher complete and supporting evidence recall than baselines and ablation variants\.Figure 8\.Average selected reranking budget under increasing QPS\.PACEreducesDDas serving pressure grows, but maintains higher evidence recall than other methods at comparable selected budgets\.The static offline results in Section[4\.2](https://arxiv.org/html/2608.25115#S4.SS2)show that evidence frontloading improves recall across different cutoffsDD\. We now evaluate whetherPACEcan translate this advantage into end\-to\-end latency gains under online serving, while preserving evidence recall after dynamically reducingDD\. We conduct online serving experiments on HotpotQA and report latency in Figure[6](https://arxiv.org/html/2608.25115#S4.F6)and recall in Figures[7](https://arxiv.org/html/2608.25115#S4.F7)\.

Pressure\-adaptive budget selection improves system latency\.Figure[6](https://arxiv.org/html/2608.25115#S4.F6)comparesPACEwith a fixedD=100D=100reranking budget under ranking\-heavy workloads\. As QPS increases, the fixed\-D=100D=100baseline quickly accumulates reranker queueing delay, causing p95 end\-to\-end latency to grow sharply\. In contrast,PACEdynamically reduces the reranking budget according to system pressure, which substantially lowers reranker queueing time and stabilizes end\-to\-end latency\. Although LLM queue time can increase slightly because more requests pass through the reranker, the dominant upstream bottleneck is relieved, leading to much lower overall latency\.

PACEpreserves recall under smaller adaptive budget\.To fairly compare methods in the online serving setting, we apply the same pressure\-adaptive budgeting policy to all reordering methods\. As a result, all adaptive methods operate with similar latency and comparable selected reranking budgets, since reordering a small candidate set is negligible compared with total serving latency; for example,PACEselection accounts for less than 0\.007% of p95 end\-to\-end latency\. This allows us to focus on how evidence recall changes under increasing serving load\. Figure[8](https://arxiv.org/html/2608.25115#S4.F8)shows that as QPS increases,PACEgradually reducesDD, similar to other adaptive methods\. At QPS=1\.8=1\.8, most adaptive methods select a budget around5050–6060documents to relieve reranker pressure\.

Figure[7](https://arxiv.org/html/2608.25115#S4.F7)reports evidence recall at two stages: recall@D before reranking, and recall@K after reranking, where we use the common settingK=5K=5\([Chirkova et al\., 2025](https://arxiv.org/html/2608.25115#bib.bib1)\)\. Before reranking, the fixedD=100D=100baseline maintains recall@D1\.01\.0because no candidates are dropped\. All adaptive methods show lower recall@D as QPS increases because their selectedDDdecreases\. However,PACEmaintains higher complete and supporting evidence recall than baselines and ablation variants at comparable budgets\. This shows that its latency reduction does not come from simply dropping candidates, but from combining adaptive budgeting to an evidence\-dense candidate ranking\.

After reranking,PACEshows thatless can be more when the top\-ranked candidates are evidence\-dense: it achieves the highest recall@5 among the documents sent to the LLM, even surpassing the fixedD=100D=100, and remains stable as QPS increases\. This suggests that a larger reranking budget does not always improve final top\-KKevidence recall, since additional candidates may introduce noise\. By frontloading useful evidence before reranking and reducingDDunder pressure,PACEprovides the reranker with a smaller but more evidence\-dense candidate set, making useful evidence more likely to appear in the final context\. For example, at QPS=1\.8=1\.8,PACEuses nearly half the reranking budget of fixedD=100D=100, but achieves about 20% higher recall@5\. In contrast, most baselines fall below fixedD=100D=100after reranking, and their recall generally decreases as QPS increases because the adaptive budget becomes smaller\. We further validate this effect under different fixed reranking budgets on two datasets in Table[2](https://arxiv.org/html/2608.25115#S4.T2):PACEachieves higher final recall atK=5K=5than the full\-budget baseline even with substantially smallerDD, showing that its gains come from evidence frontloading rather than simply preserving more candidates\.

Table 2\.Less can be more\. Complete evidence recall under fixed reranking budgets\. FixedD=100D=100denotes the full\-budget baseline, andR​@​KR@KwithK=5K=5measures final evidence recall after reranking\. Values in parentheses indicate absolute percentage\-point changes inR​@​KR@Kover the full\-budget baseline on the same dataset\.DatasetMethodDDComp\. R@DDComp\. R@KKHotpotQAStandard Dense10010041\.13Dartboard8095\.3140\.48\(−0\.65\)\(\-0\.65\)5083\.1738\.18\(−2\.95\)\(\-2\.95\)2052\.9933\.22\(−7\.91\)\(\-7\.91\)PACE8097\.2560\.81\(\+19\.68\)\(\+19\.68\)5091\.3662\.38\(\+21\.25\)\(\+21\.25\)2077\.1060\.26\(\+19\.13\)\(\+19\.13\)2WikiStandard Dense10010048\.76Dartboard8095\.1548\.10\(−0\.66\)\(\-0\.66\)5081\.9747\.40\(−1\.36\)\(\-1\.36\)2053\.4139\.96\(−8\.80\)\(\-8\.80\)PACE8097\.2454\.22\(\+5\.46\)\(\+5\.46\)5091\.0654\.39\(\+5\.63\)\(\+5\.63\)2075\.1953\.94\(\+5\.18\)\(\+5\.18\)

## 5\.Related Work

Relieving generation\-heavy bottlenecks\.Existing work on efficient RAG often reduces the cost of downstream LLM generation\. Serving systems improve throughput and latency through optimized batching, scheduling, KV\-cache management, and prefill execution\([Yu et al\., 2022](https://arxiv.org/html/2608.25115#bib.bib25);[Kwon et al\., 2023](https://arxiv.org/html/2608.25115#bib.bib35);[Zheng et al\., 2024](https://arxiv.org/html/2608.25115#bib.bib24);[Agrawal et al\., 2024](https://arxiv.org/html/2608.25115#bib.bib26)\)\. METIS\([Ray et al\., 2025](https://arxiv.org/html/2608.25115#bib.bib20)\)adapts query\-level configurations, such as retrieved chunks and synthesis methods, to balance response quality and latency\. Context compression methods, such as RECOMP\([Xu et al\., 2023](https://arxiv.org/html/2608.25115#bib.bib2)\), LLMLingua\([Jiang et al\., 2023](https://arxiv.org/html/2608.25115#bib.bib5)\), and Provence\([Chirkova et al\., 2025](https://arxiv.org/html/2608.25115#bib.bib1)\), shorten retrieved contexts by removing or summarizing less useful tokens\. These methods reduce generation\-side cost, but they do not directly address shifting bottlenecks between reranking and generation or relieve reranker\-side pressure while preserving evidence recall\.

Relieving ranking\-heavy bottlenecks\.Rerankers improve retrieval quality but become expensive when many query\-document pairs must be scored\([Déjean and Clinchant, 2026](https://arxiv.org/html/2608.25115#bib.bib7);[Yu et al\., 2024](https://arxiv.org/html/2608.25115#bib.bib9);[An et al\., 2025](https://arxiv.org/html/2608.25115#bib.bib8)\)\. Prior work mainly reduces this cost by accelerating the reranker itself\. Early\-exit methods make decisions at intermediate Transformer layers using auxiliary classifiers\([Xin et al\., 2020](https://arxiv.org/html/2608.25115#bib.bib36)\)or layer\-wise query\-document similarity estimates\([Busolin et al\., 2025](https://arxiv.org/html/2608.25115#bib.bib37)\)\. Adaptive\-K\([Taguchi et al\., 2025](https://arxiv.org/html/2608.25115#bib.bib18)\)selects the budget at the largest drop in query\-document similarity\. These methods reduce reranking computation or choose a heuristic budget point, but they do not explicitly adapt the reranking budget to real\-time reranker–LLM pressure\.PACEis complementary: it keeps the reranker unchanged, but controls which candidates enter reranking and how many are processed under online serving pressure\.

Improving evidence recall\.Evidence recall is often improved through multi\-hop retrieval or diversity\-aware ranking\. Multi\-hop retrievers acquire supporting evidence through iterative retrieval or reasoning\-guided search: MDR\([Xiong et al\., 2020b](https://arxiv.org/html/2608.25115#bib.bib38)\)recursively conditions later retrieval steps on previously retrieved evidence, Baleen\([Khattab et al\., 2021](https://arxiv.org/html/2608.25115#bib.bib40)\)uses condensed intermediate evidence to guide subsequent hops, and IRCoT\([Trivedi et al\., 2023](https://arxiv.org/html/2608.25115#bib.bib41)\)interleaves LLM reasoning with retrieval\. These methods target evidence acquisition, but require specialized retriever training or additional LLM\-retrieval iterations, which can increase serving cost\. Diversity\-aware methods, which improves recall by selecting non\-redundant documents from a candidate set\. MMR\([Carbonell and Goldstein, 1998](https://arxiv.org/html/2608.25115#bib.bib21)\)balances query relevance with novelty, xQuAD\([Santos et al\., 2010](https://arxiv.org/html/2608.25115#bib.bib39)\)promotes coverage of different query aspects, and Dartboard\([Pickett et al\., 2024](https://arxiv.org/html/2608.25115#bib.bib23)\)maximizes relevant information gain for diverse RAG retrieval\. These methods are closer to our setting because they reorder or select from existing candidates\. However, they typically rely on coarse document\-level distances or manually designed trade\-offs, while multi\-hop QA may require preserving documents that are similar but jointly necessary\.PACEdiffers bymeasuring complementarity over fine\-grained semantic dimensions,supporting both diversity and evidence chaining without additional training or tuned parameters, andproviding monotone submodular objective with a\(1−1/e\)\(1\-1/e\)approximation guarantee\.

## 6\.Conclusion

In this paper, we study RAG as an end\-to\-end serving system and show that its bottleneck is not fixed at LLM generation\. Instead, the dominant bottleneck can shift between reranking and generation as query arrival rates and reranking budgets change\. Motivated by this observation, we proposePACE, a training\-free framework that relieves upstream reranking bottlenecks while preserving evidence recall\.PACEfrontloads useful evidence into the top of the candidate ranking through a monotone submodular marginal coverage objective, and then adaptively selects the reranking budget based on real\-time reranker and LLM pressure\. Experiments on multi\-hop QA datasets and online serving simulations demonstrate thatPACEimproves evidence recall, substantially reduces latency under ranking\-heavy workloads, and shows that less can be more when the top\-ranked candidates are evidence\-dense: a smaller reranking budget can lead to higher final evidence recall\.

## References

- Agrawalet al\.\(2024\)A\. Agrawal, N\. Kedia, A\. Panwar, J\. Mohan, N\. Kwatra, B\. Gulavani, A\. Tumanov, and R\. RamjeeTaming\{\\\{throughput\-latency\}\\\}tradeoff in\{\\\{llm\}\\\}inference with\{\\\{sarathi\-serve\}\\\}\.In18th USENIX symposium on operating systems design and implementation \(OSDI 24\),pp\. 117–134\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p1.1),[§5](https://arxiv.org/html/2608.25115#S5.p1.1)\.
- Anet al\.\(2025\)Y\. An, Y\. Cheng, S\. J\. Park, and J\. JiangHyperrag: enhancing quality\-efficiency tradeoffs in retrieval\-augmented generation with reranker kv\-cache reuse\.arXiv preprint arXiv:2504\.02921\.Cited by:[§5](https://arxiv.org/html/2608.25115#S5.p2.1)\.
- Borgeaudet al\.\(2022\)S\. Borgeaud, A\. Mensch, J\. Hoffmann, T\. Cai, E\. Rutherford, K\. Millican, G\. B\. Van Den Driessche, J\. Lespiau, B\. Damoc, A\. Clark,et al\.Improving language models by retrieving from trillions of tokens\.InInternational conference on machine learning,pp\. 2206–2240\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p1.1)\.
- Busolinet al\.\(2025\)F\. Busolin, C\. Lucchese, F\. M\. Nardini, S\. Orlando, R\. Perego, S\. Trani, and A\. VeneriEfficient re\-ranking with cross\-encoders via early exit\.InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 2534–2544\.Cited by:[§5](https://arxiv.org/html/2608.25115#S5.p2.1)\.
- Carbonell and Goldstein \(1998\)J\. G\. Carbonell and J\. GoldsteinThe use of mmr, diversity\-based reranking for reordering documents and producing summaries\.\.InSIGIR,Vol\.98,pp\. 290941–291025\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p3.1),[3rd item](https://arxiv.org/html/2608.25115#S4.I1.i3.p1.1),[§5](https://arxiv.org/html/2608.25115#S5.p3.1)\.
- Chirkovaet al\.\(2025\)N\. Chirkova, T\. Formal, V\. Nikoulina, and S\. ClinchantProvence: efficient and robust context pruning for retrieval\-augmented generation\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=TDy5Ih78b4)Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p1.1),[§4\.1](https://arxiv.org/html/2608.25115#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2608.25115#S4.SS1.p2.1),[§4\.1](https://arxiv.org/html/2608.25115#S4.SS1.p5.1),[§4\.3](https://arxiv.org/html/2608.25115#S4.SS3.p4.1),[§5](https://arxiv.org/html/2608.25115#S5.p1.1)\.
- Clarket al\.\(2020\)J\. H\. Clark, E\. Choi, M\. Collins, D\. Garrette, T\. Kwiatkowski, V\. Nikolaev, and J\. PalomakiTyDi qa: a benchmark for information\-seeking question answering in typologically diverse languages\.Transactions of the Association for Computational Linguistics\.Cited by:[§4\.1](https://arxiv.org/html/2608.25115#S4.SS1.p4.1)\.
- Déjeanet al\.\(2024\)H\. Déjean, S\. Clinchant, and T\. FormalA thorough comparison of cross\-encoders and llms for reranking splade\.arXiv preprint arXiv:2403\.10407\.Cited by:[§4\.1](https://arxiv.org/html/2608.25115#S4.SS1.p1.1)\.
- Déjean and Clinchant \(2026\)H\. Déjean and S\. ClinchantEfficient listwise reranking with compressed document representations\.arXiv preprint arXiv:2604\.26483\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p1.1),[§5](https://arxiv.org/html/2608.25115#S5.p2.1)\.
- Hoet al\.\(2020\)X\. Ho, A\. Duong Nguyen, S\. Sugawara, and A\. AizawaConstructing a multi\-hop QA dataset for comprehensive evaluation of reasoning steps\.InProceedings of the 28th International Conference on Computational Linguistics,Barcelona, Spain \(Online\),pp\. 6609–6625\.External Links:[Link](https://www.aclweb.org/anthology/2020.coling-main.580)Cited by:[§4\.1](https://arxiv.org/html/2608.25115#S4.SS1.p2.1)\.
- Jianget al\.\(2023\)H\. Jiang, Q\. Wu, C\. Lin, Y\. Yang, and L\. QiuLlmlingua: compressing prompts for accelerated inference of large language models\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 13358–13376\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p1.1),[§5](https://arxiv.org/html/2608.25115#S5.p1.1)\.
- Karpukhinet al\.\(2020\)V\. Karpukhin, B\. Oguz, S\. Min, P\. Lewis, L\. Wu, S\. Edunov, D\. Chen, and W\. YihDense passage retrieval for open\-domain question answering\.InProceedings of the 2020 conference on empirical methods in natural language processing \(EMNLP\),pp\. 6769–6781\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p1.1)\.
- Khattabet al\.\(2021\)O\. Khattab, C\. Potts, and M\. ZahariaBaleen: robust multi\-hop reasoning at scale via condensed retrieval\.Advances in Neural Information Processing Systems34,pp\. 27670–27682\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p3.1),[§5](https://arxiv.org/html/2608.25115#S5.p3.1)\.
- Khattab and Zaharia \(2020\)O\. Khattab and M\. ZahariaColbert: efficient and effective passage search via contextualized late interaction over bert\.InProceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval,pp\. 39–48\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p1.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[§5](https://arxiv.org/html/2608.25115#S5.p1.1)\.
- Lassanceet al\.\(2024\)C\. Lassance, H\. Déjean, T\. Formal, and S\. ClinchantSPLADE\-v3: new baselines for splade\.arXiv preprint arXiv:2403\.06789\.Cited by:[§4\.1](https://arxiv.org/html/2608.25115#S4.SS1.p1.1)\.
- Lewiset al\.\(2020\)P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in neural information processing systems33,pp\. 9459–9474\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p1.1)\.
- Liuet al\.\(2024\)N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. LiangLost in the middle: how language models use long contexts\.Transactions of the association for computational linguistics12,pp\. 157–173\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p1.1)\.
- Nemhauseret al\.\(1978\)G\. L\. Nemhauser, L\. A\. Wolsey, and M\. L\. FisherAn analysis of approximations for maximizing submodular set functions—i\.Mathematical programming14\(1\),pp\. 265–294\.Cited by:[§3\.1](https://arxiv.org/html/2608.25115#S3.SS1.SSS0.Px3.p5.1.1)\.
- Nogueira and Cho \(2019\)R\. Nogueira and K\. ChoPassage re\-ranking with bert\.arXiv preprint arXiv:1901\.04085\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p1.1)\.
- Nogueiraet al\.\(2020\)R\. Nogueira, Z\. Jiang, R\. Pradeep, and J\. LinDocument ranking with a pretrained sequence\-to\-sequence model\.InFindings of the association for computational linguistics: EMNLP 2020,pp\. 708–718\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p1.1)\.
- Pickettet al\.\(2024\)M\. Pickett, J\. Hartman, A\. K\. Bhowmick, R\. Alam, and A\. VempatyBetter rag using relevant information gain\.arXiv preprint arXiv:2407\.12101\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p3.1),[4th item](https://arxiv.org/html/2608.25115#S4.I1.i4.p1.1),[§5](https://arxiv.org/html/2608.25115#S5.p3.1)\.
- Qwenet al\.\(2025\)A\. Y\. Qwen, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei,et al\.Qwen2\. 5 technical report\.arXiv preprint arXiv:2412\.15115,pp\. 2412–15115\.Cited by:[§4\.1](https://arxiv.org/html/2608.25115#S4.SS1.p1.1)\.
- Rauet al\.\(2024\)D\. Rau, H\. Déjean, N\. Chirkova, T\. Formal, S\. Wang, S\. Clinchant, and V\. NikoulinaBERGEN: a benchmarking library for retrieval\-augmented generation\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 7640–7663\.Cited by:[§4\.1](https://arxiv.org/html/2608.25115#S4.SS1.p2.1)\.
- Rayet al\.\(2025\)S\. Ray, R\. Pan, Z\. Gu, K\. Du, S\. Feng, G\. Ananthanarayanan, R\. Netravali, and J\. JiangMetis: fast quality\-aware rag systems with configuration adaptation\.InProceedings of the ACM SIGOPS 31st symposium on operating systems principles,pp\. 606–622\.Cited by:[§4\.1](https://arxiv.org/html/2608.25115#S4.SS1.p5.1),[§5](https://arxiv.org/html/2608.25115#S5.p1.1)\.
- Rocchio Jr \(1971\)J\. J\. Rocchio JrRelevance feedback in information retrieval\.The SMART retrieval system: experiments in automatic document processing\.Cited by:[2nd item](https://arxiv.org/html/2608.25115#S4.I1.i2.p1.1)\.
- Santoset al\.\(2010\)R\. L\. Santos, J\. Peng, C\. Macdonald, and I\. OunisExplicit search result diversification through sub\-queries\.InEuropean conference on information retrieval,pp\. 87–99\.Cited by:[§5](https://arxiv.org/html/2608.25115#S5.p3.1)\.
- Shiet al\.\(2023\)F\. Shi, X\. Chen, K\. Misra, N\. Scales, D\. Dohan, E\. H\. Chi, N\. Schärli, and D\. ZhouLarge language models can be easily distracted by irrelevant context\.InInternational Conference on Machine Learning,pp\. 31210–31227\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p1.1)\.
- Shusteret al\.\(2021\)K\. Shuster, S\. Poff, M\. Chen, D\. Kiela, and J\. WestonRetrieval augmentation reduces hallucination in conversation\.InFindings of the Association for Computational Linguistics: EMNLP 2021,pp\. 3784–3803\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p1.1)\.
- Sunet al\.\(2023\)W\. Sun, L\. Yan, X\. Ma, S\. Wang, P\. Ren, Z\. Chen, D\. Yin, and Z\. RenIs chatgpt good at search? investigating large language models as re\-ranking agents\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 14918–14937\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p1.1)\.
- Taguchiet al\.\(2025\)C\. Taguchi, S\. Maekawa, and N\. BhutaniEfficient context selection for long\-context qa: no tuning, no iteration, just adaptive\-k\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 20116–20141\.Cited by:[5th item](https://arxiv.org/html/2608.25115#S4.I1.i5.p1.1),[§5](https://arxiv.org/html/2608.25115#S5.p2.1)\.
- Trivediet al\.\(2022\)H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. SabharwalMusique: multihop questions via single\-hop question composition\.Transactions of the Association for Computational Linguistics10,pp\. 539–554\.Cited by:[§4\.1](https://arxiv.org/html/2608.25115#S4.SS1.p2.1)\.
- Trivediet al\.\(2023\)H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. SabharwalInterleaving retrieval with chain\-of\-thought reasoning for knowledge\-intensive multi\-step questions\.InProceedings of the 61st annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 10014–10037\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p3.1),[§5](https://arxiv.org/html/2608.25115#S5.p3.1)\.
- Xinet al\.\(2020\)J\. Xin, R\. Nogueira, Y\. Yu, and J\. LinEarly exiting bert for efficient document ranking\.InProceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing,pp\. 83–88\.Cited by:[§5](https://arxiv.org/html/2608.25115#S5.p2.1)\.
- Xionget al\.\(2020a\)L\. Xiong, C\. Xiong, Y\. Li, K\. Tang, J\. Liu, P\. Bennett, J\. Ahmed, and A\. OverwijkApproximate nearest neighbor negative contrastive learning for dense text retrieval\.arXiv preprint arXiv:2007\.00808\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p1.1)\.
- Xionget al\.\(2020b\)W\. Xiong, X\. L\. Li, S\. Iyer, J\. Du, P\. Lewis, W\. Y\. Wang, Y\. Mehdad, W\. Yih, S\. Riedel, D\. Kiela,et al\.Answering complex open\-domain questions with multi\-hop dense retrieval\.arXiv preprint arXiv:2009\.12756\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p3.1),[§5](https://arxiv.org/html/2608.25115#S5.p3.1)\.
- Xuet al\.\(2023\)F\. Xu, W\. Shi, and E\. ChoiRecomp: improving retrieval\-augmented lms with compression and selective augmentation\.arXiv preprint arXiv:2310\.04408\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p1.1),[§5](https://arxiv.org/html/2608.25115#S5.p1.1)\.
- Yanget al\.\(2018\)Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. ManningHotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 2369–2380\.Cited by:[§4\.1](https://arxiv.org/html/2608.25115#S4.SS1.p2.1)\.
- Yuet al\.\(2022\)G\. Yu, J\. S\. Jeong, G\. Kim, S\. Kim, and B\. ChunOrca: a distributed serving system for\{\\\{transformer\-based\}\\\}generative models\.In16th USENIX symposium on operating systems design and implementation \(OSDI 22\),pp\. 521–538\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p1.1),[§5](https://arxiv.org/html/2608.25115#S5.p1.1)\.
- Yuet al\.\(2024\)Y\. Yu, W\. Ping, Z\. Liu, B\. Wang, J\. You, C\. Zhang, M\. Shoeybi, and B\. CatanzaroRankrag: unifying context ranking with retrieval\-augmented generation in llms\.Advances in Neural Information Processing Systems37,pp\. 121156–121184\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p1.1),[§5](https://arxiv.org/html/2608.25115#S5.p2.1)\.
- Zhenget al\.\(2024\)L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez,et al\.Sglang: efficient execution of structured language model programs\.Advances in neural information processing systems37,pp\. 62557–62583\.Cited by:[§1](https://arxiv.org/html/2608.25115#S1.p1.1),[§5](https://arxiv.org/html/2608.25115#S5.p1.1)\.