When More Documents Hurt RAG: Mitigating Vector Search Dilution with Domain-Scoped, Model-Agnostic Retrieval

# When More Documents Hurt RAG: Mitigating Vector Search Dilution with Domain-Scoped, Model-Agnostic Retrieval
Source: [https://arxiv.org/html/2606.11350](https://arxiv.org/html/2606.11350)
Nabaraj Subedi1∗, Ahmed Abdelaty2, and Shivanand Venkanna Sheshappanavar1 1Dept\. of Electrical Engineering & Computer Science 2Dept\. of Civil, Architectural Engineering & Construction Management University of Wyoming, Laramie, WY 82071, USA \{nsubedi1, aahmed3, ssheshap\}@uwyo\.edu ∗\*Correspondence author

###### Abstract

Retrieval\-augmented generation degrades when scaled to large, heterogeneous document collections, where dense similarity loses discriminative power, and top\-kkretrieval increasingly returns semantically similar but contextually incorrect chunks\. We refer to this failure mode as*vector search dilution*\. Even when using hybrid dense\+sparse retrieval, we observed this firsthand in a deployed Wyoming Department of Transportation corpus, where scaling from5454to 1,128 documents \(88,907 chunks\) reduced accuracy from75%75\\%to below40%40\\%\. To address this dilution, we propose MASDR\-RAG \( Multi\-Agent Scoped Domain Retrieval for RAG\) and evaluate it on 200 expert\-validated queries across five LLM backbones, six corpora, and two index stacks\. Our results indicate that*domain scoping using organizational metadata is the key fix*, significantly improving P@10 from0\.770\.77to0\.860\.86\(p<0\.05p<0\.05\)\. Furthermore, our investigation of multi\-agent orchestration revealed that a high degree of configuration dependence results —creating what we call the*precision–faithfulness paradox*\. Based on these varied outcomes, our practical recommendation is simple:*scope first, then perform a single synthesis call*, reserving full multi\-agent orchestration for genuinely multi\-domain corpora paired with native\-tool\-call backbones\. Code and Data will be made public upon acceptance\.

When More Documents Hurt RAG: Mitigating Vector Search Dilution with Domain\-Scoped, Model\-Agnostic Retrieval

Nabaraj Subedi1∗, Ahmed Abdelaty2, and Shivanand Venkanna Sheshappanavar11Dept\. of Electrical Engineering & Computer Science2Dept\. of Civil, Architectural Engineering & Construction ManagementUniversity of Wyoming, Laramie, WY 82071, USA\{nsubedi1, aahmed3, ssheshap\}@uwyo\.edu∗\*Correspondence author

## 1Introduction

Retrieval\-augmented generation \(RAG\) has become the dominant pattern for grounding LLM outputs in external knowledge\(Lewiset al\.,[2020](https://arxiv.org/html/2606.11350#bib.bib1); Guuet al\.,[2020](https://arxiv.org/html/2606.11350#bib.bib2); Gaoet al\.,[2024](https://arxiv.org/html/2606.11350#bib.bib3)\)\. However, the standard embed–index–retrieve–generate pipeline scales poorly on regulated enterprise corpora spanning thousands of heterogeneous documents\(Barnettet al\.,[2024](https://arxiv.org/html/2606.11350#bib.bib4); Wuet al\.,[2025](https://arxiv.org/html/2606.11350#bib.bib5)\)\. As the corpus expands across heterogeneous categories, dense retrieval loses its discriminative power\. The effect persists even when the Approximate Nearest Neighbor \(ANN\) index returns the true nearest neighbors\(Malkov and Yashunin,[2020](https://arxiv.org/html/2606.11350#bib.bib15); Johnsonet al\.,[2021](https://arxiv.org/html/2606.11350#bib.bib16)\): those neighbors are semantically related to the query yet contextually irrelevant\.

We identify and characterize*vector search dilution*, a*semantic*scaling problem\. We study this problem in the current Wyoming Department of Transportation \(WYDOT\) chatbot, where scaling the corpus from5454to 1,128 documents across nine categories reduced accuracy on Standard\-Specification queries from75%75\\%to below40%40\\%\.To address this, we developed a domain\-scoped retrieval framework,MASDR\-RAG, together with a lightweight single\-call variant,Hybrid\-Routed\.

Our experiments across five LLM backbones \(Qwen2\.5\-7B\-Instruct\(Qwen Team,[2024](https://arxiv.org/html/2606.11350#bib.bib39)\), Llama\-3\-8B\-Instruct\(Grattafiori and others,[2024](https://arxiv.org/html/2606.11350#bib.bib40)\), and three commercial backbones via OpenRouter \(Claude\-Haiku\-4\.5, GPT\-5\-mini, DeepSeek\-V3\)\), six corpora \(EnterpriseComposite\-9, HotpotQA\-distractor\(Yanget al\.,[2018](https://arxiv.org/html/2606.11350#bib.bib34)\), MULTIHOP\-RAG\(Tang and Yang,[2024](https://arxiv.org/html/2606.11350#bib.bib35)\), NQ\-Open , FinanceBench, and MMLU\-Pro\), and two index stacks \(FAISS and Neo4j HNSW\) identify domain scoping over organizational metadata as the primary driver of improved retrieval performance\. In contrast, multi\-agent orchestration produces configuration\-dependent results\. Under a Gemini production stack, it reduces RAGAS faithfulness from0\.610\.61to0\.350\.35\(p<0\.01p<0\.01\), creating what we call the precision–faithfulness paradox\. However, this effect does not reproduce under an apples\-to\-apples open\-source stack\. Through controlled ablations, we further show that this degradation is not simply a consequence of splitting retrieval into multiple calls\. Instead, it arises from the difficulty of synthesizing answers across multiple sources when the retrieved evidence contains dense, near\-duplicate passages\. These findings point to a practical design principle for large\-scale RAG systems: scope retrieval first and use a single synthesis step whenever possible\. Our contributions are threefold:

1. 1\.Diagnosis:We formalize*vector search dilution*and characterize how retrieval quality degrades as corpus density increases\.
2. 2\.Architecture and analysis:We introduce the multi\-agent retrieval frameworkMASDR\-RAGand the lightweight variantHybrid\-Routed, along with controlled ablations that isolate the sources of synthesis failures\.
3. 3\.Generalization:We evaluate across five LLMs, six corpora, and two retrieval stacks, showing that the findings are robust across models and indexing implementations while reducing costs relative to iterative ReAct\-style baselines\.

## 2Related Work

#### RAG and Dense Retrieval\.

RAG\(Lewiset al\.,[2020](https://arxiv.org/html/2606.11350#bib.bib1)\)pairs a generator with a retriever and has evolved through query transformation, re\-ranking, and iterative retrieval\(Gaoet al\.,[2024](https://arxiv.org/html/2606.11350#bib.bib3)\);*agentic*variants\(Singh and others,[2025](https://arxiv.org/html/2606.11350#bib.bib31)\)let the model decide when to retrieve\. Dense bi\-encoders\(Karpukhinet al\.,[2020](https://arxiv.org/html/2606.11350#bib.bib12)\)and late\-interaction models\(Khattab and Zaharia,[2020](https://arxiv.org/html/2606.11350#bib.bib43); Santhanamet al\.,[2022](https://arxiv.org/html/2606.11350#bib.bib44)\)largely supplanted sparse retrieval\(Robertson and Zaragoza,[2009](https://arxiv.org/html/2606.11350#bib.bib14)\), while hybrid schemes\(Sawarkaret al\.,[2024](https://arxiv.org/html/2606.11350#bib.bib18)\)stay competitive on multi\-domain corpora\. Index scaling is usually framed*algorithmically*via approximate nearest neighbors\(Malkov and Yashunin,[2020](https://arxiv.org/html/2606.11350#bib.bib15); Johnsonet al\.,[2021](https://arxiv.org/html/2606.11350#bib.bib16)\); we focus instead on a complementary*semantic*degradation\. Prior work shows dense retrieval loses discriminative power as the index grows\(Reimers and Gurevych,[2021](https://arxiv.org/html/2606.11350#bib.bib59)\), irrelevant passages alter generation\(Cuconasuet al\.,[2024](https://arxiv.org/html/2606.11350#bib.bib60)\), and long contexts introduce noise\(Jinet al\.,[2025](https://arxiv.org/html/2606.11350#bib.bib61)\)\. We share this diagnosis but contribute a*retrieval\-free, corpus\-intrinsic*measurement \(the dilution factorδ\\delta, §[3](https://arxiv.org/html/2606.11350#S3)\) along with a deployable fix, and confirm that the issue is not dense\-specific by evaluating BM25 and ColBERTv2 \(§[6](https://arxiv.org/html/2606.11350#S6)\)\.

#### Query Routing and Multi\-Agent Systems\.

Two lines of prior work contextualize our approach to domain scoping\.*Strategy routing*\(Jeonget al\.,[2024](https://arxiv.org/html/2606.11350#bib.bib19); Zhanget al\.,[2025](https://arxiv.org/html/2606.11350#bib.bib20); Guoet al\.,[2025](https://arxiv.org/html/2606.11350#bib.bib21)\)chooses a retrieval*depth*or entire*pipeline*per query—deciding when and how hard to retrieve, not where\.*Metadata filtering*\(Poliakov and others,[2024](https://arxiv.org/html/2606.11350#bib.bib22)\)masks candidates post\-hoc while still indexing the whole corpus\. Our scoping is orthogonal: we route to one ofKK*pre\-existing organizational scopes*that live in the document graph as a first\-class field \(source\_type,document\_series, articlecategory\), restricting the index at query time rather than filtering after the fact\. Our trainedR2\-Routedvariant \(App\.[A33](https://arxiv.org/html/2606.11350#A33)\) demonstrates that the choice of routing*target*matters as much as the routing*model*\.

For orchestration, ReAct\(Yaoet al\.,[2023](https://arxiv.org/html/2606.11350#bib.bib23)\)and LangChain\(Chase,[2023](https://arxiv.org/html/2606.11350#bib.bib45)\)provide general scaffolding for tool use\.*Genuinely*multi\-agent RAG assigns distinct roles with inter\-agent messaging: MA\-RAG\(Nguyen and others,[2025](https://arxiv.org/html/2606.11350#bib.bib27)\)chains task\-specific agents, and SCOUT\-RAG\(Liet al\.,[2026](https://arxiv.org/html/2606.11350#bib.bib63)\)runs cooperative domain\-relevance and retrieval agents over graph domains\. OurMASDR\-RAGis deliberately simpler—a*single reasoning agent*withKKdomain\-scoped tools, where each “agent” is a scope\-bound tool configuration\. We include both multi\-agent paradigms as baselines \(§[8](https://arxiv.org/html/2606.11350#S8)\) and show that, with commercial generators, multi\-round orchestration triggers a faithfulness collapse that is absent with open\-source backbones \(Table[10](https://arxiv.org/html/2606.11350#S10.T10)\)\.

#### Reranking, Iterative, and Graph RAG:

Two\-stage pipelines rerank a bi\-encoder top\-KKwith a cross\-encoder\(Nogueiraet al\.,[2020](https://arxiv.org/html/2606.11350#bib.bib52)\); our ablation \(App\.[A34](https://arxiv.org/html/2606.11350#A34)\) shows that while cross\-encoder reranking lifts baseline faithfulness, it does*not*recover the multi\-agent collapse, ruling out within\-scope ranking noise as its sole cause\. Learned\-sparse retrievers such as SPLADE\(Formalet al\.,[2022](https://arxiv.org/html/2606.11350#bib.bib54)\)remain competitive; we evaluate the OpenSearch neural\-sparse model\(OpenSearch Project,[2024](https://arxiv.org/html/2606.11350#bib.bib56)\)as an additional retriever baseline \(§[6](https://arxiv.org/html/2606.11350#S6)\)\. Iterative methods—IRCoT\(Trivediet al\.,[2023](https://arxiv.org/html/2606.11350#bib.bib48)\), Self\-Ask\(Presset al\.,[2023](https://arxiv.org/html/2606.11350#bib.bib49)\), and Self\-RAG\(Asaiet al\.,[2024](https://arxiv.org/html/2606.11350#bib.bib46)\)—share ReAct’s multi\-round loops, which our efficiency analysis shows are costly under open\-source backbones\. WhileShiet al\.\([2023](https://arxiv.org/html/2606.11350#bib.bib47)\)notes that LLMs are distracted by irrelevant context, we demonstrate that fragmented yet domain\-precise context is similarly harmful\.Finally, unlike GraphRAG\(Edgeet al\.,[2024](https://arxiv.org/html/2606.11350#bib.bib28)\), which builds entity–relationship graphs, we use the graph’s*organizational*metadata as explicit agent boundaries\.

#### Evaluation:

RAGAS\(Eset al\.,[2024](https://arxiv.org/html/2606.11350#bib.bib32)\)measures standard retrieval quality and faithfulness metrics\. However, standard benchmarks—such as Natural Questions\(Kwiatkowskiet al\.,[2019](https://arxiv.org/html/2606.11350#bib.bib9)\), HotpotQA\(Yanget al\.,[2018](https://arxiv.org/html/2606.11350#bib.bib34)\), MultiHop\-RAG\(Tang and Yang,[2024](https://arxiv.org/html/2606.11350#bib.bib35)\), and long\-context suites\(Yenet al\.,[2025](https://arxiv.org/html/2606.11350#bib.bib62)\)—rely on homogeneous or synthetic corpora\. Consequently, they fail to capture the cross\-domain dilution typical of a regulated enterprise environment, motivating he multi\-domain evaluation frameworks introduced in this work\.

## 3Vector Search Dilution

### 3\.1System Context

The corpus comprises 1,128 documents spanning construction specifications, design manuals, materials testing procedures, crash reports, transportation improvement programs, and administrative reports, ingested into Neo4j asDocument→Section→Chunk\\text\{Document\}\\rightarrow\\text\{Section\}\\rightarrow\\text\{Chunk\}\. The production system uses Gemini Embedding \(768\-d\), HNSW, and a BM25 full\-text index\. Traffic & Crash reports contribute34\.8%34\.8\\%chunks despite being1\.9%1\.9\\%documents \(Table[1](https://arxiv.org/html/2606.11350#S3.T1)\)\.

CategoryDocsChunks%Chk/DocStandard Specs22,5192\.81,260Construction Manual216,6417\.5316Materials Testing62,1802\.5363Design Manual231,4051\.661Traffic & Crashes2230,92234\.81,406STIP5913,63415\.3231Annual Reports462,3412\.651Bridge Program285,3996\.1193Other92123,86626\.826Total1,12888,907100—

Table 1:Document and chunk distribution by literaldocument\_seriescategory\. Agent scope filters \(App\.[A11](https://arxiv.org/html/2606.11350#A11)\) span broader related\-series unions, so the per\-agent counts in Table[13](https://arxiv.org/html/2606.11350#A11.T13)exceed the per\-category counts \. Chunk density varies54×54\\timesacross categories\.
### 3\.2Formal Definition

Let𝒞=\{c1,…,cN\}\\mathcal\{C\}\{=\}\\\{c\_\{1\},\\dots,c\_\{N\}\\\}beNNchunks partitioned intoKKcategories𝒞1,…,𝒞K\\mathcal\{C\}\_\{1\},\\dots,\\mathcal\{C\}\_\{K\},e:𝒞→ℝde:\\mathcal\{C\}\\to\\mathbb\{R\}^\{d\}an embedding, andqqa query targeting categoryk⋆k^\{\\star\}\. The top\-mmretrieval set isRm​\(q\)=arg⁡maxS⊆𝒞,\|S\|=m​∑c∈Ssim​\(e​\(q\),e​\(c\)\)R\_\{m\}\(q\)=\\arg\\max\_\{S\\subseteq\\mathcal\{C\},\|S\|=m\}\\sum\_\{c\\in S\}\\mathrm\{sim\}\(e\(q\),e\(c\)\)\. Dilution occurs when global precision is much lower than scoped precision:

δ​\(q,k⋆\)=1−Pglobal​\(q\)Pscoped​\(q\),\\delta\(q,k^\{\\star\}\)\\;=\\;1\-\\frac\{P\_\{\\text\{global\}\}\(q\)\}\{P\_\{\\text\{scoped\}\}\(q\)\},wherePglobal​\(q\)P\_\{\\text\{global\}\}\(q\)is the fraction of the retrieval setRm​\(q\)R\_\{m\}\(q\)belonging to the target categoryk⋆k^\{\\star\}when retrieval ranges over all of𝒞\\mathcal\{C\}, andPscoped​\(q\)P\_\{\\text\{scoped\}\}\(q\)is the same fraction when retrieval is restricted to𝒞k⋆\\mathcal\{C\}\_\{k^\{\\star\}\}\(soPscoped≈1P\_\{\\text\{scoped\}\}\{\\approx\}1by construction\)\. Thusδ=0\\delta\{=\}0is no dilution andδ→1\\delta\\to 1severe dilution\.

### 3\.3Empirical Measurements

Categories with smaller chunk populations suffer the most severe dilution \(Designδ=0\.53\\delta\{=\}0\.53; Specsδ=0\.43\\delta\{=\}0\.43\), while high\-density categories \(Construction Manualδ=0\.10\\delta\{=\}0\.10\) largely resist it\. The Spearman correlation betweenlog⁡\(chunk count\)\\log\(\\text\{chunk count\}\)and meanδ\\deltaacross the eight scopable categories isρ=−0\.60\\rho\{=\}\{\-\}0\.60\(p=0\.12p\{=\}0\.12\)\. Withn=8n\{=\}8categories, this single correlation is suggestive rather than statistically conclusive on its own; we corroborate it on the reproducible cross\-DOT replication of §[9](https://arxiv.org/html/2606.11350#S9), where the same correlation under the open\-source BGE\-M3 stack ranges fromρ=−0\.68\\rho\{=\}\{\-\}0\.68\(WYDOT,1010categories\) toρ=−0\.95\\rho\{=\}\{\-\}0\.95\(CDOT,1010categories\)\.

Table 2:Per\-category dilution factor with per\-query ranges\.![Refer to caption](https://arxiv.org/html/2606.11350v1/x1.png)Figure 1:Dilutionδ\\deltavs\. chunk count, eight WYDOT scopes; Spearmanρ=−0\.60\\rho=\{\-\}0\.60\(p=0\.12p\{=\}0\.12\)\.scoped retrieval:8585–98%98\\%search\-space reductionUserQueryHybrid Router\(Regex→\\toLLM\)Orchestrator\(function\-calling\)Specs Agent\|𝒞\|≈2\.5\|\\mathcal\{C\}\|\{\\approx\}2\.5kConstruction Agent\|𝒞\|≈6\.6\|\\mathcal\{C\}\|\{\\approx\}6\.6kMaterials Agent\|𝒞\|≈2\.2\|\\mathcal\{C\}\|\{\\approx\}2\.2kDesign Agent\|𝒞\|≈1\.4\|\\mathcal\{C\}\|\{\\approx\}1\.4kTraffic & Crashes Agent\|𝒞\|≈30\.9\|\\mathcal\{C\}\|\{\\approx\}30\.9kBridge Agent\|𝒞\|≈5\.4\|\\mathcal\{C\}\|\{\\approx\}5\.4kSTIP Agent\|𝒞\|≈13\.6\|\\mathcal\{C\}\|\{\\approx\}13\.6kAnnual Reports Agent\|𝒞\|≈2\.3\|\\mathcal\{C\}\|\{\\approx\}2\.3kHighway Safety Agent\|𝒞\|≈30\.9\|\\mathcal\{C\}\|\{\\approx\}30\.9kNeo4jKnowledgeGraphSynthesiser LLM\(Qwen\-7B / Llama\-8B / Gemini\)Answercategoryscoped ANN searchtop\-kkchunkschunks \+ query

Figure 2:MASDR\-RAG/Hybrid\-Routeddata flow\. Regex\-then\-LLM router dispatches to one of nine WYDOT domain agents; each agent ANN\-searches itsdocument\_seriesscope in the Neo4j graph, and a Qwen\-7B / Llama\-8B / Gemini synthesizer generates the answer\.Geometrically, the dilution corresponds to the retrieval\-time source confusion: on Composite\-9, the diagonal ofP​\(retrieved source∣gold source\)P\(\\text\{retrieved source\}\\mid\\text\{gold source\}\)is only0\.590\.59under monolithic search, lifting to0\.840\.84under regex scoping and0\.900\.90underHybrid\-Routed\(Figure[5](https://arxiv.org/html/2606.11350#A6.F5), App\.[A27](https://arxiv.org/html/2606.11350#A27)\)\. Scoping does not improve the embedder; it forces the retrieval neighborhood to respect the source label already present in the document graph\. A t\-SNE projection \(App\.[A6](https://arxiv.org/html/2606.11350#A6)\) and a worked WYDOT failure case \(App\.[A26](https://arxiv.org/html/2606.11350#A26)\) illustrate the same mechanism\.

## 4Architecture: MASDR\-RAG and Hybrid\-Routed

The architecture has three components: \(1\)Domain\-scoped retrieval, where each agent restricts the search to documents matching a Neo4j metadata filter, reducing the effective search space by8585–98%98\\%\(Table[13](https://arxiv.org/html/2606.11350#A11.T13)\), \(2\)Hybrid routingthat runs a fast regex matcher first and falls back to a zero\-shot classifier using an LLM, and \(3\)Multi\-agent orchestrationthat dispatches to nine domain agents via function calling\. Figure[2](https://arxiv.org/html/2606.11350#S3.F2)summarizes the data flow\.

Each agent’s scope filter reduces its effective search space by6565–98%98\\%relative to the full corpus, with a weighted average of90\.4%90\.4\\%\(per\-agent breakdown in App\.[A13](https://arxiv.org/html/2606.11350#A13), Table[13](https://arxiv.org/html/2606.11350#A11.T13)\)\. The orchestrator uses up to five tool\-call rounds;Hybrid\-Routeduses at most two LLM calls per query \(one router, one synthesizer\)\.

We use*orchestration*for this multi\-round tool loop and are explicit about what it is not:MASDR\-RAGis a*single reasoning agent*withKKdomain\-scoped retrieval tools, and the per\-domain “agents” are scope\-bound tool configurations rather than autonomous agents that reason or communicate independently\. The contrast with genuinely multi\-agent RAG — where separate planner, extractor, and synthesis agents exchange intermediate reasoning — is drawn against the MA\-RAG and SCOUT\-RAG baselines in §[2](https://arxiv.org/html/2606.11350#S2)and §[8](https://arxiv.org/html/2606.11350#S8)\.

## 5Evaluation: Proprietary Stack

200 expert\-validated WYDOT queries \(Gemini 2\.5 Flash answer generator,95%95\\%bootstrap CIs, permutation tests atα=0\.05\\alpha\{=\}0\.05\)\.

#### Metrics:

We report four metrics throughout\.P@10andR@10are precision and recall at rank1010, computed against the expert\-labeled target scope of each query: a retrieved chunk counts as relevant if it belongs to that scope\.Correctness\(Corr\) is a binary per\-answer judgment — an LLM judge \(Qwen\-2\.5\-7B, distinct from every system under test\) marks each generated answer as correct or incorrect against the reference answer, and we report the mean; the judge prompt and rubric are in App\.[A10](https://arxiv.org/html/2606.11350#A10)and App\.[A18](https://arxiv.org/html/2606.11350#A18)\.Faithfulness\(Faith\) is the RAGAS faithfulness score in\[0,1\]\[0,1\], the fraction of claims in the generated answer that are supported by the retrieved context\. Unless noted,nnin a table is the number of queries scored\.

Table 3:WYDOT Gemini stack \(n=200n\{=\}200\): scoping lifts P@10\.77→\.86\.77\{\\to\}\.86;MASDR\-RAG’s faithfulness collapses\.61→\.35\.61\{\\to\}\.35\.p∗<\.05\{\}^\{\*\}p\{<\}\.05,p∗∗<\.01\{\}^\{\*\*\}p\{<\}\.01vs\. monolithic\.

## 6Open\-Source Reproducibility

We re\-ran all five systems with Qwen2\.5\-7B\-Instruct and Llama\-3\-8B\-Instruct synthesizers on BGE\-M3\(Chenet al\.,[2024](https://arxiv.org/html/2606.11350#bib.bib41)\)embeddings; a single L40S GPU handles the200200\-query sweep in≈40\\approx\\\!40min \(Qwen\) /≈100\\approx\\\!100min \(Llama\)\.