扫描还是索引？长文档 RAG 中的大规模证据覆盖、回答质量与成本

# Scan or Index?Evidence Coverage, Answer Quality, and Cost in Long-Document RAG
Source: [https://arxiv.org/html/2610.28957](https://arxiv.org/html/2610.28957)
Diana\-Margarita Córdova\-EsparzaAffiliation:Facultad de Informática, Universidad Autónoma de Querétaro, MéxicoJulio\-Alejandro Romero\-GonzálezAffiliation:Facultad de Informática, Universidad Autónoma de Querétaro, MéxicoEdgar Chavez\-UrbiolaAffiliation:CICATA Qro\., Instituto Politécnico Nacional, MéxicoJuan Irving VasquezAffiliation:CIDETEC, Instituto Politécnico Nacional, MéxicoJulian Andres Ramirez\-BautistaAffiliation:Fundación Universitaria de San Gil \- Unisangil, ColombiaSilvia L\. Chaparro\-CárdenasAffiliation:Universidad de Investigación y Desarrollo, ColombiaGendry Alfonso\-FranciaAffiliation:ENES, Universidad Nacional Autónoma de México

October 9, 2026

###### Abstract

Scanning every passage in a long document seems thorough, but does it preserve more useful evidence or produce better answers? We compare 11 retrieval configurations spanning lexical and dense search, hybrid reranking, and full\-document scanning with Jev, OpenAI Decisions, and Cloudflare Clef\. Our evaluation uses 396 QASPER questions and 80 questions from eight complete textbooks, with common passages, a five\-passage limit, and a fixed answer generator\. On books, Hybrid \+ Jev achieves the highest observed evidence coverage \(94\.1%\), compared with 88\.7% for Jev full scan and 81\.3% for OpenAI dense retrieval\. Full scans examine every passage initially, yet later reduction steps can discard relevant evidence\. Higher evidence coverage also does not consistently produce better answers: Hybrid \+ Decisions achieves the highest observed book Answer Accuracy among retrieval methods \(0\.892\), despite recovering less reference evidence\. Full scanning incurs substantially higher latency on books under the tested deployments, while reusable dense retrieval has lower projected API cost than paid decision\-based methods as queries accumulate\. Most QASPER articles already fit within the hybrid candidate pool, limiting the benefit of scanning\. These results highlight evidence retention, answer quality, and reuse cost as complementary criteria for choosing a long\-document retrieval strategy\.

## 1Introduction

Answering a question about a textbook can mean finding a few useful passages among hundreds of pages\. Retrieval\-augmented generation \(RAG\) supplies such passages to a language model before it writes an answer\[[2](https://arxiv.org/html/2610.28957#bib.bib2),[35](https://arxiv.org/html/2610.28957#bib.bib35)\]\. How those passages are found affects both the available evidence and the work repeated for every question\. Dense retrieval encodes passages in advance and reuses their representations across queries\[[26](https://arxiv.org/html/2610.28957#bib.bib26)\]\. Its preparation cost is therefore spread across later searches\.

Decision models offer another way to select evidence\. Jev chooses among supplied options and can use passages as those options\[[10](https://arxiv.org/html/2610.28957#bib.bib10),[12](https://arxiv.org/html/2610.28957#bib.bib12)\]\. It can rerank candidates from an existing index or examine every passage for each new query\. The latter avoids precomputing passage embeddings, but long documents must be processed in windows and reduced to a final selection\. Evidence can disappear during this reduction even when every passage was examined initially\. The practical question is whether broader access improves the final evidence enough to justify the repeated work\.

We examine this tradeoff within a specified document using 396 QASPER questions with human references and 80 questions about eight complete textbooks\. The original study compares seven lexical, dense, hybrid, and Jev\-based methods with common passage boundaries and a five\-passage retrieval limit\. Separate follow\-up evaluations add OpenAI Decisions\[[13](https://arxiv.org/html/2610.28957#bib.bib13)\]and local Cloudflare Clef\[[1](https://arxiv.org/html/2610.28957#bib.bib1)\]\. Almost all sampled articles fit inside the shared 50\-candidate pool, whereas every book requires substantial preselection or reduction\. The two corpora, therefore, test different retrieval conditions\.

Our contribution is an empirical comparison that connects retained evidence, reduction losses, and cost under document reuse\. We locate reference evidence in the source, trace what survives each selection stage, and account for preparation and query API charges\. A fixed answer generator provides a separate test of whether retrieval differences carry through to automated answer scores\. On the book sample, Hybrid \+ Jev retains the most annotated evidence on average while requiring much less query time than Jev full scanning in the measured deployment\. The results make candidate retention and document reuse central to the comparison, while keeping evidence coverage distinct from answer correctness\.

### 1\.1Related work

Passage retrieval ranges from term matching to neural representations\. BM25 provides a lexical baseline\[[24](https://arxiv.org/html/2610.28957#bib.bib24)\], while Sentence\-BERT, Dense Passage Retrieval, and Contriever support similarity search with independently encoded text\[[25](https://arxiv.org/html/2610.28957#bib.bib25),[26](https://arxiv.org/html/2610.28957#bib.bib26),[28](https://arxiv.org/html/2610.28957#bib.bib28)\]\. ColBERT retains token\-level representations for later interaction with the query\[[27](https://arxiv.org/html/2610.28957#bib.bib27)\]\. BGE\-M3 supports dense, sparse, and multi\-vector retrieval\[[29](https://arxiv.org/html/2610.28957#bib.bib29)\]; our experiments use its dense mode\. Reciprocal rank fusion combines ranked lists without requiring their scores to share a scale\[[30](https://arxiv.org/html/2610.28957#bib.bib30)\]\. These methods provide established ways to reuse document representations against which repeated passage scoring can be compared\.

Reranking spends additional computation on a candidate set\. Unsupervised Passage Re\-ranking \(UPR\) scores passages using the likelihood of generating the question\[[33](https://arxiv.org/html/2610.28957#bib.bib33)\]\. RankGPT uses language models for listwise ranking\[[3](https://arxiv.org/html/2610.28957#bib.bib3)\], pairwise ranking prompting compares two passages at a time\[[34](https://arxiv.org/html/2610.28957#bib.bib34)\], and ListT5 uses bounded comparisons with tournament selection\[[4](https://arxiv.org/html/2610.28957#bib.bib4)\]\. These studies establish model\-based passage comparison as a retrieval component\. Our focus is how reference evidence survives when such selection is repeated across a full document\.

Long\-document methods also differ in what they present to the model\. RAPTOR builds a hierarchy of clustered summaries\[[5](https://arxiv.org/html/2610.28957#bib.bib5)\], while STAIR uses document structure to retrieve relevant sections\[[6](https://arxiv.org/html/2610.28957#bib.bib6)\]\. LongBench and RULER evaluate long\-context capabilities across tasks and input lengths\[[37](https://arxiv.org/html/2610.28957#bib.bib37),[38](https://arxiv.org/html/2610.28957#bib.bib38)\], and Lost in the Middle shows that evidence position can affect its use\[[36](https://arxiv.org/html/2610.28957#bib.bib36)\]\. Comparisons of RAG and long\-context generation find that quality and cost depend on the model and task\[[39](https://arxiv.org/html/2610.28957#bib.bib39),[7](https://arxiv.org/html/2610.28957#bib.bib7)\]\. Our full scan repeatedly selects original passages and supplies only the finalists to the generator; it does not give the generator the entire document\.

Prior Jev studies include a community evaluation on a six\-book reconstruction of SearchTome and a Turkish XQuAD report with a full\-corpus hierarchical ablation\[[8](https://arxiv.org/html/2610.28957#bib.bib8),[9](https://arxiv.org/html/2610.28957#bib.bib9)\]\. An academic evaluation examines Jev across classification, reasoning, and other decision tasks\[[10](https://arxiv.org/html/2610.28957#bib.bib10)\]\. The community reports provide practical precedents, with different datasets and protocols\. We build on this work by measuring how much annotated source text remains, tracing reduction losses, and including preparation costs when estimating API charges for repeated queries\.

Evaluation must also distinguish retrieval from answer generation\. BEIR and MTEB demonstrate the need to evaluate retrieval and embeddings across tasks\[[31](https://arxiv.org/html/2610.28957#bib.bib31),[32](https://arxiv.org/html/2610.28957#bib.bib32)\]; QASPER provides human questions, answers, and supporting evidence within research papers\[[14](https://arxiv.org/html/2610.28957#bib.bib14)\]\. The RGB benchmark examines evidence integration, noise robustness, and rejection of unsupported questions\[[40](https://arxiv.org/html/2610.28957#bib.bib40)\]\. RAGAs and RAGBench assess complementary properties of retrieved context and generated answers\[[15](https://arxiv.org/html/2610.28957#bib.bib15),[17](https://arxiv.org/html/2610.28957#bib.bib17)\], while ALCE evaluates whether answers are supported by citations\[[41](https://arxiv.org/html/2610.28957#bib.bib41)\]\. Our source\-span metric measures recovery of annotated evidence\. It can miss alternative supporting passages, and neither this metric nor our answer scores verifies citation correctness\.

Reference provenance further affects what an evaluation can establish\. mtRAG includes human conversations and unanswerable questions\[[18](https://arxiv.org/html/2610.28957#bib.bib18)\], while work on synthetic RAG evaluation finds that agreement with human evaluation depends on the setting\[[19](https://arxiv.org/html/2610.28957#bib.bib19)\]\. ARES uses human annotations to correct automated estimates\[[20](https://arxiv.org/html/2610.28957#bib.bib20)\]\. Model judgments can also exhibit self\-preference and position bias\[[22](https://arxiv.org/html/2610.28957#bib.bib22),[23](https://arxiv.org/html/2610.28957#bib.bib23)\]\. We therefore report articles with human references separately from textbooks with frontier model\-generated questions and interpret answer scores within the limits of model\-based evaluation\.

## 2Experimental design

We follow evidence from the source document to the final answer\. Retrieval methods share the same documents, questions, passages, and answer generator, allowing us to compare evidence retention, latency, API cost, and answer quality\. Experimental code, configurations, and aggregate results are available at[https://github\.com/jrterven/jev\-vs\-rag](https://github.com/jrterven/jev-vs-rag)\.

### 2\.1Data and retrieval setup

We use 396 QASPER questions from 264 research articles: 320 answerable and 76 unanswerable\[[14](https://arxiv.org/html/2610.28957#bib.bib14)\]\. Selected questions have consistent answerability annotations\. Answerable cases must include a complete set of textual evidence that can be located in the source; ambiguous matches and evidence requiring images or tables are excluded\. Each answerable question keeps a fixed human reference answer for the later answer evaluation\.

The book set contains 80 questions across eight complete OpenStax textbooks, with four factual, three explanatory, and three multi\-section questions per book\. These questions and reference answers were primarily generated with the frontier model GPT\-Astra from relevant sections and neighboring text, without access to retrieval outputs or evaluation scores\. Supporting quotations were checked against the source\. These checks establish textual alignment, rather than independent human validation\. A separate development set contains 60 QASPER training questions and 20 questions from two additional textbooks\. Sources and references were fixed before testing\. Rephrasing questions does not rule out training\-data overlap\[[21](https://arxiv.org/html/2610.28957#bib.bib21)\]\.

Retrieval takes place within the specified document\. We process the complete book text, including available figure captions and alternative text\. All methods use thecl100k\_basetokenizer to form passages of approximately 500 tokens with 75\-token overlap, and return up to five passages\. Articles contain 2–70 passages \(median 11\), compared with 662–2,245 per book\. Almost all articles therefore fit inside the shared pool of up to 50 hybrid candidates, while every book requires substantial preselection or reduction\.

Figure[1](https://arxiv.org/html/2610.28957#S2.F1)outlines the three retrieval routes; Table[1](https://arxiv.org/html/2610.28957#S2.T1)summarizes the 11 configurations\. Jev, Decisions API, and Clef are each evaluated in two settings: selecting from the shared pool of up to 50 hybrid candidates, and scanning all passages before reducing them to the final results\.

Table 1:The 11 retrieval configurations\. All use common passages and return up to five results; hybrid rerankers receive the same candidate pool\.Document \+ questionCommon passages500 tokens, 75 overlapIndexed retrievalReusable index→\\rightarrowrankHybrid \+ rerankingRRF→\\rightarrowup to 50 candidates→\\rightarrowrerankFull scanAll passages→\\rightarrowwindows→\\rightarrowreduceUp to 10 candidates per windowSelectedpassagesUp to fiveFixed answergeneratorFigure 1:The three retrieval routes share a document, question, and passage boundaries\. Indexed retrieval ranks from a reusable index; hybrid reranking selects from a common pool of up to 50 candidates; full scanning considers every passage before hierarchical reduction\. Each route returns up to five passages, packed under the same 2,500\-token budget for the fixed generator\.We use Jev \(jev\-1\.13\.0\), OpenAI Decisions API \(gpt\-6\-luna\), and the 27BCloudflare/clefmodel, with Clef running locally\[[11](https://arxiv.org/html/2610.28957#bib.bib11),[13](https://arxiv.org/html/2610.28957#bib.bib13),[1](https://arxiv.org/html/2610.28957#bib.bib1)\]\. All three process passages in windows with a 12,000\-token request limit, including overhead\. Jev uses conservative token estimates, Decisions counts its serialized requests, and Clef checks both native and common\-token counts\. Decisions also includes an explicit no\-match option, which is excluded from the returned passages\.

Each window keeps up to ten candidates\. Their complete text is evaluated again in successive rounds until a final comparison returns up to five passages\. Scores are used within each comparison and are not treated as global scores across windows\[[12](https://arxiv.org/html/2610.28957#bib.bib12)\]\. Clef uses the probabilities from its released implementation, with source\-order tie breaking\. In all three full\-scan variants, every passage is considered initially, but later reductions can still discard relevant evidence\.

### 2\.2Evidence, latency, and API cost

Evidence coverage measures how much annotated support appears in the retrieved passages\. LetRkR\_\{k\}be the union of source\-token intervals covered by the firstkkpassages, and letSjS\_\{j\}be one complete alternative set of reference evidence units:

EC​@​k=maxj⁡1\|Sj\|​∑e∈Sj\|e∩Rk\|\|e\|\.\\mathrm\{EC\}@k=\\max\_\{j\}\\frac\{1\}\{\|S\_\{j\}\|\}\\sum\_\{e\\in S\_\{j\}\}\\frac\{\|e\\cap R\_\{k\}\|\}\{\|e\|\}\.Evidence units receive equal weight, overlapping tokens count once, and units from different alternatives are never combined\. Complete evidence recovery requires one entire alternative set\. An exact oracle gives the best attainable coverage with up to five passages\. These metrics apply to answerable questions; failed assigned retrievals receive zero coverage\. For Jev, Decisions, and Clef, reduction traces track evidence through the initial candidate pool, first selection, final candidate pool, and returned passages\.

We average scores within each document and then across documents\. Paired differences use the same questions and 10,000 bootstrap resamples of whole documents\. We report descriptive pointwise 95% intervals without adjustment for multiple comparisons\.

We report median and 95th\-percentile latency over valid completed executions\. The comparison among Jev, Decisions, and Clef uses the first retrieval execution for each question in the full test set\. Hybrid times cover reranking of the shared candidate pool, excluding candidate generation; full\-scan times cover passage selection and reduction\. Clef latency summaries use these single executions\. End\-to\-end measurements for BM25, dense and hybrid baselines, Jev, and Decisions also cover a 100\-question subset, equally divided between corpora, with three assigned executions per question and randomized method order\. These measurements include query embeddings, candidate generation, and ranking\. Jev summaries pool valid repetitions; Decisions summaries distinguish exclusive and shared GPU availability\.

Document preparation and answer generation are timed separately\. Measured retrieval uses no automatic retries or query\-result cache\. BGE models run on an NVIDIA GB10 in FP32 with batch size eight\. Clef runs on a separate shared GB10 in BF16, with one forward pass at a time; Jev and Decisions use two concurrent API calls\. Latencies describe these deployments and their hardware, precision, concurrency, and service conditions\.

API charges use reported consumption and the prices recorded during execution\. LetPmP\_\{m\}be the mean preparation cost per document andQm¯\\overline\{Q\_\{m\}\}the mean cost per assigned base query for methodmm\. ForNNqueries, total and average cost are projected as

Cm​\(N\)=Pm\+N​Qm¯,Am​\(N\)=Cm​\(N\)N\.C\_\{m\}\(N\)=P\_\{m\}\+N\\overline\{Q\_\{m\}\},\\qquad A\_\{m\}\(N\)=\\frac\{C\_\{m\}\(N\)\}\{N\}\.Query costs are averaged within documents and then across documents, including unanswerable queries and recorded charges from failed attempts\. When consumption is unknown, the confirmed subtotal is reported as a lower bound\. These curves project repeated use from measured calls\. Clef inference incurs no API charge\. Generation and judging are accounted for separately, and local computing resources remain unpriced\.

### 2\.3Answer evaluation

We evaluate answers on a fixed 200\-question sample: 100 answerable and 40 unanswerable QASPER questions, plus 60 book questions\. We usegpt\-4\.1\-mini\-2025\-04\-14to generate answers for all 11 retrieval configurations, with temperature zero and a 512\-token output limit\. Two controls provide either no evidence, explicitly allowing prior knowledge, or the reference evidence\.

Retrieved passages receive citation labels, and overlapping source text is deduplicated\. Whole passage contributions are packed in rank order within a 2,500\-token content limit, including separators\. Contributions that do not fit are omitted, so the generation context may contain fewer than five passages\. We measure this packing loss separately\. Grounded prompts instruct the generator to abstain when the supplied evidence is insufficient\. Reference\-evidence controls that exceed the limit are marked unavailable\.

We evaluate answers with RAGAS 0\.4\.3 Faithfulness and Answer Accuracy, usinggpt\-6\.1\-solas judge without revealing method names\[[15](https://arxiv.org/html/2610.28957#bib.bib15),[16](https://arxiv.org/html/2610.28957#bib.bib16)\]\. Faithfulness measures support for answer claims in the actual generation context; abstentions, empty answers, and no\-evidence controls are excluded\. Answer Accuracy measures agreement with the fixed reference through two normalized rubric judgments, both of which must complete\. It includes abstaining answers to answerable questions; unanswerable cases are assessed through abstention\.

Missing answers, failed scores, and inapplicable metrics remain separate from numerical scores\. Answer Accuracy is a rubric score, and neither metric establishes the correctness of individual citation labels\. Development checks assess metric behavior on controlled examples; the answer evaluation remains automated, without independent human assessment\.

## 3Results

Across the 11 methods, the clearest differences appear on books\. Hybrid \+ Jev achieves the highest observed evidence coverage, while Hybrid \+ Decisions obtains the highest observed Answer Accuracy among retrievers\. Full scanning does not consistently improve either result on books and takes substantially longer\.

### 3\.1Evidence coverage and reduction

Table[2](https://arxiv.org/html/2610.28957#S3.T2)reports how much reference evidence reaches the final five passages\. On QASPER, Jev, Decisions, and Clef all exceed 96% coverage in both hybrid and full\-scan configurations, compared with 90\.4% for OpenAI dense retrieval\. Both Clef variants reach 98\.8%\. However, the hybrid candidate pool already contains the complete article in 395 of 396 questions, including 319 of the 320 answerable cases\. QASPER therefore mainly tests passage selection; it offers little separation between preselection and full scanning\.

Table 2:Evidence coverage in the final five passages, averaged within documents and then across documents\. QASPER includes 320 answerable questions from 230 articles; books include 80 questions from eight textbooks\. Failed retrievals receive zero coverage\. Brackets show pointwise 95% document\-bootstrap intervals\. Bold marks the highest observed mean in each corpus\.Books expose a different tradeoff\. Hybrid \+ Jev reaches 94\.1% coverage, compared with 88\.7% for Jev full scan, 76\.6% for Decisions full scan, and 91\.6% for Clef full scan\. Failures contribute to these differences: Jev and Decisions fail on four and eight book scans, respectively, while Clef completes all 80\. Among the 76 book questions completed by both Jev variants, hybrid coverage is 94\.9% and full\-scan coverage is 93\.2%\.

Full scans can still improve on dense retrieval\. On books, the paired coverage gains over OpenAI dense are 7\.4 percentage points for Jev \(95% interval: \[2\.0, 12\.8\]\) and 10\.3 points for Clef \(\[5\.6, 14\.8\]\)\. Yet Clef’s advantage over Jev full scan is uncertain: 2\.9 points \[−0\.6\-0\.6, 8\.1\]\. The results support gains over dense retrieval without establishing one full\-scan method as best across comparisons\.

The reduction traces \(Figure[2](https://arxiv.org/html/2610.28957#S3.F2)\) explain why reading every passage is insufficient\. Across 76 completed Jev book scans, coverage falls from 100% initially to 99\.4% after the first selection, 95\.7% before the final comparison, and 93\.2% in the final five\. Most loss occurs in intermediate reductions\. Clef retains 98\.8% before the final comparison but ends at 91\.6%, placing most loss in the last selection\. Decisions ends at 85\.2% across its 72 completed scans\. These traces describe completed scans; Table[2](https://arxiv.org/html/2610.28957#S3.T2)also includes failed attempts\.

Figure 2:Evidence retained through completed textbook full scans: Jev 76, Decisions 72, and Clef 80\. Curves are document\-macro means over each method’s completed cases, not a common successful subset\. Failed scans are excluded here but included in Table[2](https://arxiv.org/html/2610.28957#S3.T2)\. Initial access to every passage does not prevent loss during reduction\.
### 3\.2Latency and API cost

On books, hybrid reranking is much faster than full scanning for all three decision models \(Table[3](https://arxiv.org/html/2610.28957#S3.T3)\)\. Median reranking takes 1\.72 s with Jev, 4\.51 s with Decisions, and 54\.69 s with local Clef\. Full scans take 25\.10 s, 63\.36 s, and 23\.3 min, respectively\. On QASPER, hybrid and full\-scan times are nearly identical within each model, consistent with most articles fitting in the hybrid candidate pool\.

Table 3:Median and 95th\-percentile latency using the first execution of each assigned query, with failures excluded\. Completed counts are relative to assigned queries\. Hybrid times cover reranking only, excluding candidate generation; full\-scan times include selection and reduction\. Jev and Decisions allow two concurrent API calls\. Clef runs serially in BF16 on a shared GB10 and includes local logging\. Measurements describe these deployments, with different dates and hardware conditions\.The repeated end\-to\-end benchmark provides timing context for the retrieval baselines\. On its 50 book questions with three assigned runs each, BM25, BGE\-M3 dense, and Hybrid RRF have median retrieval times below 0\.05 s; OpenAI dense and Hybrid \+ BGE take 0\.56 s and 2\.05 s\. Document preparation is separate: the shared local indexes take a mean of 56\.66 s per book, and OpenAI document embeddings take 28\.98 s\. This work is reused across queries\.

Reuse strongly favors dense retrieval in API spending \(Figure[3](https://arxiv.org/html/2610.28957#S3.F3); Table[4](https://arxiv.org/html/2610.28957#S3.T4)\)\. For 1,000 queries per book, OpenAI dense projects to US$0\.01351 including preparation\. Hybrid \+ Jev costs at least US$2\.52 and Jev full scan at least US$64\.43; the corresponding Decisions projections are US$6\.63 and at least US$155\.85\. Under the measured book\-query mix, dense retrieval becomes cheaper than Hybrid \+ Jev by the sixth query per document, even against the hybrid’s lower bound based on confirmed charges\. Unresolved charges could move this crossover earlier\.

Figure 3:Projected cumulative retrieval API cost per book,C⁡\(N\)=P\+N​Q¯C\(N\)=P\+N\\overline\{Q\}, from measured preparation and query costs\. Broken lines are known\-charge lower bounds\. Dense and Hybrid \+ Jev cross atN≈5\.21N\\approx 5\.21, so dense is cheaper by the sixth query; unknown charges could move the crossover earlier\. Both axes are logarithmic\. Clef’s zero API charges are noted separately; generation, judging, and unpriced local resources are excluded\.Table 4:Projected retrieval API cost in US dollars for 1,000 queries per document, including preparation:C⁡\(1000\)=P\+1000​Q¯C\(1000\)=P\+1000\\overline\{Q\}\. Query costs average all assigned base queries within each document and then across documents\. Unknown consumption produces lower bounds, rounded down\. Clef and the other local retrieval methods incur no API charge; local computing resources are unpriced\. Generation and judging are excluded\.
### 3\.3Answer quality

More reference evidence does not consistently produce better answers \(Figure[4](https://arxiv.org/html/2610.28957#S3.F4)\)\. Table[5](https://arxiv.org/html/2610.28957#S3.T5)reports automated scores for the common answer sample\. On books, Hybrid \+ Decisions has the highest observed Answer Accuracy among retrievers \(0\.892\), followed by Decisions full scan \(0\.888\) and Jev full scan \(0\.874\)\. Hybrid \+ Jev leads evidence coverage but scores 0\.864 on Answer Accuracy\.

Figure 4:Textbook evidence coverage versus automatic Answer Accuracy\. EC@5 uses all 80 retrieval assignments; Answer Accuracy uses 53–60 successfully scored responses from 60 generation assignments per method\. Both axes show document\-macro means over different eligible subsets\. Points are descriptive, not paired question\-level observations; uncertainty in method differences is assessed by the paired comparisons in Table[11](https://arxiv.org/html/2610.28957#A3.T11)\.Table 5:Document\-macro automated answer scores on 100 answerable QASPER questions and 60 book questions assigned per method\. AA: Answer Accuracy; FA: Faithfulness\. Brackets show scored/requested evaluations\. Missing answers and failed evaluations are excluded from these means; FA also requires supplied evidence and a non\-abstaining answer\. Bold marks the highest observed mean among retrievers, excluding controls\.Paired comparisons support some book improvements over dense retrieval\. Hybrid \+ Decisions gains 0\.057 in Answer Accuracy \(95% interval: \[0\.019, 0\.103\]; 60 jointly scored questions\), while Jev full scan gains 0\.048 \(\[0\.003, 0\.098\]; 57 questions\)\. Hybrid \+ Decisions does not clearly exceed Hybrid \+ Jev: their difference is 0\.028 \[−0\.005\-0\.005, 0\.062\]\. Clef full scan improves evidence coverage over dense retrieval, but its Answer Accuracy difference is only 0\.011 \[−0\.013\-0\.013, 0\.035\]\. These intervals use common scored questions and eight book clusters; they are pointwise comparisons without adjustment for multiple testing\.

The large coverage gains on QASPER are not accompanied by a clear improvement in answer scores\. Jev full scan differs from dense retrieval by−0\.011\-0\.011in paired Answer Accuracy \[−0\.051\-0\.051, 0\.030\], and Clef full scan by 0\.021 \[−0\.018\-0\.018, 0\.062\]\. The two Clef variants return identical ordered passages on every QASPER query, yet their answer scores differ\. That difference cannot be attributed to retrieval: generation and judging use separate calls\.

Two checks help interpret the answer scores\. First, the no\-evidence control scores 0\.808 on books but only 0\.052 on QASPER, showing that a high book score alone does not establish a strong dependence on retrieved evidence\. Second, context packing can remove evidence after retrieval: across 57 completed Jev full\-scan book answers, coverage falls from 92\.0% in the retrieved passages to 90\.4% in the supplied context\.

Abstention remains limited\. On the 40 unanswerable QASPER questions, the six Jev, Decisions, and Clef configurations abstain in only 11–14 cases, compared with 17 for OpenAI dense and all 40 for the reference\-context control\. High coverage on answerable questions therefore does not establish reliable handling of questions without supporting evidence\.

## 4Limitations

The findings apply to retrieval within a known document, with up to five passages and a 2,500\-token answer context\. The book evaluation covers eight textbooks and relies largely on model\-generated references and automated answer scores\. Most QASPER articles fit within the hybrid candidate pool, limiting what this dataset reveals about the benefit of full scanning\. Coverage includes failed retrievals, while answer scores use only completed evaluations\.

Latency reflects the tested deployments: hardware, precision, concurrency, and hosted service conditions differ across methods\. Hosted model behavior can also vary between calls\. API charges exclude local computing resources, including Clef’s GPU use, so they do not represent total operating cost\.

## 5Conclusion

We compared 11 retrieval configurations on research articles and complete textbooks\. On books, Hybrid \+ Jev achieved the highest observed evidence coverage \(94\.1%\), while Hybrid \+ Decisions obtained the highest observed Answer Accuracy among retrievers \(0\.892\)\. Both Clef variants reached 98\.8% evidence coverage on QASPER\. Across the comparisons, recovering more reference evidence did not consistently improve answer quality\.

Full scanning reached every passage initially but lost evidence during reduction\. On books, hybrid reranking was much faster and achieved the highest observed coverage and answer scores\. For 1,000 queries per document, dense retrieval had much lower projected spending than the paid decision APIs, including its initial embedding cost\. Choosing a retrieval method therefore requires considering evidence retention, answer quality, and cost over repeated use together\.

## References

- \[1\]Cloudflare\. Clef model card and released inference code\. Frozen revisioned3eed331870, 2026\.[https://huggingface\.co/Cloudflare/clef](https://huggingface.co/Cloudflare/clef)\.
- \[2\]P\. Lewis et al\. Retrieval\-Augmented Generation for Knowledge\-Intensive NLP Tasks\. NeurIPS, 2020\.[https://arxiv\.org/abs/2005\.11401](https://arxiv.org/abs/2005.11401)\.
- \[3\]W\. Sun et al\. Is ChatGPT Good at Search? Investigating Large Language Models as Re\-Ranking Agents\. EMNLP, 2023\.[https://aclanthology\.org/2023\.emnlp\-main\.923/](https://aclanthology.org/2023.emnlp-main.923/)\.
- \[4\]S\. Yoon et al\. ListT5: Listwise Reranking with Fusion\-in\-Decoder Improves Zero\-shot Retrieval\. ACL, 2024\.[https://aclanthology\.org/2024\.acl\-long\.125/](https://aclanthology.org/2024.acl-long.125/)\.
- \[5\]P\. Sarthi et al\. RAPTOR: Recursive Abstractive Processing for Tree\-Organized Retrieval\. ICLR, 2024\.[https://arxiv\.org/abs/2401\.18059](https://arxiv.org/abs/2401.18059)\.
- \[6\]V\. Kumar et al\. STAIR \(STructure Aware Information Retriever\): A novel dataset and LLM based retriever for document structure augmentation\. arXiv:2609\.03874, 2026\.[https://arxiv\.org/abs/2609\.03874](https://arxiv.org/abs/2609.03874)\.
- \[7\]K\. Li et al\. LaRA: Benchmarking Retrieval\-Augmented Generation and Long\-Context LLMs—No Silver Bullet for LC or RAG Routing\. arXiv:2502\.09977, 2025\.[https://arxiv\.org/abs/2502\.09977](https://arxiv.org/abs/2502.09977)\.
- \[8\]A\. Chauhan\. Experiment 02: Does combining IBM’s STAIR with search and Jev help? Community experiment report, September 20, 2026\.[https://github\.com/aryanchauhanoffical/no\-hallucination/blob/main/experiments/02\-searchtome\-stair\-jev/REPORT\.md](https://github.com/aryanchauhanoffical/no-hallucination/blob/main/experiments/02-searchtome-stair-jev/REPORT.md)\.
- \[9\]erendikmenn\. XQuAD\-TR benchmark—2026\-09\-20\. Community experiment report, September 2026\.[https://github\.com/erendikmenn/jev\-rag\-benchmark/blob/main/docs/benchmark\-2026\-09\-20\.md](https://github.com/erendikmenn/jev-rag-benchmark/blob/main/docs/benchmark-2026-09-20.md)\.
- \[10\]T\. Deußer, L\. Sparrenberg, and R\. Sifa\. Evaluating and Benchmarking the System One Model Jev\. arXiv:2609\.37647, 2026\.[https://arxiv\.org/abs/2609\.37647](https://arxiv.org/abs/2609.37647)\.
- \[11\]TypeSafe AI\. Models: Jev 1\.13, context limits and pricing\. Documentation, accessed October 2026\.[https://docs\.typesafe\.ai/models](https://docs.typesafe.ai/models)\.
- \[12\]TypeSafe AI\. Line\-by\-line search\. Documentation, accessed October 2026\.[https://docs\.typesafe\.ai/cookbooks/semantic\_find](https://docs.typesafe.ai/cookbooks/semantic_find)\.
- \[13\]OpenAI\. Decisions API guide\. Public\-beta documentation, accessed October 7, 2026\.[https://developers\.openai\.com/api/docs/guides/decisions](https://developers.openai.com/api/docs/guides/decisions)\.
- \[14\]P\. Dasigi et al\. A Dataset of Information\-Seeking Questions and Answers Anchored in Research Papers\. NAACL, 2021\.[https://aclanthology\.org/2021\.naacl\-main\.365/](https://aclanthology.org/2021.naacl-main.365/)\.
- \[15\]S\. Es et al\. RAGAs: Automated Evaluation of Retrieval Augmented Generation\. EACL System Demonstrations, 2024\.[https://aclanthology\.org/2024\.eacl\-demo\.16/](https://aclanthology.org/2024.eacl-demo.16/)\.
- \[16\]Ragas contributors\. Answer Accuracy \(NVIDIA metrics\), version 0\.4\.3\. Software documentation, accessed October 2026\.[https://docs\.ragas\.io/en/v0\.4\.3/concepts/metrics/available\_metrics/nvidia\_metrics/](https://docs.ragas.io/en/v0.4.3/concepts/metrics/available_metrics/nvidia_metrics/)\.
- \[17\]R\. Friel, M\. Belyi, and A\. Sanyal\. RAGBench: Explainable Benchmark for Retrieval\-Augmented Generation Systems\. arXiv:2407\.11005, 2024, revised 2025\.[https://arxiv\.org/abs/2407\.11005](https://arxiv.org/abs/2407.11005)\.
- \[18\]Y\. Katsis et al\. mtRAG: A Multi\-Turn Conversational Benchmark for Evaluating Retrieval\-Augmented Generation Systems\. Transactions of the Association for Computational Linguistics, 2025\.[https://aclanthology\.org/2025\.tacl\-1\.36/](https://aclanthology.org/2025.tacl-1.36/)\.
- \[19\]J\. van Elburg, P\. van der Putten, and M\. Marx\. Can we Evaluate RAGs with Synthetic Data? SynDAiTE Workshop at ECML\-PKDD, 2025\.[https://arxiv\.org/abs/2508\.11758](https://arxiv.org/abs/2508.11758)\.
- \[20\]J\. Saad\-Falcon et al\. ARES: An Automated Evaluation Framework for Retrieval\-Augmented Generation Systems\. NAACL, 2024\.[https://aclanthology\.org/2024\.naacl\-long\.20/](https://aclanthology.org/2024.naacl-long.20/)\.
- \[21\]S\. Yang et al\. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples\. arXiv:2311\.04850, 2023\.[https://arxiv\.org/abs/2311\.04850](https://arxiv.org/abs/2311.04850)\.
- \[22\]Z\.\-Y\. Chen et al\. Beyond the Surface: Measuring Self\-Preference in LLM Judgments\. EMNLP, 2025\.[https://aclanthology\.org/2025\.emnlp\-main\.86/](https://aclanthology.org/2025.emnlp-main.86/)\.
- \[23\]L\. Shi et al\. Judging the Judges: A Systematic Study of Position Bias in LLM\-as\-a\-Judge\. IJCNLP\-AACL, 2025\.[https://aclanthology\.org/2025\.ijcnlp\-long\.18/](https://aclanthology.org/2025.ijcnlp-long.18/)\.
- \[24\]S\. Robertson and H\. Zaragoza\. The Probabilistic Relevance Framework: BM25 and Beyond\. Foundations and Trends in Information Retrieval, 2009\.[https://doi\.org/10\.1561/1500000019](https://doi.org/10.1561/1500000019)\.
- \[25\]N\. Reimers and I\. Gurevych\. Sentence\-BERT: Sentence Embeddings using Siamese BERT\-Networks\. EMNLP\-IJCNLP, 2019\.[https://aclanthology\.org/D19\-1410/](https://aclanthology.org/D19-1410/)\.
- \[26\]V\. Karpukhin et al\. Dense Passage Retrieval for Open\-Domain Question Answering\. EMNLP, 2020\.[https://aclanthology\.org/2020\.emnlp\-main\.550/](https://aclanthology.org/2020.emnlp-main.550/)\.
- \[27\]O\. Khattab and M\. Zaharia\. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT\. SIGIR, 2020\.[https://arxiv\.org/abs/2004\.12832](https://arxiv.org/abs/2004.12832)\.
- \[28\]G\. Izacard et al\. Unsupervised Dense Information Retrieval with Contrastive Learning\. Transactions on Machine Learning Research, 2022\.[https://arxiv\.org/abs/2112\.09118](https://arxiv.org/abs/2112.09118)\.
- \[29\]J\. Chen et al\. M3\-Embedding: Multi\-Linguality, Multi\-Functionality, Multi\-Granularity Text Embeddings Through Self\-Knowledge Distillation\. Findings of ACL, 2024\.[https://aclanthology\.org/2024\.findings\-acl\.137/](https://aclanthology.org/2024.findings-acl.137/)\.
- \[30\]G\. V\. Cormack, C\. L\. A\. Clarke, and S\. Büttcher\. Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods\. SIGIR, 2009\.[https://doi\.org/10\.1145/1571941\.1572114](https://doi.org/10.1145/1571941.1572114)\.
- \[31\]N\. Thakur et al\. BEIR: A Heterogeneous Benchmark for Zero\-shot Evaluation of Information Retrieval Models\. NeurIPS Datasets and Benchmarks, 2021\.[https://datasets\-benchmarks\-proceedings\.neurips\.cc/paper/2021/hash/65b9eea6e1cc6bb9f0cd2a47751a186f\-Abstract\-round2\.html](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/65b9eea6e1cc6bb9f0cd2a47751a186f-Abstract-round2.html)\.
- \[32\]N\. Muennighoff et al\. MTEB: Massive Text Embedding Benchmark\. EACL, 2023\.[https://aclanthology\.org/2023\.eacl\-main\.148/](https://aclanthology.org/2023.eacl-main.148/)\.
- \[33\]D\. Sachan et al\. Improving Passage Retrieval with Zero\-Shot Question Generation\. EMNLP, 2022\.[https://aclanthology\.org/2022\.emnlp\-main\.249/](https://aclanthology.org/2022.emnlp-main.249/)\.
- \[34\]Z\. Qin et al\. Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting\. Findings of NAACL, 2024\.[https://aclanthology\.org/2024\.findings\-naacl\.97/](https://aclanthology.org/2024.findings-naacl.97/)\.
- \[35\]G\. Izacard and E\. Grave\. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering\. EACL, 2021\.[https://aclanthology\.org/2021\.eacl\-main\.74/](https://aclanthology.org/2021.eacl-main.74/)\.
- \[36\]N\. F\. Liu et al\. Lost in the Middle: How Language Models Use Long Contexts\. Transactions of the Association for Computational Linguistics, 2024\.[https://aclanthology\.org/2024\.tacl\-1\.9/](https://aclanthology.org/2024.tacl-1.9/)\.
- \[37\]Y\. Bai et al\. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding\. ACL, 2024\.[https://aclanthology\.org/2024\.acl\-long\.172/](https://aclanthology.org/2024.acl-long.172/)\.
- \[38\]C\.\-P\. Hsieh et al\. RULER: What’s the Real Context Size of Your Long\-Context Language Models? COLM, 2024\.[https://arxiv\.org/abs/2404\.06654](https://arxiv.org/abs/2404.06654)\.
- \[39\]Z\. Li et al\. Retrieval Augmented Generation or Long\-Context LLMs? A Comprehensive Study and Hybrid Approach\. EMNLP Industry Track, 2024\.[https://aclanthology\.org/2024\.emnlp\-industry\.66/](https://aclanthology.org/2024.emnlp-industry.66/)\.
- \[40\]J\. Chen et al\. Benchmarking Large Language Models in Retrieval\-Augmented Generation\. AAAI, 2024\.[https://ojs\.aaai\.org/index\.php/AAAI/article/view/29728](https://ojs.aaai.org/index.php/AAAI/article/view/29728)\.
- \[41\]T\. Gao et al\. Enabling Large Language Models to Generate Text with Citations\. EMNLP, 2023\.[https://aclanthology\.org/2023\.emnlp\-main\.398/](https://aclanthology.org/2023.emnlp-main.398/)\.

## Appendix ACorpus and reproducibility

Table[6](https://arxiv.org/html/2610.28957#A1.T6)lists the complete textbooks and the subsets used for timing repetitions and answer evaluation\. Each book has ten questions: four factual, three explanatory, and three requiring multiple sections\. The eight test books contain 4\.46 million tokens and 10,508 passages\. Development uses two separate books\. The test subsets are fixed across methods, with 11 retrieval configurations and two answer controls\.

BookTokensPassagesTiming Q\.Answer Q\.Test: 80 questionsAstronomy 2e749,8261,76549Biology 2e954,0822,24549Psychology 2e515,3991,21396Principles of Economics 3e635,7811,49688American Government 3e509,4721,19985Introduction to Business420,03498988Introduction to Sociology 3e398,80293948Introduction to Philosophy281,39266257Total test4,464,78810,5085060Development: 20 questionsU\.S\. History605,9291,426——Introduction to Anthropology454,9041,071——Table 6:Book sizes and test subsets\. Timing questions receive three assigned retrieval executions; this column does not define the full\-test primary timing comparison used for Jev, Decisions, and Clef\. Answer questions are shared by all 13 answer arms\. Development books are outside these test subsets\. Token counts use the common text tokenizer\.The QASPER test contains 396 questions from 264 articles\. Evidence coverage uses its 320 answerable questions from 230 articles; costs and execution counts include all 396 questions\. The exact oracle reaches 100% reference coverage with at most five passages for every answerable test question: 320 from QASPER and 80 from books\. Thus, the five\-passage limit permits complete reference recovery in this dataset\. Fitting those passages into the answer context remains a separate constraint\.

The recorded checkpoint prefixes are5617a9f61b02for BGE\-M3,953dc6f6f85afor the BGE reranker, anded3eed331870for Clef\. Clef uses SDPA attention and the PyTorch CUDA convolution fallback, without quantization or parameter offloading to the CPU\. Full revision identifiers, environments, prompts, source locks, and execution records are documented in the[code repository](https://github.com/jrterven/jev-vs-rag)\.

The evaluation code was checked with synthetic examples covering paraphrases, contradictions, missing support, partial answers, and abstention\. All eight Faithfulness groups and eleven Answer Accuracy groups met their expected ranges in three repetitions\. A separate Factual Correctness diagnostic failed one condition and was not used for test scoring\. A separate model also reviewed the 100 development and test book questions, reference answers, and 148 supporting spans without seeing retrieval outputs or answer scores\. It recorded three minor observations and no confirmed reference error; no questions or evidence spans were changed\. These checks document expected metric behavior and reference consistency\.

## Appendix BRetrieval diagnostics and resource use

All 5,236 primary retrieval assignments have a recorded outcome: 5,222 completed and 14 failed\. Table[7](https://arxiv.org/html/2610.28957#A2.T7)identifies the failures and the valid repeated timing observations\. Failed primary retrievals remain in the evidence evaluation with zero coverage\. A failed or invalid timing observation is excluded from latency statistics\.

Table 7:Primary failures and repeated end\-to\-end timing availability\. Every method has 396 primary QASPER and 80 primary book assignments\. The repeated benchmark assigns 50 questions per corpus and three runs per question\. Hybrid RRF has one completed run with invalid timing; other exclusions in the repeated\-run columns are failed runs\. Dashes indicate that Clef has no comparable completed repetition summary\.The Decisions repetition counts include valid runs under both exclusive and shared GPU conditions; their latency summaries remain separate by condition\. Clef completed all 952 primary searches, but only 18 of 400 additional timing repetitions, with 382 incomplete\. Its reported latency therefore uses primary executions only\. These omissions do not count as primary retrieval failures\. Service interruptions and varying deployment conditions remain part of the timing records\.

The shared hybrid pool of up to 50 candidates already contains 98\.75% of book reference evidence before reranking, averaged across all 80 questions and eight books\. This coverage refers to the entire candidate pool, without the final five\-passage restriction\. Table[8](https://arxiv.org/html/2610.28957#A2.T8)shows where full scans lose evidence\. Jev and Decisions lose most during intermediate reductions; Clef loses most when selecting the final five\. These traces include completed scans only; primary means also include failed retrievals\.

Table 8:Book evidence coverage \(%\) through the reduction stages, averaged within documents and then across eight books\. Each row uses that method’s completed questions; the rows are not restricted to a common successful subset\. Stages describe candidate sets, and no confidence intervals are inferred for these trace summaries\.Leaving out one book at a time gives a Jev full\-scan advantage over OpenAI dense of 5\.8–8\.9 percentage points\. Against Hybrid \+ BGE, the range is 0\.2–4\.3 points, but the full\-sample 95% bootstrap interval \[−5\.6\-5\.6, 9\.7\] includes zero\. These omission ranges describe variation across book subsets and are distinct from the paired bootstrap intervals\.

Preparation is a one\-time document cost \(Table[9](https://arxiv.org/html/2610.28957#A2.T9)\)\. Local preparation covers the shared lexical and BGE indexes, including initial model loading\. It is not multiplied by the number of methods that reuse those indexes\.

Table 9:Document preparation, measured separately from retrieval and answer generation\. Local time covers shared indexes; OpenAI time covers document embedding calls\.Table[10](https://arxiv.org/html/2610.28957#A2.T10)gives the per\-query inputs to the reuse projection\. Only OpenAI dense has a paid preparation term: US$0\.000109950985 per article and US$0\.0131306925 per book\. Other methods may require local preparation, which is not priced here\. Charges from failed base queries are included when known; unknown consumption produces a lower bound\. Timing repetitions, answer generation, and judging are outside this projection\.

Table 10:Mean retrieval API cost per assigned base query, averaging within documents and then across documents\. Values use the rates recorded during execution; lower bounds are rounded down\. The other local retrieval methods also have zero API charges, with computing resources unpriced\.For books, the crossover between dense retrieval and the Hybrid \+ Jev lower bound based on confirmed charges is

0\.01313069250\.002522472225−0\.0000003805=5\.2063​queries\.\\frac\{0\.0131306925\}\{0\.002522472225\-0\.0000003805\}=5\.2063\\ \\text\{queries\}\.Thus, dense retrieval is cheaper by the sixth query per book under this projection\. Unknown Jev charges can move the crossover earlier\.

## Appendix CAnswer comparisons and evaluation outcomes

Table[11](https://arxiv.org/html/2610.28957#A3.T11)collects paired Answer Accuracy comparisons against dense retrieval and between decision models\. Each comparison uses only questions scored for both methods and then averages within documents\. Consequently, these effects cannot be reconstructed by subtracting the conditional means in the main results\. Book comparisons use eight documents\. Intervals are descriptive and are not adjusted for multiple comparisons\.

Table 11:Paired Answer Accuracy differences, first method minus second\. “Scan” denotes full scan\. Each cell reports the difference, its 95% interval, and the number of jointly scored questionsnn\. Intervals use 10,000 whole\-document bootstrap resamples with seed 42\. An interval containing zero does not establish equivalence\.The paired book Faithfulness gains for Jev full scan are 0\.069 \[0\.050, 0\.090\] over OpenAI dense \(57 pairs\), and 0\.028 \[0\.010, 0\.050\] over Hybrid \+ BGE \(56 pairs\)\. These comparisons apply to answers with supplied evidence that did not abstain\. They do not include every assigned answer\.

Across the 13 answer arms, 2,586 of 2,600 assignments produced usable answers\. Ten missing answers followed retrieval failures: three for Jev full scan and seven for Decisions full scan, all on books\. Four others resulted from generation errors, one each for BGE\-M3 on unanswerable QASPER, Hybrid \+ BGE on books, and Hybrid \+ Jev and Hybrid \+ Decisions on answerable QASPER\. All reference\-context controls were eligible\. Table[12](https://arxiv.org/html/2610.28957#A3.T12)distinguishes completed answers from abstentions in each group\.

Table 12:Each cell gives completed answers, with abstentions in parentheses\. Abstention is an observed response behavior; these are question counts, not document\-macro percentages\. Rates use completed answers as the denominator: BGE\-M3, for example, abstains in 16 of its 39 completed unanswerable cases\. Missing answers remain separate\.The judge scored 4,233 of 4,244 requested evaluations; 11 failed\. Table[13](https://arxiv.org/html/2610.28957#A3.T13)shows the full accounting, including unanswerable questions\. Faithfulness is not requested for abstentions or the no\-evidence control; Answer Accuracy is not requested for unanswerable references\. The 928 inapplicable answer–metric combinations and all missing or failed outputs remain outside numerical scores\. No requested evaluation remains pending or unscorable\.

Table 13:Evaluation outcomes across all 13 answer arms\. N/A counts metrics not requested for completed answers; it excludes missing generations\. Requested evaluations equal scored evaluations plus errors\. Failed and inapplicable metrics receive no imputed score\.