不留遗漏：从大规模随机评估语料库中寻找证据

# Nothing Left Behind: Finding Evidence from a Large Randomized Evaluation Corpus
Source: [https://arxiv.org/html/2610.36910](https://arxiv.org/html/2610.36910)
Technical ReportAffiliation:Giuliano Martinelli, Abelardo Carlos Martinez Lorenzo, Satvik Garg, Linxi Wang, Riccardo Orlando, Piriyakorn Piriyatamwong, Samuel Paul Fraiberger

October 2026

###### Abstract

Many AI systems need to find every relevant items in a large collection and then organize what they found\. The standard way to do this is to tag each item with labels from a curated vocabulary and search on those labels\. Those labels are useful for organizing results, but searching on them misses items that do not fit a label and returns too much when labels are broad\. We compare this approach with an embedding\-based approach: find candidates by meaning using text embeddings, then have a language model keep only the relevant ones\. In a corpus of 7,354 randomized trials, label search found 45% of the evidence a systematic review would include, while the new approach found 92%, at a cost per query that rises with search depth\. These results support a two\-stage architecture that combines broad candidate retrieval with query\-specific evidence filtering\.

## IIntroduction

An AI assistant can often answer a question from a few useful passages\. Retrieval\-augmented generation \(RAG\) connects document retrieval to language generation in this way\[[1](https://arxiv.org/html/2610.36910#bib.bib1)\]\. But some applications need the retrieved set itself to be complete enough for further work\. A synthesis of whether school meals improve attendance, for example, should consider all eligible trials available to the analyst, not simply several convincing examples\. Missing studies can change a pooled estimate or the interpretation of the evidence\. A fluent answer does not establish that the underlying search was complete\.

This report concerns that second kind of task\. The goal is to find relevant evidence within an existing collection and then organize it for comparison, screening, or aggregation\. Recall—the share of relevant items recovered—is therefore a primary objective, alongside precision—the share of returned items that are relevant\. High recall does not make irrelevant results harmless: someone or something must still screen them\. The engineering problem is to recover a sufficiently complete set at an acceptable screening and computational cost\.

Systematic reviews provide a practical example of this requirement\. Review guidance recommends sensitive searches combining controlled vocabulary and free text, followed by eligibility assessment\[[2](https://arxiv.org/html/2610.36910#bib.bib2)\]\. Related completeness requirements arise in document review, prior\-art search, and audit\.

![Refer to caption](https://arxiv.org/html/2610.36910v1/image.png)Fig\. 1:The ImpactAI evidence search interface\.ImpactAI111[https://impactai\.worldbank\.org](https://impactai.worldbank.org/)supports the discovery and synthesis of evidence from randomized controlled trials \(RCTs\) in development economics\. Its original retrieval architecture translated a user’s question into labels from a curated taxonomy, then returned evidence associated with those labels\. This provides explicit concepts and useful organization, but makes discovery depend on the taxonomy’s coverage, granularity, and linking decisions\. A question finer than a label may retrieve too much; a concept absent from the catalog may be missed or mapped to an imperfect substitute\.

The alternative retrieves intervention–outcome pairs directly by the semantic similarity of their textual descriptions to the question\. A language model then judges each candidate for relevance\. This separates broad candidate generation from query\-specific screening, in the spirit of multistage neural retrieval\[[3](https://arxiv.org/html/2610.36910#bib.bib3)\]\. The second stage selects the evidence set used in subsequent analysis\. Labels can subsequently organize that set without determining the initial candidate pool\. Embeddings remove dependence on an explicit label match, but, on the other hand, introduce their own limits through representation quality and candidate budget\.

We use ImpactAI as a case study to compare the architectures, evaluate retrieval and filtering, and examine their operational trade\-offs\. The collection and its extracted records are taken as given; corpus construction is described in the accompanying extraction post\. Our focus is which evidence is found before downstream synthesis\.

## IITwo Retrieval Designs: The ImpactAI Case

ImpactAI retrieves evidence from a heterogeneous corpus of RCTs \(Figure[1](https://arxiv.org/html/2610.36910#S1.F1)\)\. We describe the evidence records and the two architectures used to search them\.

### II\-AEvidence Structure

ImpactAI contains an expert\-curated corpus of 7,354 randomized controlled trials \(RCTs\) spanning multiple sectors, including health, education, agriculture, public finance, and social protection \.222[https://impactai\.worldbank\.org/blog/extracting\-experimental\-data\-from\-studies](https://impactai.worldbank.org/blog/extracting-experimental-data-from-studies)Because individual trials can evaluate multiple interventions and outcomes, the corpus contains 74,826 intervention–outcome pairs\. Each pair is associated with a set of attributes describing the underlying evidence, including the study population, context, design and setting, time horizon, implementation characteristics, measurement details, and sample size\.

As illustrated in Table[I](https://arxiv.org/html/2610.36910#S2.T1), the intervention–outcome pair is the fundamental retrieval unit in our system\. Each pair includes an intervention descriptionIiI\_\{i\}and an outcome descriptionOiO\_\{i\}, together with metadata and information about the corresponding study and effect estimate\. Given a user queryqq, the task is to retrieve pairs whose associated evidence addresses the user’s information need\. Each pair remains linked to its source study, allowing retrieved evidence to be traced back to the original RCT\.

Retrieving at the intervention–outcome level provides finer granularity than paper\-level retrieval\. A single RCT may examine several interventions or outcomes, and only a subset of these may be relevant to a particular query\.

Data FieldExamplePaper id7077Paper TitleThe impact of mRDT malaria tests in UgandaIntervention–Outcome pairs1\. Pair id767171\. Intervention NamemRDT malaria tests1\. Intervention DescriptionThe intervention involves the introduction of malaria rapid diagnostic tests \(mRDTs\) in public health centers, where healthcare personnel diagnose malaria using mRDTs instead of relying solely on clinical judgment\. ACT is only administered when the mRDT result is positive\.1\. Outcome NameSocietal Cost1\. Outcome DescriptionAggregate societal cost \(health sector \+ household\) per 1000 fever episodes over 2 years\.2\. Pair id767182\. Intervention NamemRDT malaria tests2\. Intervention DescriptionThe intervention involves the introduction of malaria rapid diagnostic tests \(mRDTs\) in public health centers, where healthcare personnel diagnose malaria using mRDTs instead of relying solely on clinical judgment\. ACT is only administered when the mRDT result is positive\.2\. Outcome NameChildren impacted2\. Outcome DescriptionPercentage of children with malaria receiving appropriate treatment\.

TABLE I:An RCT and two intervention–outcome pairs in ImpactAI\.
### II\-BTaxonomy\-based Retrieval

Architecture \(A\) in Figure[2](https://arxiv.org/html/2610.36910#S2.F2)uses a taxonomy for both discovery and organization\. Evidence is tagged offline with catalog entities\. At query time, an LLM identifies spans and assigns entity types from the development\-economics catalog, including intervention, outcome, population group, and geography\. For each span, the system retrieves candidate entities within the assigned type by combining semantic similarity and BM25 keyword matching, with up to 50 candidates per span\. An LLM selects the entity that best fits the span in the context of the complete question\. The selected IDs are then used to retrieve associated evidence\.

![Refer to caption](https://arxiv.org/html/2610.36910v1/images.png)Fig\. 2:Taxonomy\-based entity linking and semantic retrieval with relevance filtering\.This design normalizes synonymous expressions and exposes inspectable query constraints\. Results can be grouped by shared categories, and a user can inspect or correct the linked concepts\. A category lookup can also return all evidence associated with a selected label without a semantic top\-KKcutoff, subject to implementation limits and tagging completeness\.

The same representation constrains discovery\. A question about one particular form of computer\-assisted learning may be mapped to the broader computer\-assisted\-learning category, losing distinctions needed by the user\. A concept absent from the catalog has no exact entity to link to\. Compound questions may span several categories, while errors in span detection, entity typing, selection, or evidence tagging can remove relevant evidence\. Finer labels can reduce some ambiguities but increase mapping and maintenance work\.

For a fixed catalog and linking decision, increasing a semantic candidate budget does not change the taxonomy result\. This explains the flat baseline in the recall comparison\.

### II\-CSemantic Retrieval and Relevance Filtering

Architecture \(B\) combines neural retrieval with a query\-specific evidence\-filtering stage\. Instead of using query spans and retrieving taxonomy entities, the neural retriever compares the query directly with textual descriptions of intervention–outcome pairs using embedding\-based similarity\. This allows retrieval to draw on the semantic information expressed in the evidence descriptions without relying on an explicit taxonomy match\.

Semantic similarity alone, however, does not guarantee relevance to the user’s specific information need\. The retriever may return candidates that discuss a similar topic but differ in the intervention, outcome, population, or context of interest\. We therefore apply an LLM\-based filter to the retrieved candidates\. Given the query and a candidate pair, the filter assesses relevance and removes candidates that do not satisfy the query’s requirements\.

The two stages serve complementary purposes: neural retrieval generates a broad candidate set to preserve recall, while relevance filtering aims to improve precision by removing irrelevant candidates\. This separation allows candidate generation and relevance assessment to be evaluated independently\.

Each evidence record is indexed using intervention and outcome names and descriptions together with study context\. The query is embedded into the same space, and the closestKKpairs form the candidate set\. The filter retains candidates judged relevant to the full query\. For the October latency configuration, candidates are grouped by source paper, with at most 15 pairs per prompt, while retaining a separate relevance decision for each pair\.

The output remains a set of evidence pairs linked to their source papers\. Grouping, category browsing, and meta\-analysis can follow; a positive relevance decision alone does not establish comparability for pooling\. IncreasingKKmakes more candidates available to the filter, but the filter cannot recover a pair that candidate generation omitted\. A relevant set larger than the budget cannot be completely recovered in one top\-KKretrieval\.

## IIIExperimental Setup

### III\-AQuestions

We evaluate the retrieval system through four experimental questions:

1. 1\.Filtering agreement:To what extent do Gemini 2\.5 Flash relevance judgments agree with human annotations?
2. 2\.Retrieval quality:Which retrieval system produces the strongest candidate set for downstream relevance filtering?
3. 3\.End\-to\-end evidence coverage:How much of the evidence identified through systematic reviews does the selected pipeline recover?
4. 4\.Operational trade\-offs:How does search depth affect module latency?

### III\-BModels and Configuration

We compare Gemini embeddings\[[4](https://arxiv.org/html/2610.36910#bib.bib4)\], Voyage, E5\-large\-v2\[[5](https://arxiv.org/html/2610.36910#bib.bib5)\], and Granite\. The taxonomy quality baseline uses Gemini 2\.5 Flash for entity linking\. Our semantic pipeline uses Gemini embeddings for candidate generation and Gemini 2\.5 Flash\[[6](https://arxiv.org/html/2610.36910#bib.bib6)\]for relevance filtering\.

### III\-CDatasets

We use several datasets to evaluate the filtering and retrieval components separately and to assess the resulting pipeline\. Table[II](https://arxiv.org/html/2610.36910#S3.T2)summarizes the datasets and their role in the evaluation\.

TABLE II:Datasets used to evaluate retrieval and relevance filtering\.The filtering experiments use 100 human\-generated queries, 100 synthetic queries, and a smaller set of 10 double\-annotated queries\. Each query has 30 candidates retrieved with Gemini embeddings\. Binary evaluation treats*Relevant*as positive and merges*Potentially Relevant*with*Negative*\. The descriptive counts in Table[II](https://arxiv.org/html/2610.36910#S3.T2)include both relevant and partially relevant passages, averaging 24\.02 and 23\.82 per query in the Beta Tester and Synthetic sets\. The double\-annotated set has no single adjudicated relevance count\. The ten\-query human–human comparison uses the 276 passages with both annotations; 24 passages lack one annotation\. Reviewer positions are assigned within each query, not to fixed individuals across the dataset\.

Candidate retrieval is evaluated on two larger benchmarks: 374 Beta Tester queries selected from questions submitted by World Bank researchers and policymakers, and 1,000 Synthetic queries generated from specific RCTs\. For each query, an LLM judges a pool containing the top 1,000 results from Gemini embeddings, the top 1,000 from Voyage, and all taxonomy retrieval results\. The resulting labels provide the reference set for pooled retrieval precision and recall\.

Finally, the review\-derived benchmark contains 348 formulations in 116 query families from 29 review sources\. Expected paper sets come from review inclusion lists intersected with the ImpactAI corpus, with LLM\-assisted assignments for narrower questions\. The benchmark contains 1,416 paper–query memberships, averaging 4\.07 per formulation\.

Separately, users assessed returned passages through upvote/downvote feedback in a web interface, yielding 45 queries with an average of 17 relevant assessed passages\.

### III\-DEvaluation Units and Metrics

The retrieval unit is an intervention–outcome pair, andKKcounts candidate pairs\. Filtering agreement and pooled retrieval metrics operate on passages\. For the review benchmark, returned pairs are mapped to distinct source papers, and recall is averaged equally across query formulations\. Reference sets contain known positives: papers outside them are unjudged, not confirmed false positives\.

Lower\-KKrecall is calculated from ranked prefixes of saved top\-2,500 outputs\. Entity linking has no semantic candidate cutoff\. The larger Beta Tester and Synthetic benchmarks use LLM judgments over pooled retrieval results, so their scores are relative to that pool\.

## IVResults

### IV\-AFiltering Module

Table[III](https://arxiv.org/html/2610.36910#S4.T3)compares Gemini 2\.5 Flash judgments with human labels and reports human–human agreement on the double\-annotated set\. Micro\-F1 measures overall agreement; class\-specific F1 combines precision and recall for each class\.

TABLE III:Binary relevance agreement with human labels\.Gemini 2\.5 Flash reaches Cohen’sκ=0\.5026\\kappa=0\.5026on Beta Tester queries and0\.50450\.5045on Synthetic queries, indicating moderate agreement\. The corresponding micro\-F1 scores are 0\.7513 and 0\.7510\. Human–human agreement on the separate ten\-query sample isκ=0\.4340\\kappa=0\.4340\. These results indicate disagreement in relevance assessment and support clearer annotation criteria\.

TABLE IV:Retrieval performance on the Beta Tester Queries\. Recall \(R\), precision \(P\), and F1 at different retrieval depths\.TABLE V:Retrieval performance on the Synthetic Queries benchmark\. R: recall; P: precision\.
### IV\-BRetrieval Module

The second experiment evaluates the candidate\-retrieval component\. We compare the existing ImpactAI system with Gemini embeddings, Voyage, E5\-large\-v2, and Granite on the Beta Tester and Synthetic Query benchmarks\. For each query, we retrieve the top\-kkintervention–outcome pairs, withk∈\{10,100,150,300,500,1000\}k\\in\\\{10,100,150,300,500,1000\\\}\. We report recall, precision, and F1 at each depth\.

Table[IV](https://arxiv.org/html/2610.36910#S4.T4)reports results on the Beta Tester Queries\. Gemini embeddings achieve the highest recall and F1 among the evaluated embedding models across retrieval depths\. Atk=100k=100, Gemini reaches 0\.47 recall and 0\.49 F1, compared with F1 scores of 0\.27 for Voyage, 0\.20 for E5\-large\-v2, and 0\.26 for Granite\. Atk=1000k=1000, Gemini reaches 0\.93 recall and 0\.24 F1, compared with 0\.67 recall and 0\.15 F1 for Voyage, 0\.50 and 0\.11 for E5\-large\-v2, and 0\.62 and 0\.14 for Granite\.

We observe the same pattern on the Synthetic Queries benchmark, shown in Table[V](https://arxiv.org/html/2610.36910#S4.T5)\. Gemini embeddings again achieve the highest recall at every depth\. Atk=100k=100, Gemini achieves 0\.53 recall and 0\.53 F1, compared with F1 scores of 0\.31 for Voyage and 0\.28 for Granite\.

Atk=1000k=1000, Gemini reaches 0\.96 recall and 0\.23 F1\. The corresponding F1 scores are 0\.17 for Voyage, 0\.13 for E5\-large\-v2, and 0\.14 for Granite\.

The higher candidate recall across both benchmarks motivates using Gemini embeddings in the remaining experiments\. The relevance filter then screens its candidates against the full query\.

### IV\-CDownstream Evidence Retrieval

The separate feedback evaluation reports a relevance rate of 94\.7% for 348 assessed passages across 45 queries\. We evaluate paper recall against review\-derived reference sets below\.

Table[VI](https://arxiv.org/html/2610.36910#S4.T6)separates candidate coverage, filtering, and the union with entity linking \(EL\)\. AtK=2,500K=2\{,\}500, candidate retrieval recovers 96\.7% of expected papers on the macro measure, compared with 45\.1% for EL\. Gemini 2\.5 Flash filtering retains 85\.0%, a loss of 11\.7 percentage points relative to the candidate set\.

TABLE VI:Macro paper recall \(%\) on the review\-derived benchmark\.The union with EL adds 12\.6 percentage points over filtered semantic retrieval atK=100K=100, falling to 2\.9 points atK=2,500K=2\{,\}500\(computed before rounding\)\. The two routes therefore recover complementary evidence, especially at smaller semantic budgets\. The union retains the EL results without applying the semantic filter to that branch\.

1001005005001,0001\{,\}0002,5002\{,\}500404060608080100100Candidate pairs \(KK\)Macro known\-positive paper recall \(%\)EL \(unbudgeted\)Before filteringGemini 2\.5 filterEL union, Gemini 2\.5Fig\. 3:Recall by candidate depth\.TABLE VII:Module latency in seconds\.EL andK=2,500K=2\{,\}500timings were measured on 30 queries;

## VWhat It Costs: Module Latency and Search Depth

Semantic retrieval moves screening work to query time\. We evaluated module latency on 30 review\-derived queries run serially, with Gemini 2\.5 Flash filtering candidates in paper batches of up to 15\. The median time atK=2,500K=2\{,\}500is 1\.48 seconds for retrieval alone and 77\.22 seconds with filtering, compared with 6\.00 seconds for the entity\-linking module \(Table[VII](https://arxiv.org/html/2610.36910#S4.T7)\)\. Grouping, data analysis, and final answer generation are excluded\. The filter accounts for most of the query time because it reads every candidate\. Batching reduces the number of prompts, but the workload still grows with depth and depends on how candidates are distributed across papers\. Depth is therefore a product choice: a user seeking comprehensive evidence may accept a longer wait, while an exploratory search may favor a smaller candidate set\.

## VIDiscussion

### VI\-AUse Meaning to Find and Labels to Organize

The main finding is a substantial gain in evidence coverage: filtered semantic retrieval reaches 85\.0% macro paper recall atK=2,500K=2\{,\}500, compared with 45\.1% for entity linking\. Directly searching evidence descriptions allows the system to handle questions whose concepts or combinations are poorly represented in the taxonomy\.

Labels remain useful\. They normalize terminology, provide inspectable concepts, and organize evidence for browsing and comparison\. They are particularly effective when the user wants everything under a specified category\. Semantic retrieval is more flexible about wording, but its output still needs organization and comparability checks before pooling\. Separating discovery from organization lets each representation serve the task it handles best\.

### VI\-BRecall, Precision, and Cost

Increasing depth improves candidate coverage while bringing more irrelevant material into the pool\. On Beta Tester queries, Gemini recall rises from 0\.47 atK=100K=100to 0\.93 atK=1,000K=1\{,\}000, while precision falls from 0\.52 to 0\.13\. Relevance filtering screens this wider pool, but also loses some known\-positive evidence: recall falls from 96\.7% before filtering to 85\.0% afterwards atK=2,500K=2\{,\}500\.

The appropriate operating point depends on the cost of missing evidence relative to screening extra candidates and waiting for a result\. The hybrid union with entity linking helps most at small budgets, suggesting that existing taxonomy infrastructure can complement semantic search\. Its additional screening and execution costs also need to be considered\.

### VI\-CMaintenance and Scale

Taxonomy search requires maintaining the vocabulary and evidence tags as the collection evolves\. Semantic search avoids mapping each new concept to a catalog, but still requires embedding, index maintenance, and monitoring the relevance model\. It also replaces explicit linked labels with scores and judgments that can be harder to inspect as a whole\.

A fixed candidate budget can truncate broad queries, particularly as the relevant evidence set grows\. Larger budgets may preserve coverage at the expense of filtering work\. Taxonomy\-based retrieval can remain useful for exhaustive enumeration within a known label\. These scaling considerations motivate further measurement rather than implying that either design has constant cost as the corpus grows\.

## VIILimitations and Next Steps

The results come from one corpus of development\-related RCTs\. Both labels and evidence descriptions depend on extraction quality\. The review benchmark measures known\-positive paper recall within that corpus, not coverage of the entire literature or every eligible effect\. Some narrower query assignments and the larger retrieval benchmarks use LLM judgments; the human feedback and double\-annotation samples are limited\. Exact Voyage and Granite checkpoints were not recorded consistently in the source report\.

The latency comparison covers serial modules with different service configurations\. The timed entity\-linking service used Gemini 3 Flash Preview, whereas its quality baseline used Gemini 2\.5 Flash; the October batched filter also differs from the earlier quality run\. Controlled throughput, monetary cost, and effects on downstream meta\-analysis remain to be evaluated\.

Next steps are to expand human annotation, test smaller relevance models, choose depth per query, and measure performance under concurrent load\. End\-to\-end synthesis experiments should establish whether more complete evidence sets change pooled estimates or conclusions\.

## VIIIConclusion

For tasks where completeness matters, such as automatic meta\-analysis, finding a few convincing passages is insufficient\. In this report, we show that semantic retrieval with Gemini 2\.5 Flash filtering raises macro known\-positive paper recall from 45\.1% to 85\.0% at 2,500 candidates, yielding a passage set with high\-perceived precision \(94\.7%\)\. This improvement comes with a substantial increase in query\-time cost, making search depth and filtering policy central design choices\. This case study supports using semantic retrieval to find evidence and labels to organize it, while potentially retaining taxonomy search for category browsing and complementary coverage\.

## IXAcknowledgements

All the manual Annotations were produced by the Development ImpactAI Lab Data Team, specifically: Nihaa Sajid, Clémence Gall, Rida Ali Khan, Saqib Hussain, Aarushi Aggarwal, Ejigayehu Diriba\. We also thank the Google\.org Foundation for the financial support\.

## References

- \[1\]P\. Lewis et al\., “Retrieval\-Augmented Generation for Knowledge\-Intensive NLP Tasks,” 2020\.[https://arxiv\.org/abs/2005\.11401](https://arxiv.org/abs/2005.11401)\.
- \[2\]C\. Lefebvre et al\., “Chapter 4: Searching for and selecting studies,” in*Cochrane Handbook for Systematic Reviews of Interventions*, version 6\.5\.1, updated March 2025\.[https://www\.cochrane\.org/authors/handbooks\-and\-manuals/handbook/current/chapter\-04](https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-04)\.
- \[3\]R\. Nogueira and K\. Cho, “Passage Re\-ranking with BERT,” 2019\.[https://arxiv\.org/abs/1901\.04085](https://arxiv.org/abs/1901.04085)\.
- \[4\]J\. Lee et al\., “Gemini Embedding: Generalizable Embeddings from Gemini,” 2025\.[https://arxiv\.org/abs/2503\.07891](https://arxiv.org/abs/2503.07891)\.
- \[5\]L\. Wang et al\., “Text Embeddings by Weakly\-Supervised Contrastive Pre\-training,” 2022, revised 2024\.[https://arxiv\.org/abs/2212\.03533](https://arxiv.org/abs/2212.03533)\.
- \[6\]G\. Comanici et al\., “Gemini 2\.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities,” 2025\.[https://arxiv\.org/abs/2507\.06261](https://arxiv.org/abs/2507.06261)\.