瓶颈在于检索而非阅读：用RAG诊断长上下文问答中的基于证据的生成

# The Bottleneck Is Retrieval, Not Reading: Diagnosing Evidence-Based Generation with RAG for Long Context Question Answering
Source: [https://arxiv.org/html/2610.26951](https://arxiv.org/html/2610.26951)
Owais AijazAhmer TabassumZhuohan XieAffiliation:MBZUAI\{deb\.ban89, vesko\.st\}@gmail\.compreslav\.nakov@mbzuai\.ac\.ae[Project](https://debobanerjee.github.io/LC-EBG-project/)[Code](https://github.com/debobanerjee/LC-EBG)Preslav NakovAffiliation:MBZUAI\{deb\.ban89, vesko\.st\}@gmail\.compreslav\.nakov@mbzuai\.ac\.ae[Project](https://debobanerjee.github.io/LC-EBG-project/)[Code](https://github.com/debobanerjee/LC-EBG)Veselin StoyanovAffiliation:MBZUAI\{deb\.ban89, vesko\.st\}@gmail\.compreslav\.nakov@mbzuai\.ac\.ae[Project](https://debobanerjee.github.io/LC-EBG-project/)[Code](https://github.com/debobanerjee/LC-EBG)Affiliation:Inception42

###### Abstract

Retrieval\-augmented generation \(RAG\) is the dominant paradigm for grounding large language models in external corpora, but its similarity\-driven, chunked pipeline has structural blind spots: it cannot anticipate which passages a question requires and discards document\-level context that resolves cross\-passage references\. We introduce*evidence\-based generation*\(EBG\), requiring models to return an answer with line\-level citations, and a jointAns\+Evidmetric that scores correctness and grounding simultaneously\. We reformulate two corpora into evidence\-grounded multi\-document QA benchmarks \-MultiHop\-RAGandQASPER\-MP, a multi\-paper adaptation of QASPER with realistic short paper cues, and pair them with a controlled synthetic two\-hop haystack and an evidence\-augmented NoLiMa spanning 10K–1M characters\. Across ten proprietary frontier and three open\-weight LLMs, EBG isolates two RAG limitations that long\-context models avoid: similarity retrieval surfaces*any*relevant chunk far more reliably than*all*required chunks for multi\-fact queries, and chunking destroys document meta\-information needed to resolve identity\-bearing queries\. EBG also exposes grounding failures invisible to answer\-only evaluation, including incomplete citations in two\-needle settings and a widening gap betweenAnsandAns\+Evidwhen context contradicts parametric knowledge\. Data and code are available at[https://github\.com/debobanerjee/LC\-EBG](https://github.com/debobanerjee/LC-EBG)\.

## 1Introduction

Large Language Models \(LLMs\) can rapidly process large text volumes to identify information relevant to a query\. This capability suits applications such as legal discovery, medical analysis, financial due diligence, literature review, etc\., which require synthesis from large document collections\.

The predominant paradigm for grounding LLM outputs in external knowledge is retrieval\-augmented generation\([Lewis et al\., 2020](https://arxiv.org/html/2610.26951#bib.bib3), RAG;\), which retrieves passages via embedding similarity and conditions the generator on those passages alone\. While effective for*single\-fact lookups*, this design has several structural limitations\. First, each retrieved chunk is stripped of its surrounding context: the model cannot see where a paragraph falls within a document or how it relates to neighboring sections\. Second, document\-level metadata \(authorship, date, section hierarchy\) is typically lost during chunking\. Third, and most critically, similarity\-based retrieval cannot anticipate which combination of passages will be needed for questions requiring integration across multiple locations\. This is particularly acute for*multi\-hop questions*, where the answer depends on chaining facts that may appear in distant, semantically dissimilar passages\([Karpukhin et al\., 2020](https://arxiv.org/html/2610.26951#bib.bib1);[Li et al\., 2024](https://arxiv.org/html/2610.26951#bib.bib9)\)\.

Our goal in this paper is not to advocate replacing RAG with long\-context \(LC\) models outright, but to make these structural limitations measurable\. RAG systems are typically evaluated on answer correctness alone, which conflates two very different failure modes: the retriever fails to surface the right passages, or the reader fails to use them\. To separate the two, we propose*evidence\-based generation \(EBG\)*, a paradigm where the model must produce both an answer and explicit references to the input lines that support it\. EBG inherits RAG’s built\-in verifiability\([Lewis et al\., 2020](https://arxiv.org/html/2610.26951#bib.bib3)\), but extends it to long\-context settings where the model is given the full document neighborhood and must*select*its own evidence\. The jointAns\+Evidmetric pinpoints where a pipeline breaks: in retrieval, in grounding, or in reading\. Recent work shows LC models can match or exceed RAG accuracy given sufficient context\([Li et al\., 2024](https://arxiv.org/html/2610.26951#bib.bib9);[Laitenberger et al\., 2025](https://arxiv.org/html/2610.26951#bib.bib13)\), but without provenance these comparisons are inconclusive; EBG closes that gap\.

We make the following contributions:

- •EBG framework\.We define the evidence\-based generation task and a jointAns\+Evidmetric, and evaluate ten proprietary frontier and three open\-weight LLMs under matched LC and RAG conditions across 10K–1M character contexts\.
- •Evidence\-grounded benchmarks\.We reformulate two corpora into evidence\-grounded multi\-document QA benchmarks—a re\-instrumentedMultiHop\-RAGwith line\-level evidence labels, andQASPER\-MP, a multi\-paper QASPER adaptation with realistic paper cues, and pair them with a controlled two\-hop synthetic haystack and an evidence\-augmentedNoLiMa\. The first two stress retrieval; the last two stress long\-context reading\.
- •Characterizing RAG’s limitations\.Using EBG, we isolate two failure modes of similarity\-based RAG that LC sidesteps: \(i\) retrieval recovers*any*relevant chunk far more reliably than*all*required chunks for multi\-fact queries, and \(ii\) chunking discards document meta\-information that identity\-bearing queries depend on\. OnMultiHop\-RAG, the retrieval ceiling at realistic top\-k budgets, not the reader, is the binding constraint\.
- •Diagnostic analysis of LC grounding\.Beyond the RAG comparison, EBG reveals that citation acts as a reasoning scaffold, that models cite incomplete evidence in two\-needle settings, and that grounding degrades sharply when in\-context evidence contradicts parametric knowledge\.

## 2Related Work

### 2\.1Long\-Context Language Models

Advances in transformer architectures have expanded context windows from thousands to hundreds of thousands of tokens through improved positional encoding such as ALiBi\([Press et al\., 2022](https://arxiv.org/html/2610.26951#bib.bib6)\), RoPE extensions\([Su et al\., 2024](https://arxiv.org/html/2610.26951#bib.bib7)\), and memory\-augmented attention\([Wu et al\., 2022](https://arxiv.org/html/2610.26951#bib.bib8)\)\. Training data matters equally, as pretraining corpora rarely contain long\-range dependencies;[Zhuang et al\. \(2025\)](https://arxiv.org/html/2610.26951#bib.bib14)construct data with long\-distance references encouraging use of distant context\. Yet larger windows do not automatically yield better reasoning\.[Du et al\. \(2025\)](https://arxiv.org/html/2610.26951#bib.bib11)show accuracy degrades with context length even under perfect retrieval, suggesting a processing bottleneck\. Similarly,[Liu et al\. \(2024\)](https://arxiv.org/html/2610.26951#bib.bib10)show performance drops when information appears mid\-input rather than at boundaries\.

### 2\.2Evaluating Long\-Context Reasoning

Early long\-context evaluations relied on needle\-in\-a\-haystack tests\([Kamradt, 2024](https://arxiv.org/html/2610.26951#bib.bib18)\), which mainly measure retrieval of planted information rather than reasoning\. Subsequent work argues that such evaluations set the bar too low, since strong performance often reflects shallow heuristics rather than genuine understanding\([Liu et al\., 2024](https://arxiv.org/html/2610.26951#bib.bib10);[Schuster et al\., 2025](https://arxiv.org/html/2610.26951#bib.bib19);[Modarressi et al\., 2025](https://arxiv.org/html/2610.26951#bib.bib5);[Zhang et al\., 2024](https://arxiv.org/html/2610.26951#bib.bib12);[Yen et al\., 2025](https://arxiv.org/html/2610.26951#bib.bib15)\)\. Recent benchmarks shift toward more demanding evaluations:∞\\inftyBench\([Zhang et al\., 2024](https://arxiv.org/html/2610.26951#bib.bib12)\)studies degradation beyond 100K characters, HELMET\([Yen et al\., 2025](https://arxiv.org/html/2610.26951#bib.bib15)\)shows that benchmark choice substantially alters rankings, and NoLiMa\([Modarressi et al\., 2025](https://arxiv.org/html/2610.26951#bib.bib5)\)removes lexical shortcuts by requiring semantic matching between queries and evidence\. However, these benchmarks still score models on final answers alone, leaving open whether correct answers stem from actual use of relevant context or from priors, surface cues, or chance\. Unlike these answer\-only benchmarks, our work focuses on grounding by requiring models to return both answers and supporting input lines, distinguishing genuine reasoning from lucky guessing and enabling a more faithful assessment of whether models use long contexts effectively and reliably\.

### 2\.3Retrieval\-Augmented Generation

Retrieval\-augmented generation \(RAG\)\([Lewis et al\., 2020](https://arxiv.org/html/2610.26951#bib.bib3)\)remains the dominant approach for grounding LLMs in external knowledge\. Built on dense retrieval\([Karpukhin et al\., 2020](https://arxiv.org/html/2610.26951#bib.bib1)\)and multi\-passage readers such as Fusion\-in\-Decoder\([Izacard and Grave, 2021](https://arxiv.org/html/2610.26951#bib.bib2)\), RAG provides provenance by construction since retrieved passages serve as evidence\. Recent work revisits RAG alongside long\-context models, highlighting a trade\-off between the two paradigms\([Li et al\., 2024](https://arxiv.org/html/2610.26951#bib.bib9);[Laitenberger et al\., 2025](https://arxiv.org/html/2610.26951#bib.bib13)\): long\-context prompting generally improves accuracy but at substantially higher cost, motivating hybrid systems that route queries by difficulty\([Li et al\., 2024](https://arxiv.org/html/2610.26951#bib.bib9)\)\. Controlled comparisons also show well\-tuned single\-stage RAG can match elaborate multi\-stage pipelines once token budgets are equalized\([Laitenberger et al\., 2025](https://arxiv.org/html/2610.26951#bib.bib13)\), while adaptive retrieval and self\-reflection improve factuality and citation quality\([Asai et al\., 2024](https://arxiv.org/html/2610.26951#bib.bib4)\)\. Existing work primarily evaluates these approaches on accuracy or efficiency\. We instead focus on grounding, distinguishing whether evidence was available from whether the model actually used it\.

## 3Evidence\-based Generation \(EBG\)

We formalizeevidence\-based generation \(EBG\)as a grounded question\-answering paradigm that extends standard answer\-only evaluation by requiring models to generate both answers and explicit citations to the supporting input evidence\.

### 3\.1Answer\-only vs Evidence\-based Generation

Existing long\-context evaluations, including NoLiMa\([Modarressi et al\., 2025](https://arxiv.org/html/2610.26951#bib.bib5)\), use answer\-only evaluation, omitting the grounding needed to validate model reasoning\. EBG extends this formulation by requiring models to generate both explicitevidencereferences to supporting input lines and the finalanswer, yielding a stricter evaluation setting in which correctness depends on both accurate reasoning and faithful grounding\.

To illustrate the distinction, consider a haystack containing the sentence“Then Stuart mentioned that he has been vegan for years\.”at line 1,284 and the question“Which character cannot eat Brandade?”\- a two\-hop query requiring the inference thatBrandadecontainsfishand is therefore incompatible withveganism\. The answer\-only and EBG settings differ in both the system instruction and the required output format\.

Answer\-only Prompt \(User\)You will answer a question based on the following book snippet: \{haystack\} Use the information provided in the book snippet to answer the question\. Your answer should be short and based on either explicitly stated facts or strong, logical inferences\. Question: Which character cannot eat Brandade? Return only the final answer with no additional explanation or reasoning\.

Model ResponseStuart

Under answer\-only evaluation, the response‘‘Stuart’’is simply marked correct\. But under EBG, the same response is evaluated using three complementary metrics: answer accuracy \(Ans\), evidence accuracy \(Evid\), and their conjunction \(Ans\+Evid\)\. WhileAnspreserves standard answer correctness,Ans\+Evidadditionally requires the model to cite the supporting evidence at line 1,284\. Together, these metrics separate answer correctness from evidence\-grounded correctness\.

EBG Prompt \(System \+ User\)\[system\]Your job is to answer the question entirely from the context and provide a reference\. Your answer should cite all lines the answer is based on\. \[user\]<context\>\{haystack\}</context\> Answer the question based on information only from the context\. If the question is not answerable from the context, answer NA\. Your response should comprise only the answer and all lines the answer is based on in json format\. For example: \{ ‘‘lines’’: \[25, 412\],‘‘answer’’: ‘‘John’’ \} Question: Which character cannot eat Brandade?

Model Response\{ ‘‘lines’’: \[1284\], ‘‘answer’’: ‘‘Stuart’’ \}

This decomposition exposes silent failures hidden by answer\-only evaluation: a model may arrive at the correct answer while citing irrelevant lines, or retrieve the correct evidence while producing the wrong answer\. The jointAns\+Evidmetric therefore serves as the strictest measure of evidence\-based reasoning, requiring both accurate reasoning and faithful grounding\.

We also evaluate vanilla Chain\-of\-Thought \(CoT\) prompting, following[Modarressi et al\. \(2025\)](https://arxiv.org/html/2610.26951#bib.bib5), and a CoT\-based EBG prompting variant\. The prompt templates are in Appendix[A](https://arxiv.org/html/2610.26951#A1)\.

### 3\.2RAG under EBG

In this work, we evaluate RAG systems*within the EBG framework*by requiring models to not only answer using retrieved chunks, but also to explicitly cite the supporting evidence\. This yields two RAG variants:

- •Answer\-only RAG:Standard formulation requiring answer correctness only\.
- •Answer \+ Evidence RAG:EBG formulation requiring correct answer and correct citation\.

This unified view allows us to decompose end\-to\-end performance into two factors:

- •Evidence availability:whether the retriever surfaces the relevant needle\(s\) \(Hit@k\),
- •Evidence utilization:whether the model correctly uses and cites the retrieved evidence \(long\-context accuracy\)\.

Overall, our formulation shows that RAG and EBG are not competing paradigms butcomplementary components: retrieval determines*what evidence is available*, while EBG determines*whether that evidence is correctly used and cited*\.

## 4Benchmarks

To systematically evaluate EBG and RAG under EBG, we construct four complementary benchmarks: a fully synthetic benchmark, an evidence\-augmented version of NoLiMa, MultiHop\-RAG, and QASPER\-MP \(an adapted version of QASPER\)\. Together, they evaluate EBG under both controlled synthetic settings and realistic long\-context scenarios\.

### 4\.1Synthetic Dataset

The synthetic benchmark provides precise control over haystack content and evidence placement\. It consists of a distractor fact pool, a curated needle set, and a haystack generation procedure\. This controlled construction allows us to vary context length independently of task difficulty while retaining exact ground\-truth evidence locations\.

##### Distractor fact pool\.

We construct a fixed pool of66,834 unique single\-sentence factsusing iterative LLM prompting followed by deduplication, ensuring that distractors are not confusable with inserted needles\. The large pool also limits repeated distractors across independently generated trials and provides sufficient diversity for constructing long haystacks\.

##### Needle set\.

We manually curate10 two\-hop needles, each containing: i\) two*atomic*supporting facts, ii\) a compositional question requiring*both*facts, and iii\) accepted answer surface forms\. The needles span relations such as*company–product*associations,*founder–specialty*links, and*birthplace*lookups\. By requiring composition across two independently placed facts, the benchmark tests both evidence retrieval and integration rather than simple single\-fact recall\.

##### Haystack assembly\.

For a target lengthLL, distractor facts are shuffled and accumulated until reachingLLcharacters, after which the two needle facts are inserted at independently sampled positions and recorded as gold evidence\. Haystacks are formatted as line\-numberedi: factentries and evaluated under the EBG prompt template \(§[A](https://arxiv.org/html/2610.26951#A1)\)\. Each trial uses a newly generated haystack and a uniformly sampled needle\. Consequently, performance differences cannot be attributed to a fixed ordering of distractors or evidence positions\. Appendix[B\.1](https://arxiv.org/html/2610.26951#A2.SS1)provides a representative example\.

##### Sanity check\.

AtL=10​KL\{=\}10\\text\{K\}\(≈2​K\{\\approx\}2\\text\{K\}tokens\), all evaluated frontier models achieve at least0\.980\.98jointAns\+Evidaccuracy overN=100N\{=\}100trials, confirming that the inserted needles are unambiguous and readily solvable when context length is modest\. Any degradation at largerLLtherefore reflects failures in long\-context evidence retrieval or integration rather than intrinsic task ambiguity\. This makes the synthetic benchmark a controlled probe of how model reliability changes as relevant evidence becomes increasingly sparse within the surrounding context\.

### 4\.2NoLiMa\-based Dataset

We adapt the NoLiMa haystack construction procedure to create long\-context inputs ranging fromL=10​KL\{=\}10\\text\{K\}to1​M1\\text\{M\}characters and require models to both answer questions and cite supporting needles as evidence\. This converts NoLiMa from an answer\-only benchmark into anevidence\-grounded evaluationthat tests whether models truly use the provided context\.

##### Needle set\.

We use the originalsingle\-needle\(SN\) setting from NoLiMa\([Modarressi et al\., 2025](https://arxiv.org/html/2610.26951#bib.bib5)\), specifically the0402and0405needle groups with both default and inverse variants under thecommonsensecategory\. Each needle contains a character name and a single factual attribute, and supports both one\-hop and two\-hop questions\.

We further introduce atwo\-needle\(TN\) setting, in which theexplicit secondary needleis consistent with world knowledge, and acontradictory two\-needle\(CTN\) variant, in which it conflicts with world knowledge\. Appendix[C\.1](https://arxiv.org/html/2610.26951#A3.SS1)provides additional details and templates forSN,TN, andCTNsettings\.

##### Haystack assembly\.

We adopt the random book generation procedure from NoLiMa\([Modarressi et al\., 2025](https://arxiv.org/html/2610.26951#bib.bib5)\)to construct haystacks ranging from 10K to 1M characters \(10K–50K in 10K increments and 100K–1M in 100K increments\), generating three random booksB1c,B2c,B3c\{B^\{c\}\_\{1\},B^\{c\}\_\{2\},B^\{c\}\_\{3\}\}for each context lengthcc\. Because tokenization is model\-dependent, we vary input length in characters rather than tokens\. Each haystack is split into line\-indexed entries using the ‘\\n’ delimiter\. In the single\-needle setting, the needle is inserted at one of four normalized depths: 0, 0\.33, 0\.67, or 1\.0\. In the two\-needle setting, the primary needle is fixed at depth 0\.5, while the secondary needle is inserted at one of the same four depths\.

### 4\.3Multi\-document QA Datasets

To stress the retrieval pipeline on the dimensions it is meant to handle, we reformulate two existing corpora into evidence\-grounded multi\-document question answering \(QA\) benchmarks\. Both are designed to make RAG’s retrieval\-vs\.\-reading split observable: gold evidence is labeled at the line \(sentence or paragraph\) level, so the same scoring code works for LC and any retriever\-ordered context\.

##### MultiHop\-RAG\([Tang and Yang, 2024](https://arxiv.org/html/2610.26951#bib.bib16)\)\.

The official release contains609 news articles\(4\.6M characters, 1\.2M tokens with thecl100k\_basetokenizer\) and 2,556 queries in four templated types:*inference*,*comparison*,*temporal*, andnull\_query\(unanswerable\)\. Each non\-null query carries anevidence\_listof2–4 gold facts, each consisting of a source article title and an exact textual span\. Removing the 301null\_queryitems leaves2,255 usable queries; for the pilot reported in this paper, we uniformly sample 50 queries \(seed 0\), naturally stratified across the three answerable types \(ninf=13n\_\{\\text\{inf\}\}\{=\}13,ncomp=21n\_\{\\text\{comp\}\}\{=\}21,ntemp=16n\_\{\\text\{temp\}\}\{=\}16\)\.

For the LC condition, we construct an*oracle\-containing*per\-query mini\-haystack comprising every gold\-evidence article plus 20 distractor articles sampled from the remainder of the corpus\. The articles are concatenated in random order and split into sentence\-per\-line entries, producing contexts of approximately 241K characters on average\. For the RAG condition, we perform*open\-corpus retrieval*over the full 609\-article corpus: we divide the corpus into 256\-token sliding windows \(6,296 chunks\), embed the chunks withtext\-embedding\-3\-large, and rank them by cosine similarity to produce top\-KKcontexts forK∈\{4,10,25,50,100,200\}K\\in\\\{4,10,25,50,100,200\\\}\.

The two conditions are therefore not pool\-matched: LC evaluates reading and evidence utilization when all gold source articles are guaranteed present, whereas RAG must first retrieve the required evidence from the full corpus\. Accordingly, this experiment diagnoses the contribution of open\-corpus retrieval failures to end\-to\-end performance; it is not a direct LC–RAG comparison under identical candidate pools\. In both conditions, gold facts are re\-mapped to line numbers inside the assembled context; in RAG, a gold fact that is not retrieved has no corresponding gold line\. Appendix[D](https://arxiv.org/html/2610.26951#A4)reports the corresponding retrieval ceilings \(Hits@kkfor*any*vs\.*all*gold facts\) and reproduces the answer\-only RAG baselines from the original paper\.

##### QASPER\-MP\.

QASPER\([Dasigi et al\., 2021](https://arxiv.org/html/2610.26951#bib.bib17)\)contains 5,049 information\-seeking questions over 1,585 NLP papers, each annotated by domain experts with paragraph\-level evidence and a short free\-form or extractive answer \(includingUnanswerable\)\. The original task is single\-document and, therefore, does not exercise the retrieval pipeline\.

We constructQASPER\-MP\(multi\-paper\) by pairing each gold paper withten distractor paperssampled from the QASPER pool, yielding an 11\-paper haystack per query, and rewrite every question to include a realistic paper cue \- in∼\\sim50% of cases a first\-author surname \(e\.g\., “Smith et al\.”\), and otherwise a 2–3 word verbatim title fragment \- split by a hash of the question id\. This forces the system to resolve document identity from a realistic cue rather than the full paper title\. The task jointly tests document identification, evidence localization, and answer generation\. Since distractors come from the same NLP\-paper collection, they provide domain\-matched context\. A successful system must identify the relevant paper and supporting paragraph\(s\) before answering, while retaining QASPER’s expert\-annotated answers and evidence\.

For LC, papers are serialized in an XML\-like format that preserves titles, abstracts, and section structure \(<paper\>,<title\>,<abstract\>,<section\>\); paragraphs are line\-numbered, and the same paragraph\-level gold labels are used for evidence scoring\. For RAG, we evaluate both a*flat*variant \(concatenate the top\-KKretrieved paragraphs in rank order\) and a*structured*variant that re\-emits the surrounding<paper\>/<title\>/<abstract\>scaffolding and inserts “\.\.\.” markers for discontinuities, so that retrieved fragments retain their document provenance\.

We evaluateQASPER\-MPat two scales\. For proprietary models, we conduct a pilot onN=50N\{=\}50queries, comprising 38 answerable and 12 unanswerable examples\. For open\-weight models, we expand the evaluation toN=200N\{=\}200queries, providing broader benchmark coverage and a more stable estimate of aggregate performance\. Because the proprietary\-model results are based on the smaller pilot, comparisons across the two model groups should be interpreted with the corresponding difference in sampling uncertainty in mind\. Construction details, sampling procedures, prompts, and the LLM\-judge protocol are provided in Appendix[E](https://arxiv.org/html/2610.26951#A5)\.

## 5Experiments

We organize our experiments around four research questions:

- •RQ1\.What are the practical limitations of RAG, and can long\-context EBG address them?
- •RQ2\.Does requiring models to explicitly cite evidence improve answer accuracy?
- •RQ3\.Can models correctly cite evidence lines even in a two\-needle setting?
- •RQ4\.Can models remain context\-faithful even when evidence contradicts their built\-in knowledge?

### 5\.1Experimental Setup

We evaluate 10 state\-of\-the\-art proprietary models from three major providers across different experiments:GPT\-4o,GPT\-4\.1,o3\-mini,GPT\-5,GPT\-5\.5,Sonnet\-4,Sonnet\-4\.5,Opus\-4\.6,Gemini\-2\.5\-Fl, andGemini\-3\-Fl\. Additionally we evaluate three open\-weight models served through Fireworks\.ai111[https://fireworks\.ai/](https://fireworks.ai/)\-GPT\-OSS\-120B,GLM\-5\.2, andNemotron\-3\.5\-Lightning\-30B\-A3B\(orNemo\-3\.5\-Lin short\) \- to test whether our findings hold beyond proprietary systems\. We evaluate evidence\-based generation \(EBG\) under both long\-context \(LC\) andRAGsettings\. Prompt templates for the*answer\-only*and*EBG*conditions are provided in Appendix[A](https://arxiv.org/html/2610.26951#A1)\.

#### 5\.1\.1Multi\-document QA setup\.

ForRQ1, we use two multi\-document QA datasets \-MultiHop\-RAGandQASPER\-MP\. Construction details, retrieval variants, and the LLM\-judge protocol are in Appendices[D](https://arxiv.org/html/2610.26951#A4)and[E](https://arxiv.org/html/2610.26951#A5)\.

#### 5\.1\.2EBG Setup

##### Synthetic Dataset\.

We sweep context lengths from 10K to 1M characters\. For each length, we runN=100N\{=\}100trials: each trial draws a fresh distractor haystack and samples one needle uniformly from the 10 curated two\-hop needles\. Here we apply theanswer\-onlyandEBGprompts and reportAns\*,Ans,Evid, andAns\+Evidaccuracies\. A sanity check atL=10​KL\{=\}10\\text\{K\}confirms all models achieve≥0\.98\{\\geq\}0\.98joint accuracy, establishing that degradation at longer lengths reflects context handling rather than needle ambiguity\. This setup addressesRQ2andRQ3; there are no contradictory needles, so RQ4 is not evaluated here\.

##### NoLiMa\-based Dataset\.

Under the NoLiMa setting, we useB1cB^\{c\}\_\{1\}as the haystack source across all context lengths to maintain a controlled evaluation setup\. We use theT01test variant throughout, as it provides exactly one input argument for the one\-hop question and one for the two\-hop question associated with each needle\. This enables a consistent comparison between direct retrieval and commonsense composition while holding the underlying needle fixed\. For example, for0405, the one\-hop question is instantiated withfish\-based meals, whereas the two\-hop question usesBrandade; similarly,0402usesmilkanda cappuccino, respectively\. We test two characters,YukiandStuart, at the insertion depths defined in §[4\.2](https://arxiv.org/html/2610.26951#S4.SS2), yielding 16 needle\-placement conditions per context length per model \(4 needles×\\times4 depths\)\.

We evaluate three settings:single\-needle\(one\-hop and two\-hop, addressingRQ2\),two\-needle\(two\-hop with explicit secondary evidence, addressingRQ3\), andcontradictory two\-needle\(CTN, addressingRQ4\)\. Results are reported separately for one\-hop and two\-hop questions and averaged across the corresponding conditions\.

#### 5\.1\.3NoLiMa\-based RAG Setup

We compare long\-contextLCandRAGon the NoLiMa\-based dataset underRQ1\(Appendix[G](https://arxiv.org/html/2610.26951#A7)\)\. We first analyze RAG retrieval performance for the single\-needle and two\-needle settings using one\-hop and two\-hop questions, examining how the retrieval depth required for full evidence coverage changes with haystack length and reasoning complexity\. After needle insertion, each haystack line is embedded using OpenAI’stext\-embedding\-3\-smallmodel and ranked by cosine similarity to the query\.

We then compareRAGandLCunder the sameEBGprompting, reportingAnsaccuracy\. Since RAG contexts can be smaller than their source haystacks, we use matched character budgets\. For each haystack length, we identify the minimum top\-kkproviding full evidence coverage, compute its character count, and match it to the closest available LC context length\. Table[15](https://arxiv.org/html/2610.26951#A7.T15)summarizes this mapping\. For example, full\-coverage retrieval from the 1M\-character haystack in the two\-needle setting requires 920 vectors \(217,152 characters\) and is matched to the 200K LC condition\. This approximately controls for the amount of text presented to the model and isolates retrieval\-ranked versus long\-context inputs under comparable budgets\.

### 5\.2Results

We report results organized by the research questions introduced in §[5](https://arxiv.org/html/2610.26951#S5)\.

##### RQ1: LC\-EBG addresses two RAG limitations\.

OnMultiHop\-RAG, queries require 2–4 evidence facts spread across distinct articles, so faithful answering requires the retriever to surface all required facts within its top\-kkbudget\. For this benchmark, the two conditions use different evidence\-access regimes:*open\-corpus RAG*searches 6,296 chunks constructed from all609 articles, whereas the*oracle\-containing LC*condition receives every gold\-evidence article plus 20 distractor articles\. The conditions are therefore not pool\-matched; this comparison diagnoses the cost of open\-corpus retrieval relative to reading when the gold source articles are guaranteed to be available\.

We first reproduce the answer\-only RAG baselines from the original benchmark under our pipeline \(text\-embedding\-3\-large\+ 256\-token windows; Appendix[D](https://arxiv.org/html/2610.26951#A4)\) and measure annotated\-gold retrieval coverage directly\. Atk=10k\{=\}10, the retriever covers*any*gold fact for82%82\\%of queries but covers*all*gold facts for only19%19\\%\(and just8%8\\%on inference queries\)\.This all\-vs\-any gap exposes a limitation of the evaluated similarity\-retrieval setup: it often retrieves some relevant evidence but fails to recover the complete evidence set required for multi\-fact queries within a practical top\-kkbudget\.When the gold annotations are exhaustive and every annotated fact is necessary, all\-gold coverage bounds the faithfully grounded performance of the downstream RAG reader\. The reader results corroborate this limitation \(Appendix[D\.5](https://arxiv.org/html/2610.26951#A4.SS5), Table[6](https://arxiv.org/html/2610.26951#A4.T6)\)\.

UnderEBGprompting atk=10k\{=\}10, open\-corpusRAGachievesAnsaccuracies of60/48/44%60/48/44\\%forGPT\-5,Sonnet\-4\.5, andGemini\-3\-Fl, respectively, whereas the oracle\-containingLC\-EBGcondition achieves86/82/78%86/82/78\\%on the same queries\. The resulting2626–3434pp gap includes the cost of retrieving the required evidence from the full corpus and should not be interpreted as a pool\-matched estimate of an intrinsic LC–RAG difference\. The gap remains atk=200k\{=\}200, where the open\-corpus RAG context approaches the size of the LC haystack despite comparable context sizes, but itsAnsaccuracy remains lower for all three models\. Underanswer\-onlyprompting, open\-corpus RAG atk=200k\{=\}200reaches parity with the oracle\-containing LC condition only forSonnet\-4\.5\(68%68\\%vs\.68%68\\%\)\.

Consistent with this, ourNoLiMa\-based RAG analysis \(Table[13](https://arxiv.org/html/2610.26951#A7.T13)\) shows thatretrieval costs scale sharply with reasoning complexity and haystack length\. While one\-hop full coverage remains relatively efficient, two\-hop retrieval grows from 1\.9K retrieved characters at 10K context to over 202K at 1M context, effectively pushing RAG toward brute\-force long\-context processing without guaranteeing complete evidence recovery\.

Table 1:QASPER\-MPanswerable\-subset LLM\-judge scores \(n=38n\{=\}38, EBG\)\.LCwins for all five models; the largest gaps are for the strongest readers \(Opus\-4\.6andGPT\-5\.5\)\. Higher \(↑\\uparrow\) values are better\. Full breakdown in Appendix[E](https://arxiv.org/html/2610.26951#A5)\.OnQASPER\-MP, the query refers to a paper by an author or short title cue; answering requires document\-level identity, not just topical fragments\.This exposes a second limitation of RAG: chunk\-level similarity retrieval struggles with document identity and provenance when queries refer indirectly to a source\.Table[1](https://arxiv.org/html/2610.26951#S5.T1)reports answerable\-subset judge scores for the pool\-matched comparison, whereLC\-EBGandRAGuse the same 11\-paper pool\.LC\-EBGexceeds the best pool\-matched RAG variant for all five frontier models by\+0\.05\+0\.05to\+0\.13\+0\.13judge points\. Aggregate scores look tied only becauseRAGover\-refuses on missed retrievals \(2424–40%40\\%on answerable queries vs\.1313–18%18\\%forLC\-EBG\), accidentally matching QASPER’sUnanswerablegold label on the 24% of pilot questions for which it is correct\. Under the open\-corpus condition \(flat\_full@10\), all\-needles coverage drops to 48%, and the judge\-score gap relative to LC is 5–13 points \(Appendix[E](https://arxiv.org/html/2610.26951#A5)\)\.

Across both datasets, retrieval is a major error source despite different evidence\-access regimes:MultiHop\-RAGcompares open\-corpus RAG with oracle\-containing LC, whereasQASPER\-MPuses a pool\-matched 11\-paper comparison\. This pattern persists for three open\-weight readers onn=200n\{=\}200queries, with LC exceeding RAG by\+13\+13to\+36\+36pp in EBGAnsaccuracy onMultiHop\-RAGand by\+0\.07\+0\.07to\+0\.24\+0\.24answerable\-subset judge points onQASPER\-MP, alongside a large residual grounding gap \(Appendices[D\.7](https://arxiv.org/html/2610.26951#A4.SS7)and[E\.6](https://arxiv.org/html/2610.26951#A5.SS6)\)\.

![Refer to caption](https://arxiv.org/html/2610.26951v1/answer_only_delta_heatmap_extended.png)Figure 1:NoLiMa\-basedLCbenchmark comparingEBG \(Ans\)andanswer\-only\(Ans∗\)prompting across eight proprietary and three open\-weight models over 10K–1M context lengths\. Values report theaccuracy difference \(Δ\\Delta\), with larger positive values \(↑\\uparrow\) indicating greater EBG gains;N/Adenotes inputs exceeding the model’s context window\.
##### RQ2: Citation improves answer accuracy\.

On the synthetic benchmark \(Fig\.[4\(a\)](https://arxiv.org/html/2610.26951#A5.F4.sf1), Appendix[F\.1](https://arxiv.org/html/2610.26951#A6.SS1)\), all three frontier models \- GPT\-5, Claude Sonnet\-4\.5, and Gemini\-3\-Flash \- approach ceiling answer accuracy under both prompting conditions \(answer\-onlyandEBG\)\. GPT\-5 and Gemini\-3\-Flash maintain≥98%\\geq 98\\%accuracy across all tested context lengths, while Sonnet\-4\.5 remains within the9393–100%100\\%range up to 700K characters\. Consequently, the synthetic dataset shows no measurable citation\-as\-scaffold effect on answer accuracy:Ans∗andAnsremain within sampling noise across all cells,including at the longest contexts\. This does not refute RQ2 but reflects the controlled benchmark design: lexically dissimilar distractors make the two\-hop facts relatively easy to isolate, leaving little headroom for EBG’s benefits\.

A different pattern emerges on the more challenging NoLiMa\-basedLCbenchmark \(Fig\.[1](https://arxiv.org/html/2610.26951#S5.F1)\), where the scaffolding effect becomes apparentunder long\-context stress\. Sonnet and Gemini models gain between\+9\+9and\+19\+19pp fromEBGprompting across context lengths from 300K to 900K\. Among open\-weight models, the smallerNemo\-3\.5\-Limproves by\+3\+3to\+12\+12pp, whileGLM\-5\.2reaches a maximum gain of\+16\+16pp at 700K; in contrast,GPT\-OSS\-120Bshows no benefit\. Taken together, the contrast between the synthetic and NoLiMa settings suggests thatscaffolding is most useful when long\-context retrieval is sufficiently challenging, while its effectiveness remains model\-dependent and may not simply scale with model capacity\.

##### RQ3: Asymmetric Retrieval Behavior\.

Figure 2:Two\-needle citation breakdown: rate of correctly citing theprimary needle vs\.thesecondary needleunder NoLiMa\-basedLCbenchmark\. Higher \(↑\\uparrow\) values are better\.As shown in Fig\.[2](https://arxiv.org/html/2610.26951#S5.F2), recovery of the two evidence types varies substantially across models\.Gemini\-3\-FlandGPT\-5achieve near\-perfect and balanced citation of both the primary and secondary needles \(99\.5/99\.8% and 95\.1/98\.6%, respectively\)\.Gemini\-2\.5\-Flalso performs relatively well, but exhibits an inversion, with secondary\-needle citation exceeding primary\-needle citation \(80\.8% vs\. 67\.4%\)\. A similar, more pronounced inversion appears foro3\-mini\(88\.4% vs\. 54\.9%\)\. These cases suggest that successful recovery of one evidence type does not necessarily imply comparable recovery of the other\.

This asymmetry reverses for several other models, where primary\-needle recovery substantially exceeds secondary\-needle recovery\.GPT\-4\.1,GPT\-4o, and theSonnetmodels exhibit this pattern, with the largest disparities observed forGPT\-4o\(71\.3% vs\. 20\.9%\) andSonnet\-4\.5\(64\.6% vs\. 21\.4%\)\. This disparity indicates that retrieving the primary evidence does not guarantee successful attribution of the complementary evidence required for the task\. Moreover, primary\-needle citation itself remains imperfect, showing that retrieval failures extend beyond the secondary signal alone\.

The open\-weight models show substantial variation in retrieval\.GLM\-5\.2achieves balanced recovery \(78\.5% primary and 85\.2% secondary\), whereasGPT\-OSS\-120Brecords lower rates for both needles \(26\.1% and 29\.4%\)\.Nemo\-3\.5\-Lperforms poorly on both \(11\.8% and 6\.5%\), indicating broader retrieval limitation rather than directional asymmetry\.Overall, multi\-evidence retrieval varies across models and evidence types, reinforcing model\-dependent patterns as in RQ1\.

Figure 3:TN vs\. CTN,Ans\+Evid\. TN uses a factual secondary needle; CTN uses a contradictory secondary needle, conflicting with world knowledge, requiring context\-faithful reasoning and citation\. Models to the right of the dashed line are open\-weight\. Higher \(↑\\uparrow\) values are better\.Ansaccuracy for the same runs is reported in Figure[6](https://arxiv.org/html/2610.26951#A6.F6)\.
##### RQ4: CTN amplifies failure\.

Performance generally declines under contradiction, with substantial variation across models \(Fig\.[3](https://arxiv.org/html/2610.26951#S5.F3)\); answer accuracy is reported separately in Figure[6](https://arxiv.org/html/2610.26951#A6.F6)\.GPT\-5andGemini\-3\-Flremain relatively context\-faithful, preserving high answer accuracy, though theirAns\+Evidstill drops, indicating greater sensitivity of evidence grounding to contradiction\. In contrast,GPT\-4\.1,GPT\-4o, and theSonnetvariants show sharper declines, particularly inAns\+Evid, suggesting they can recover the expected answer while failing to reconcile contradictory evidence\.

CTN robustness is also uneven among open\-weight models\.GLM\-5\.2is notably resilient:Ansdecreases only from 79% to 72%, whileAns\+Evidchanges from 73% to 70%, largely preserving the small gap between the two metrics\. By contrast,GPT\-OSS\-120BandNemo\-3\.5\-Lstart from substantially weaker TN baselines: the former falls from 28/8% \(Ans/Ans\+Evid\) to 8/5% under CTN, and the latter from 12/6% to 5/4%\. These models therefore already struggle to extract and ground answers under TN, with CTN further compounding this weakness\.

Overall,Ans\+Evidis generally more vulnerable to CTN thanAns, showing thatresolving contradiction while grounding the answer in supplied evidence is harder than producing the correct answer itself\.The wideningAns–Ans\+Evidgap highlights a common failure mode: models reach the expected answer without faithfully using the context that should justify it, even when answers remain correct\. See Appendix[F\.4](https://arxiv.org/html/2610.26951#A6.SS4)for more details\.

## 6Conclusion and Future Work

We introducedevidence\-based generation \(EBG\), a framework that requires models to produce both an answer and line\-level citations, together with a jointAns\+Evidmetric that measures correctness and grounding simultaneously\. By separating retrieval from evidence\-grounded reasoning, EBG exposes failures obscured by answer\-only evaluation\. We evaluate these failures using evidence\-grounded versions ofMultiHop\-RAGandQASPER\-MP, a controlled synthetic two\-hop haystack, and an evidence\-augmented NoLiMa spanning 10K–1M characters\.This evaluation suite spans both realistic and controlled settings, allowing retrieval, reasoning, and grounding errors to be examined under complementary conditions\.

Across ten proprietary frontier and three open\-weight LLMs, EBG reveals two key RAG limitations:*\(i\)*similarity retrieval finds*any*relevant chunk more reliably than*all*evidence required for multi\-fact queries, imposing a retrieval ceiling at realistic top\-kkbudgets; and*\(ii\)*chunking can discard document\-level meta\-information needed by identity\-bearing queries, causing RAG to over\-refuse on retrieval misses\. EBG further exposes grounding failures hidden by answer\-only metrics: models often cite incomplete evidence, and the gap between answer and joint accuracy widens when in\-context evidence conflicts with parametric knowledge\. These findings show that correct answers alone can substantially overstate a model’s ability to faithfully use long\-context evidence\. They also highlight evidence attribution as a useful diagnostic for distinguishing retrieval limitations from downstream reasoning failures\.Consequently, EBG provides a more informative basis for comparing RAG and long\-context systems than evaluations that collapse retrieval, reasoning, and attribution into answer accuracy alone\.

These results do not imply that “LC replaces RAG”; rather, LC and RAG exhibit distinct failure modes that EBG helps localize, motivating evaluations that measure both answer accuracy and evidence fidelity\.The appropriate architecture may therefore depend not only on answer quality, but also on which sources of failure are acceptable for a given task and context regime\.Future work includes scaling EBG to the full benchmarks and testing document\-aware chunking\. Richer evidence attribution and the observed reasoning benefits of evidence generation also motivate investigating grounding objectives as part of model training\.

## Limitations

Our evaluation focuses on synthetic and semi\-synthetic long\-context benchmarks designed to isolate grounding and evidence attribution behavior under controlled conditions\. While these settings enable precise analysis, they do not capture the full complexity of real\-world domains such as legal, scientific, or medical reasoning\. In addition, our evidence metric operates at the line level and therefore measures coarse\-grained provenance rather than fine\-grained span\-level attribution\. The benchmarks also assume that the reference evidence is sufficiently complete and unambiguous, whereas real\-world questions may admit multiple valid evidence sets or interpretations\.

Our RAG analysis estimates end\-to\-end performance through a decomposition of retrieval and long\-context accuracy rather than exhaustive evaluation of full production RAG pipelines across all configurations\. We therefore do not systematically explore retrieval variants such as query rewriting, hybrid search, or adaptive context selection, which could alter the observed retrieval ceilings\. Finally, because we evaluate proprietary frontier and open\-weight models through API access, results may vary as deployed systems evolve over time\.

## Ethics and Broader Impact

This work studies evidence grounding and provenance in long\-context language models\. Our benchmarks are constructed from synthetic or publicly available text and do not contain sensitive personal information\. We do not train new foundation models; all experiments are conducted through inference\-only evaluation of existing systems\.

We believe evidence\-based evaluation can improve transparency and verifiability in LLM\-based systems by encouraging models to explicitly identify the context used to support their answers\. Such capabilities may be particularly important in high\-stakes applications such as legal analysis, scientific search, and medical document understanding, where unsupported or hallucinated answers can have significant consequences\.

At the same time, citation generation should not be interpreted as a guarantee of faithful reasoning\. A model may cite irrelevant or incomplete evidence while producing a plausible answer\. We therefore view evidence\-based generation as a complementary framework rather than a complete solution to factuality or interpretability in language models\.

## References

- Asaiet al\.\(2024\)A\. Asai, Z\. Wu, Y\. Wang, A\. Sil, and H\. HajishirziSelf\-RAG: learning to retrieve, generate, and critique through self\-reflection\.InProceeding of the 12th International Conference on Learning Representations,ICLR ’24\.External Links:[Link](https://openreview.net/forum?id=hSyW5go0v8)Cited by:[§2\.3](https://arxiv.org/html/2610.26951#S2.SS3.p1.1)\.
- Dasigiet al\.\(2021\)P\. Dasigi, K\. Lo, I\. Beltagy, A\. Cohan, N\. A\. Smith, and M\. GardnerA dataset of information\-seeking questions and answers anchored in research papers\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,K\. Toutanova, A\. Rumshisky, L\. Zettlemoyer, D\. Hakkani\-Tur, I\. Beltagy, S\. Bethard, R\. Cotterell, T\. Chakraborty, and Y\. Zhou \(Eds\.\),NAACL ’21,Online,pp\. 4599–4610\.External Links:[Link](https://aclanthology.org/2021.naacl-main.365/),[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.365)Cited by:[Appendix E](https://arxiv.org/html/2610.26951#A5.p1.1),[Table 17](https://arxiv.org/html/2610.26951#A7.T17.2.1.5.2),[Appendix H](https://arxiv.org/html/2610.26951#A8.p1.1),[§4\.3](https://arxiv.org/html/2610.26951#S4.SS3.SSS0.Px2.p1.1)\.
- Duet al\.\(2025\)Y\. Du, M\. Tian, S\. Ronanki, S\. Rongali, S\. B\. Bodapati, A\. Galstyan, A\. Wells, R\. Schwartz, E\. A\. Huerta, and H\. PengContext length alone hurts llm performance despite perfect retrieval\.InFindings of the Association for Computational Linguistics,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),EMNLP ’25,Suzhou, China,pp\. 23281–23298\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.1264/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1264)Cited by:[§2\.1](https://arxiv.org/html/2610.26951#S2.SS1.p1.1)\.
- Izacard and Grave \(2021\)G\. Izacard and E\. GraveLeveraging passage retrieval with generative models for open domain question answering\.InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume,P\. Merlo, J\. Tiedemann, and R\. Tsarfaty \(Eds\.\),EACL ’21,Online,pp\. 874–880\.External Links:[Link](https://aclanthology.org/2021.eacl-main.74/),[Document](https://dx.doi.org/10.18653/v1/2021.eacl-main.74)Cited by:[§2\.3](https://arxiv.org/html/2610.26951#S2.SS3.p1.1)\.
- Kamradt \(2024\)G\. KamradtNeedle In A Haystack\.Note:Accessed: 2026\-08\-30External Links:[Link](https://github.com/gkamradt/needle-in-a-haystack)Cited by:[§2\.2](https://arxiv.org/html/2610.26951#S2.SS2.p1.1)\.
- Karpukhinet al\.\(2020\)V\. Karpukhin, B\. Oguz, S\. Min, P\. Lewis, L\. Wu, S\. Edunov, D\. Chen, and W\. YihDense passage retrieval for open\-domain question answering\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing,B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),EMNLP ’20,Online,pp\. 6769–6781\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.550/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.550)Cited by:[§1](https://arxiv.org/html/2610.26951#S1.p2.1),[§2\.3](https://arxiv.org/html/2610.26951#S2.SS3.p1.1)\.
- Laitenbergeret al\.\(2025\)A\. Laitenberger, C\. D\. Manning, and N\. F\. LiuStronger baselines for retrieval\-augmented generation with long\-context language models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),EMNLP ’25,Suzhou, China,pp\. 32559–32569\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1656/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1656)Cited by:[§1](https://arxiv.org/html/2610.26951#S1.p3.1),[§2\.3](https://arxiv.org/html/2610.26951#S2.SS3.p1.1)\.
- Lewiset al\.\(2020\)P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. KielaRetrieval\-augmented generation for knowledge\-intensive nlp tasks\.InProceeding of the 2020 Conference on Advances in Neural Information Processing Systems,NIPS ’20\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html)Cited by:[§1](https://arxiv.org/html/2610.26951#S1.p2.1),[§1](https://arxiv.org/html/2610.26951#S1.p3.1),[§2\.3](https://arxiv.org/html/2610.26951#S2.SS3.p1.1)\.
- Liet al\.\(2024\)Z\. Li, C\. Li, M\. Zhang, Q\. Mei, and M\. BenderskyRetrieval augmented generation or long\-context llms? A comprehensive study and hybrid approach\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track,F\. Dernoncourt, D\. Preoţiuc\-Pietro, and A\. Shimorina \(Eds\.\),EMNLP ’24,Miami, Florida, USA,pp\. 881–893\.External Links:[Link](https://aclanthology.org/2024.emnlp-industry.66/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-industry.66)Cited by:[§1](https://arxiv.org/html/2610.26951#S1.p2.1),[§1](https://arxiv.org/html/2610.26951#S1.p3.1),[§2\.3](https://arxiv.org/html/2610.26951#S2.SS3.p1.1)\.
- Liuet al\.\(2024\)N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. LiangLost in the Middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.External Links:[Link](https://aclanthology.org/2024.tacl-1.9/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00638)Cited by:[§2\.1](https://arxiv.org/html/2610.26951#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2610.26951#S2.SS2.p1.1)\.
- Modarressiet al\.\(2025\)A\. Modarressi, H\. Deilamsalehy, F\. Dernoncourt, T\. Bui, R\. A\. Rossi, S\. Yoon, and H\. SchuetzeNoLiMa: long\-context evaluation beyond literal matching\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 44554–44570\.External Links:[Link](https://proceedings.mlr.press/v267/modarressi25a.html)Cited by:[Table 17](https://arxiv.org/html/2610.26951#A7.T17.2.1.3.2),[Appendix H](https://arxiv.org/html/2610.26951#A8.p1.1),[§2\.2](https://arxiv.org/html/2610.26951#S2.SS2.p1.1),[§3\.1](https://arxiv.org/html/2610.26951#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2610.26951#S3.SS1.p9.1),[§4\.2](https://arxiv.org/html/2610.26951#S4.SS2.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2610.26951#S4.SS2.SSS0.Px2.p1.1)\.
- Presset al\.\(2022\)O\. Press, N\. Smith, and M\. LewisTrain short, test long: attention with linear biases enables input length extrapolation\.InProceedings of the 10th International Conference on Learning Representations,ICLR ’22\.External Links:[Link](https://openreview.net/forum?id=R8sQPpGCv0)Cited by:[§2\.1](https://arxiv.org/html/2610.26951#S2.SS1.p1.1)\.
- Schusteret al\.\(2025\)T\. Schuster, M\. Lambert, N\. Döring, and J\. TrögeleNeedle\-in\-the\-Haystack testing LLMs with a complex reasoning task\.InProceedings of the 26th International Conference on Engineering Applications of Neural Networks,L\. Iliadis, I\. Maglogiannis, E\. Kyriacou, and C\. Jayne \(Eds\.\),EANN ’25,Limassol, CyprusLimassol, Cyprus,pp\. 254–266\.External Links:[Link](https://link.springer.com/chapter/10.1007/978-3-031-96196-0_19),ISBN 978\-3\-031\-96196\-0,[Document](https://dx.doi.org/10.1007/978-3-031-96196-0%5F19)Cited by:[§2\.2](https://arxiv.org/html/2610.26951#S2.SS2.p1.1)\.
- Suet al\.\(2024\)J\. Su, M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. LiuRoFormer: enhanced transformer with rotary position embedding\.Neurocomputing568\(C\)\.External Links:ISSN 0925\-2312,[Link](https://doi.org/10.1016/j.neucom.2023.127063),[Document](https://dx.doi.org/10.1016/j.neucom.2023.127063)Cited by:[§2\.1](https://arxiv.org/html/2610.26951#S2.SS1.p1.1)\.
- Tang and Yang \(2024\)Y\. Tang and Y\. YangMultiHop\-RAG: benchmarking retrieval\-augmented generation for multi\-hop queries\.arXiv:2401\.15391\.External Links:2401\.15391,[Link](https://arxiv.org/abs/2401.15391)Cited by:[Appendix D](https://arxiv.org/html/2610.26951#A4.p1.1),[Table 17](https://arxiv.org/html/2610.26951#A7.T17.2.1.4.2),[Appendix H](https://arxiv.org/html/2610.26951#A8.p1.1),[§4\.3](https://arxiv.org/html/2610.26951#S4.SS3.SSS0.Px1)\.
- Wuet al\.\(2022\)Q\. Wu, Z\. Lan, K\. Qian, J\. Gu, A\. Geramifard, and Z\. YuMemformer: a memory\-augmented transformer for sequence modeling\.InFindings of the Association for Computational Linguistics,Y\. He, H\. Ji, S\. Li, Y\. Liu, and C\. Chang \(Eds\.\),AACL\-IJCNLP ’22,Online only,pp\. 308–318\.External Links:[Link](https://aclanthology.org/2022.findings-aacl.29/),[Document](https://dx.doi.org/10.18653/v1/2022.findings-aacl.29)Cited by:[§2\.1](https://arxiv.org/html/2610.26951#S2.SS1.p1.1)\.
- Yenet al\.\(2025\)H\. Yen, T\. Gao, M\. Hou, K\. Ding, D\. Fleischer, P\. Izsak, M\. Wasserblat, and D\. ChenHELMET: how to evaluate long\-context models effectively and thoroughly\.InProceedings of the 13th International Conference on Learning Representations,ICLR ’25\.External Links:[Link](https://openreview.net/forum?id=293V3bJbmE)Cited by:[§2\.2](https://arxiv.org/html/2610.26951#S2.SS2.p1.1)\.
- Zhanget al\.\(2024\)X\. Zhang, Y\. Chen, S\. Hu, Z\. Xu, J\. Chen, M\. Hao, X\. Han, Z\. Thai, S\. Wang, Z\. Liu, and M\. Sun∞\\inftyBench: extending long context evaluation beyond 100k tokens\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),ACL ’24,Bangkok, Thailand,pp\. 15262–15277\.External Links:[Link](https://aclanthology.org/2024.acl-long.814/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.814)Cited by:[§2\.2](https://arxiv.org/html/2610.26951#S2.SS2.p1.1)\.
- Zhuanget al\.\(2025\)Y\. Zhuang, L\. Hu, L\. Yun, S\. Kundu, Z\. Liu, E\. P\. Xing, and H\. ZhangScaling long context training data by long\-distance referrals\.InProceedings of the 13th International Conference on Learning Representations,ICLR ’25\.External Links:[Link](https://openreview.net/forum?id=tePFpDgyqg)Cited by:[§2\.1](https://arxiv.org/html/2610.26951#S2.SS1.p1.1)\.

## Appendix APrompt Templates

The haystack is line\-numbered \(\[1\],\[2\], …,\[N\]\) and substituted into thehaystackplaceholder of each task template\. We use four prompt variants: an*answer\-only*variant \(no system prompt\), a*Chain\-of\-Thought \(CoT\)*variant \(no system prompt\), an*evidence\-based generation \(EBG\)*variant that asks the model to cite supporting line numbers in JSON, and a*CoT\-based EBG*variant \(no system prompt\) that requests both brief reasoning and the EBG JSON answer\-and\-evidence output\.

Answer\-only Prompt \(User Message\)You will answer a question based on the following book snippet: \{haystack\} Use the information provided in the book snippet to answer the question\. Your answer should be short and based on either explicitly stated facts or strong, logical inferences\. Question: \{question\} Return only the final answer with no additional explanation or reasoning\.

CoT Prompt \(User Message\)You will answer a question based on the following book snippet: \{haystack\} Use the information provided in the book snippet to answer the question\. Be aware that some details may not be stated directly, and you may need to INFER the answer based on the given information\. Begin with a brief explanation of your reasoning in NO MORE THAN THREE \(3\) sentences\. Then, return the final answer on a new line\. Question: \{question\}

EBG System PromptYour job is to answer the question entirely from the context and provide a reference\. Your answer should cite all lines the answer is based on\.

EBG User Prompt<context\>\{haystack\}</context\> Answer the question based on information only from the context\. If the question is not answerable from the context, answer NA\. Your response should comprise only the answer and all lines the answer is based on in json format\. For example: \`\`\`json \{ ‘‘lines’’: \[25, 412\], ‘‘answer’’: ‘‘John’’ \} \`\`\` Question: \{question\}

CoT\-based EBG Prompt \(User Message\)You will answer a question based on the following book snippet: <context\>\{haystack\}</context\> Use the information provided in the book snippet to answer the question\. Be aware that some details may not be stated directly, and you may need to INFER the answer based on the given information\. Begin with a brief explanation of your reasoning in NO MORE THAN THREE \(3\) sentences\. Then, answer the question based on information only from the context\. If the question is not answerable from the context, answer NA\. Provide the answer \(just the character name\) and all lines the answer is based on in json format\. For example: \`\`\`json \{ ‘‘lines’’: \[25, 412\], ‘‘answer’’: ‘‘John’’ \} \`\`\` Question: \{question\}

## Appendix BAdditional Details of Synthetic Dataset

We provide additional details on the construction and structure of our synthetic dataset and present a representative example to illustrate the resulting task format and evidence organization\.

### B\.1Example

Consider facts“Himalaya Inc\. launched FitBand”and“Himalaya Inc\. is based in Mumbai”, with question“Which smart wearable was launched by a company based in Mumbai?”AtL=10​KL\{=\}10\\text\{K\}the two facts land at lines 56 and 67 among random distractors:

Haystack excerpt and question \(synthetic dataset\)… 54: Tech research group DataSynergy optimized machine learning algorithms\. 55: A renowned sculptor in Rome crafted a public monument\. 56: Himalaya Inc\. launched a new smart wearable called FitBand\. 57: A notable charity, WarmHearts, distributed winter supplies to families in need\. 58: GreenWave unveiled a new system for harnessing tidal energy\. … 65: Fitness startup VitalMotion developed smart workout equipment\. 66: GlobalVoyager published a feature on little\-known cultural festivals\. 67: Himalaya Inc\. is based in Mumbai\. 68: Director James Park released a documentary on endangered rainforests\. 69: Education reformist Tara Singh promoted progressive teaching methods\. … Question: Which smart wearable was launched by a company based in Mumbai?

Neither line alone is sufficient: line 56 identifies the product but not the company location, while line 67 identifies the location but not the product\. The model must therefore compose evidence across both lines\. Under the EBG prompt template, the expected output is a JSON object containing both the answer and the cited line numbers:

Expected EBG response\{ ‘‘lines’’: \[56, 67\], ‘‘answer’’: ‘‘FitBand’’ \}

Table 2:Needle templatesused in the evaluation\. Original NoLiMa needles under thesingle\-needle\(SN\) setting evaluate both one\-hop and two\-hop reasoning\. We add0405\-R, a gender\-neutral rephrased version of the0405primary needle template under SN, to examine the influence of gendered wording in the needle template\. Explicit secondary needles under thetwo\-needle\(TN\) setting directly provide the required world\-knowledge fact and evaluate two\-hop reasoning under fully contextualized evidence\. Thecontradictory two\-needle\(CTN\) setting instead introduces secondary needles that conflict with world knowledge, enabling evaluation of whether models remain strictly faithful to the provided context during two\-hop reasoning\.Such a response is scored as correct onAns,Evid, andAns\+Evid\. The remaining nine needles follow the same structure with varied entities, relations, and answer forms\.

## Appendix CNoLiMa Datatset Adaptation Details

### C\.1Needle settings

Table[2](https://arxiv.org/html/2610.26951#A2.T2)presents the different needle templates used across different settings\.

### C\.2Single Needle \(SN\) setting

In this setting,0402and0402Invencode the samedietaryfact about a character \(\{CHAR\}\) using different surface forms with altered word order\. We refer to the first variant as the*default*form and the second as the*inverse*form\. Similarly, needles0405and0405Invstate that a character \(\{CHAR\}\) isvegan\. These needles support bothone\-hopquestions answerable directly from the needle \(e\.g\., “Which character is lactose intolerant?”\) andtwo\-hopquestions requiring an additional world\-knowledge inference \(e\.g\., “Which character cannot drink a cappuccino?”, which requires knowing thatcappuccinocontainsmilk\)\. The SN context supplies the entity\-specific primary needle but not the complete inference chain required by the question\. For example, a needle stating that a character is vegan must be combined with parametric commonsense knowledge that veganism excludes eating fish\.

We therefore refer to the cited primary needle in SN as*anchor evidence*\. For SN, evidence accuracy measures whether the model cites the correct contextual anchor, and joint accuracy measures answer correctness together with anchor citation; these metrics should not be interpreted as establishing a complete context\-only evidence chain\. This distinction is specific to SN because part of the information required to answer the query may reside in the model’s parametric knowledge\. Accordingly, SN citations cannot directly attribute the contribution of such internally stored information\.

### C\.3Two\-needle \(TN\) setting

As shown in Table[2](https://arxiv.org/html/2610.26951#A2.T2), we introduce two explicit secondary needles under the two\-needle \(TN\) setting:TN\-0402\-SandTN\-0405\-S\. These needles explicitly provide the world\-knowledge fact required for two\-hop reasoning \(e\.g\.,“CHAR said, ‘Cappuccino is a classic Italian coffee made with espresso and steamed milk\.’,”\)\. Each secondary needle pairs with the corresponding primary needles\. For example,TN\-0402\-Spairs with0402and0402Inv, whileTN\-0405\-Spairs with0405and0405Inv\. Answering a two\-hop question, therefore, requires retrieving and jointly reasoning over both the primary and secondary needles\.

### C\.4Contradictory Two\-needle \(CTN\) setting

This setting introduces secondary needles that conflict with world knowledge\. For example,CTN\-0402\-Sclaims thatblack coffeecontainssteamed milk, whileCTN\-0405\-Sclaims thatfalafelcontainsminced lamb\. As inTN, each secondary needle pairs with its primary needles \(e\.g\.,CTN\-0402\-Swith0402and0402Inv\), testing whether models follow provided context or override it with parametric knowledge\. For both question types, the primary needle provides an entity\-specific anchor while inference requires unstated bridge knowledge\. One\-hop questions require a commonsense bridge \(e\.g\., veganism implies avoiding fish\), while two\-hop questions require an item\-composition bridge \(e\.g\., Brandade is fish\-based\)\.

## Appendix DLong\-context EBG vs\. RAG on MultiHop\-RAG

This appendix analyzes an*oracle\-containing long\-context \(LC\)*reading condition and an*open\-corpus retrieval\-augmented generation \(RAG\)*condition on theMultiHop\-RAGbenchmark\([Tang and Yang, 2024](https://arxiv.org/html/2610.26951#bib.bib16)\)\. The LC condition guarantees that every gold\-evidence article is available to the reader together with 20 distractor articles, whereas the RAG condition must retrieve the required evidence from the full 609\-article corpus\. The purpose of this comparison is therefore to diagnose the contribution of open\-corpus retrieval failures to end\-to\-end performance; it is not a pool\-matched comparison of LC and RAG under identical candidate document sets\. The motivation is direct: our synthetic two\-hop benchmark and the NoLiMa\-adapted benchmark are constructed haystacks, which leaves open the question of whether the trends we observe carry over to realistic, web\-derived multi\-document QA whereRAGis the natural baseline\.

The filteredMultiHop\-RAGsubset \(§[4\.3](https://arxiv.org/html/2610.26951#S4.SS3)\) of 2,255 queries tests bothEBG\(line\-level evidence\) andRAG\(chunked corpus\)\. We run experiments on a 50\-query pilot set, sampled with seed 0 and stratified across three types:ninf=13n\_\{\\mathrm\{inf\}\}\{=\}13,ncomp=21n\_\{\\mathrm\{comp\}\}\{=\}21, andntemp=16n\_\{\\mathrm\{temp\}\}\{=\}16\. We use the same queries, prompts, and scoring throughout, while LC and RAG differ in candidate\-document pools as described below\.

##### LC haystacks\.

For each query, we assemble a mini\-haystack of \(i\) all gold\-evidence documents and \(ii\) 20 distractor documents sampled without replacement from the remaining corpus\. Documents are randomly concatenated, split into sentence\-level lines, and rendered in thelineno: textformat used by our synthetic benchmark\. Gold evidence facts are re\-mapped to line numbers so EBG labels match the prompt\. Mini\-haystacks average∼241\\sim 241K characters \(∼58\\sim 58K tokens\), within every model’s context window\. This is the*LC \(full\)*condition\.

##### RAG haystacks\.

For each query, we construct RAG contexts atK∈4,10,25,50,100,200K\\in\{4,10,25,50,100,200\}\. The retriever searches the full 609\-document corpus, divided into 256\-token sliding windows \(6,296 chunks\)\. We embed chunks and queries withtext\-embedding\-3\-large, rank by cosine similarity, and use the top\-KKchunks as the RAG context\. Each chunk is prefixed with\[CHUNK <rank\> \| <title\>\], and rendered with the same sentence\-per\-line numbering as LC\. Gold evidence facts are re\-mapped to lines in the retrieved context; unretrieved facts have no corresponding gold line\.

The conditions are not pool\-matched: LC has all gold source articles plus 20 distractors, whereas RAG must locate the required evidence across all 609 articles\. This design deliberately preserves the native evidence\-access setting of each approach in practice, enabling a realistic comparison between them under their respective retrieval assumptions\.

### D\.1Setup

##### Prompt modes\.

Each \(query, context\) pair is evaluated under two prompts differing in one sentence:

- •*Answer\-only \(Ans∗\\texttt\{Ans\}^\{\*\}\)\.*The model returns a JSON object with only ananswerfield\. We reportAns∗\\texttt\{Ans\}^\{\*\}accuracy\.
- •*EBG\.*The model returns a JSON object withanswerandlinesfields\. Scoring usesAns\(string match against the gold answer\),Evid,\(strict\)\(every gold line appears in the prediction\), andJoint≡\\equivAns\+Evid,\(strict\)\(answer\-and\-evidence\)\.

Both prompts share the same system message, instructions, and JSON example template, modulo the citation field\. The system message and prompt templates are reproduced verbatim in our accompanying public code release for full reproducibility\.

##### Models\.

We evaluate three frontier LLMs whose context windows comfortably fit under theLCcondition:GPT\-5,Sonnet\-4\.5, andGemini\-3\-Fl\. All calls use temperature 0 \(or provider equivalent\), default sampling parameters, andparallel=5\. This setup keeps decoding conditions consistent across models and experiments\.

##### Scoring\.

Ansmatches the model’s answer string against the gold answer using the same normalization used in the synthetic and NoLiMa experiments\.Evid \(strict\)requires the predicted line\-id set to contain*every*gold line for that query; partial matches are scored separately asEvid \(recall\)but onlyEvid \(strict\)is used forJoint\. API failures \(timeouts, safety refusals, transient connection errors\) are excluded from the denominator; in practice this affected at most one cell \(Sonnet\-4\.5,RAG@200EBG,n=49/50n\{=\}49/50\)\. Thus, reported scores reflect successfully completed model calls only\.

### D\.2Experiment A: Long\-context EBG onMultiHop\-RAG

We run the LC condition to establish base rates and test RQ2 \(*Does requiring explicitly citing evidence improve answer accuracy?*\) on a non\-synthetic benchmark\. Each model sees the full mini\-haystack \(∼241\\sim 241K characters\) and is prompted inanswer\-onlyorEBGmode\. Table[3](https://arxiv.org/html/2610.26951#A4.T3)summarizes results\.

Table 3:Ans∗\(answer\-only\) vs\.Ans\(EBG\)on the 50 pilot queries, full mini\-haystack \(∼241\\sim 241K characters, 20 distractors\)\. Requiring citations raises answer accuracy for two of the three models, with a striking\+14\+14pp improvement forSonnet\-4\.5\. This replicates the citation\-as\-scaffold effect that was invisible in our \(saturated\) synthetic benchmark\. Higher \(↑\\uparrow\) values are better forAns∗andAns\.Joint accuracy is much harder than*Ans*: under EBG, joint \(Ans\+Evid\\textit\{Ans\}\{\+\}\\textit\{Evid\}\) is only 28% \(GPT\-5\), 8% \(Sonnet\-4\.5\), and 28% \(Gemini\-3\-Flash\)\. The gap*Ans*−\-*Joint*ranges from 50 pp \(Gemini\) to 74 pp \(Sonnet\) and is concentrated on inference queries, mirroring the*asymmetric retrieval*finding from our synthetic two\-needle setting: models routinely produce a correct answer while citing only one of the two required hops\.

### D\.3Experiment B: Retrieval analysis

Before running any LLM we measure how hard the retrieval problem itself is\. For every query and every gold fact we record the rank of the first chunk that contains that fact;*Hits@kk\(all gold\)*counts the fraction of queries for which*every*gold fact appears in the top\-kk\. We compare two embedding models \(text\-embedding\-3\-small/large\) and two chunkers \(256\-token sliding windows vs\. sentence\-level chunks\); full results are in Table[4](https://arxiv.org/html/2610.26951#A4.T4)\(corpus\-wide,N=2255N\{=\}2255\), with our chosen configuration in bold\.

Table 4:Corpus\-wide retrievalmetrics onMultiHop\-RAG\(N=2255N\{=\}2255\);*any gold*reports the fraction of queries for which at least one annotated gold fact appears in the top\-kkretrieved set;*all gold*reports the fraction for which every annotated gold fact appears\. The latter measures strict annotated\-gold coverage, not an upper bound on answer accuracy\. Under the stronger assumptions that the gold annotations are exhaustive and that every annotated fact is necessary,*all gold*upper\-bounds on what any faithful RAG pipeline can achieve at top\-kk: if even one of the 2–4 gold facts is missing from the retrieved set, downstream answers cannot be faithful\. The token\-windowedlargeconfiguration dominates and is used for all RAG experiments below\. Higher \(↑\\uparrow\) values are better\.The key qualitative finding is that retrieving*at least one*gold fact is easy \(Hits@10 \(any\)=0\.82=0\.82for our chosen config\), but retrieving*all*gold facts is hard \(Hits@10 \(all\)=0\.19=0\.19\)\. Inference \(multi\-hop\) queries are the most punishing: Hits@10 \(all\) is only0\.080\.08for that slice\. For RAG to faithfully ground a multi\-hop answer atk=10k\{=\}10, more than 90% of inference queries are doomed before the LLM is even called\. This establishes a concrete retrieval ceiling on downstream RAG performance\. Crucially, the bottleneck is not finding relevant evidence, but retrieving all pieces of evidence needed for a complete answer\. Thus, many apparent reasoning failures downstream are already determined by missing context at retrieval time\.

### D\.4Experiment B′: Needle coverage on the pilot

For the 50 pilot queries, we repeat this analysis at every evaluatedKK\. Table[5](https://arxiv.org/html/2610.26951#A4.T5)reports the fraction of queries with all gold facts \(“needles”\) retrieved in the top\-KKand the fraction of individual gold facts appearing anywhere in the top\-KK\. These quantities upper\-bound faithful joint accuracy and answer recall, respectively, for RAG on the pilot set\.

Table 5:Needle coverageon the 50 pilot queries \(retriever:text\-embedding\-3\-large\+ 256\-token chunks\)\. At realistic RAG budgets \(K≤25K\{\\leq\}25\) fewer than half of queries have all gold facts retrieved; only atK=200K\{=\}200does coverage approach saturation\. Higher \(↑\\uparrow\) values are better\.
### D\.5Experiment C: Oracle\-containing LC versus open\-corpus RAG

The central experiment sweeps open\-corpus RAG over six values ofKKand compares it with oracle\-containing LC under both prompt modes\. All conditions use the same 50 queries, prompts, models, and scoring code, but differ in candidate\-document pools: LC receives every gold\-evidence article plus 20 distractors, whereas RAG searches the 609\-article corpus\. Table[6](https://arxiv.org/html/2610.26951#A4.T6)thus measures the end\-to\-end cost of open\-corpus retrieval relative to gold\-source access, not a pool\-matched comparison\. The table tells three stories, one per metric\.

Table 6:Oracle\-containing LC versus open\-corpus RAG on the 50\-queryMultiHop\-RAGpilot\.LC receives every gold\-evidence article plus 20 distractor articles; RAG retrieves from the full 609\-article corpus\. The conditions use the same queries, models, prompt templates, and scoring procedure but are not pool\-matched\. All values are percentages overn=50n\{=\}50\(Sonnet\-4\.5RAG@200 EBG:n=49n\{=\}49\)\. Boldface marks the per\-row maximum within the corresponding prompt\-mode block\. Higher \(↑\\uparrow\) values are better\.##### Answer accuracy: oracle\-containing LC exceeds open\-corpus RAG under the evaluated budgets\.

Across both prompts and all three models, oracle\-containing LC outperforms open\-corpus RAG forK≤100K\\leq 100\. AtK=200K\{=\}200, with 96% gold\-fact recall and 90% all\-needles coverage, RAG ties LC forSonnet\-4\.5\(68%68\\%\) but remains 4,pp below forGPT\-5\(78%78\\%vs\.82%82\\%\) and 10,pp below forGemini\-3\-Flash\(70%70\\%vs\.80%80\\%\)\. AtK=10K\{=\}10, the LC–RAG gap is2222–4444,pp under answer\-only prompting and2626–3434,pp under EBG\. These gaps include full\-corpus retrieval and thus compare open\-corpus RAG with evidence\-guaranteed LC, not an intrinsic LC advantage under identical candidate pools\.

##### Strict evidence: small\-KKRAG and LC are comparable\.

*Evid \(strict\)*is roughly flat acrossKKfor each model \(10–40% absolute\), with no clean monotone trend\. This reflects two distinct requirements for evidence correctness:*all needles must be retrieved*\(coverage, Table[5](https://arxiv.org/html/2610.26951#A4.T5)\) and the model must*cite all of them correctly*\(citation behaviour\)\. SmallKKfavors the latter because the citation surface is limited, whereas largeKKfavors the former as retrieval coverage improves\. IncreasingKKtherefore trades easier citation selection for greater evidence availability, and these opposing effects roughly cancel\. Consequently, small\-KKRAG can remain comparable to LC on strict evidence despite substantially lower retrieval coverage\.

##### Joint \(faithful\) accuracy: mixed\.

Joint=Ans∧\\wedgeEvid\(also written asAns\+Evid\) is the most diagnostic number\. Two trends are visible\. First, joint*decreases*withKKfor every model: at largeKKthe citation surface grows, the model loses track of which retrieved chunk it used, and joint collapses despite the higher answer recall\. Second, the best RAG joint is always at smallKK\(K=4K\{=\}4for all three models\), where the citation problem is trivial and the only queries that score are the ones whose needles happened to be in the top 4 \(14% of queries\)\. Aggregate joint therefore over\-rewards a regime that only works on the easy fraction of queries\. LC’s joint sits at 28%, 8%, 28% \- competitive with RAG@4 for GPT\-5 and Gemini, behind for Sonnet \- and is computed over the full pilot set rather than the small subset where retrieval succeeded\.

### D\.6Discussion

We highlight four takeaways for the manuscript\.

##### Takeaway 1: LC EBG dominates RAG for answer accuracy, at every realistic retrieval budget\.

Under the answer\-only prompt, the oracle\-containing LC condition exceeds open\-corpus RAG@10 by2222–4444pp and exceeds RAG@200 by00–1010pp\. The same qualitative pattern holds under EBG\. Because the two conditions do not share a candidate\-document pool, these numbers quantify the combined cost of document discovery and passage retrieval relative to evidence\-guaranteed reading; they are not a pool\-matched estimate of an architecture\-level LC–RAG effect\.

##### Takeaway 2: RAG’s retrieval bottleneck is multi\-hop coverage, not single\-hop recall\.

Hits@10 \(any gold\) is 82% but Hits@10 \(all gold\) is only 19% \(Table[4](https://arxiv.org/html/2610.26951#A4.T4)\); on inference queries the all\-gold number falls to 8%\. The principal source of the observed gap relative to the oracle\-containing LC control is that, at smallKK, open\-corpus RAG rarely retrieves the complete union of annotated gold facts\.

##### Takeaway 3: Citations as a reasoning scaffold replicate outside the synthetic setting\.

*Ans*accuracy under EBG is higher than under the answer\-only prompt for two of three models, with a striking\+14\+14pp improvement for Sonnet\-4\.5 \(Table[3](https://arxiv.org/html/2610.26951#A4.T3)\)\. This is the RQ1 effect that was masked on our synthetic benchmark by ceiling performance\.

##### Takeaway 4: The grounding gap is a real\-data phenomenon\.

The*Ans*−\-*Joint*gap in the LC condition is 50–74 pp depending on model, and is largest on inference queries\. Models routinely produce a correct multi\-hop answer while citing only one of the two required hops \- the same asymmetric\-retrieval failure mode we documented on the synthetic benchmark, here observed on naturalistic web data\.

### D\.7Generalization to open\-weight models

The preceding comparison uses three proprietary frontier LLMs\. To test whether the observed pattern generalizes across model families, we repeat the same two evidence\-access regimes, an oracle\-containing LC condition and open\-corpus RAG, with three open\-weight models\. To test whether the LC\-EBG vs\. RAG ordering is a property of the task rather than of a particular model family, we re\-run the same conditions, using same mini\-haystacks, same RAG contexts, same prompts and scoring code, with three open\-weight models served through Fireworks\.ai:GPT\-OSS\-120B,GLM\-5\.2, andNemotron\-3\.5\-Lightning\-30B\-A3B\(or Nemo\-3\.5\-L in short\)\. To obtain tighter estimates we scale this comparison to a larger stratified sample ofn=200n\{=\}200queries\. We evaluate the two most informative RAG budgets, a realistic operating point \(K=10K\{=\}10\) and the near\-saturation budget \(K=200K\{=\}200, 90% all\-needles coverage; Table[5](https://arxiv.org/html/2610.26951#A4.T5)\), against the full LC condition\. Table[7](https://arxiv.org/html/2610.26951#A4.T7)reports the results\.

Table 7:LC\-EBG vs\. RAG\-EBG for open\-weight modelson a200200\-query stratified sample\. All numbers are percentages overn=200n\{=\}200\(a handful of cells returnn=198n\{=\}198–199199where 1–2 queries exceed a provider’s context limit\)\. Boldface marks the per\-row maximum within each mode block\. Higher \(↑\\uparrow\) values are better\. Atn=200n\{=\}200the95%95\\%bootstrap CI half\-width onAns/Jointis≈\\approx66–77pp\. LC is the oracle\-containing gold\-article\-plus\-20\-distractor condition; RAG@10 and RAG@200 use open\-corpus retrieval over all 609 articles\.The qualitative picture from Table[6](https://arxiv.org/html/2610.26951#A4.T6)carries over\.\(i\) Answer accuracy: oracle\-containing LC exceeds the evaluated open\-corpus RAG setup\.At the realisticK=10K\{=\}10operating point, LC’s*Ans*exceeds RAG@10 for all three models under both prompts: by\+28\+28to\+35\+35pp under the answer\-only prompt and by\+13\+13\(GPT\-OSS\-120B\),\+36\+36\(GLM\-5\.2\), and\+20\+20pp \(Nemotron\) under EBG \- the same\+26\+26to\+34\+34pp EBG gap we observe for the frontier models\. At the near\-saturation budgetK=200K\{=\}200RAG closes some of the gap but overtakes LC for none of the three readers: LC*Ans*still leads by\+2\+2\(GPT\-OSS\-120B\),\+14\.5\+14\.5\(GLM\-5\.2\), and\+19\.7\+19\.7pp \(Nemotron\), with the GPT\-OSS\-120B margin inside the bootstrap CI\. The larger sample thus removes the onen=50n\{=\}50anomaly \(GPT\-OSS\-120B RAG@200 nominally above LC\) and confirms LC≥\\,\\geq\\,RAG@200 throughout\.\(ii\) The grounding gap is model\-agnostic\.Under LC, the*Ans*−\-*Joint*gap is5252\(GPT\-OSS\-120B\),58\.558\.5\(GLM\-5\.2\), and42\.742\.7pp \(Nemotron\) \- the same 40–60 pp range as the frontier models\. GPT\-OSS\-120B is the extreme case: it answers54%54\\%of queries correctly but achieves only2%2\\%joint accuracy, essentially never citing both required hops\.\(iii\) Citation\-as\-scaffold is uneven for weaker readers\.The*Ans*∗→\{\}^\{\*\}\\\!\\to*Ans*\(EBG\) change on LC is\+0\.5\+0\.5pp for GLM\-5\.2 but−21\-21pp for GPT\-OSS\-120B and−27\-27pp for Nemotron: the weakest open readers, unlike the frontier models, struggle to answer and cite simultaneously, so the citation requirement can hurt rather than help\. Overall, the pattern that oracle\-containing LC achieves higher answer accuracy than the evaluated open\-corpus RAG pipeline at practical retrieval budgets, while retaining a substantial grounding gap, generalizes across proprietary and open\-weight readers\. This result continues to include the cost of full\-corpus retrieval and is not a pool\-matched comparison\.

### D\.8Limitations and next steps

The frontier\-model experiments use a 50\-query pilot, so the reported estimates remain subject to sampling variation, particularly for small question\-type slices such as the 13 inference queries\. Evaluating all 2,255 usable queries would provide tighter confidence intervals\. We also hold the retriever fixed totext\-embedding\-3\-largewith 256\-token sliding windows\. Reranking, hybrid sparse–dense retrieval, query expansion, longer chunks, and document\-aware indexing could improve the open\-corpus RAG results\. For the evaluated retriever, corpus\-wide all\-gold Hits@1010is19%19\\%over the 2,255 usable queries; on the 50\-query pilot, all\-gold coverage atK=10K\{=\}10is28%28\\%\. Under the assumptions that the gold annotations are exhaustive and that every annotated fact is necessary, these coverage rates strongly constrain faithfully grounded performance at this retrieval budget\.

A key design limitation is that theMultiHop\-RAGLC and RAG conditions are not pool\-matched\. The LC condition is an*oracle\-containing reading control*: every gold\-evidence article is guaranteed to be present together with 20 distractor articles\. The RAG condition instead performs*open\-corpus retrieval*over the full 609\-article corpus\. Consequently, the reported LC–RAG gaps combine the effects of full\-corpus document discovery, passage retrieval, and downstream reading; they should not be interpreted as an intrinsic ranking of LC and RAG under matched evidence access\.

A complementary pool\-matched RAG condition that retrieves from the same per\-query gold\-plus\-20 candidate set would separate within\-pool chunking and ranking effects from the cost of full\-corpus document discovery\. We leave this controlled comparison, together with evaluation of stronger retrieval pipelines and the full query set, to future work\.

### D\.9Reproducibility

All code, queries, and intermediate artifacts live underbaselines/multihop\_rag/in the experiment repository:

- •build\_haystack\.py: builds per\-query mini\-haystacks \(LC condition\)\.
- •embed\_and\_rank\.py: chunks the corpus, embeds, ranks gold facts; producessummary\.json,fact\_ranks\.jsonl, andchunks\.jsonlunderretrieval\_results/<config\>/\.
- •build\_rag\_haystack\.py: assembles per\-KKRAG contexts from the retrieval artifacts\.
- •run\_baseline\.py: runs the LLM evaluation on either LC or RAG haystacks, supports\-\-answer\-only\.
- •aggregate\_pilot\.py,aggregate\_rag\_sweep\.py: produce the tables above\.

The 50\-query pilot is reproducible with\-\-seed 0onprocessed/queries\_d20\.jsonl\(built from the officialMultiHop\-RAGHuggingFace release of the corpus and queries\)\.

## Appendix ELong\-context EBG vs\. RAG on QASPER\-MP

This appendix documents a second comparison of long\-context evidence\-based generation \(LC EBG\) against retrieval\-augmented generation \(RAG\), this time on a multi\-paper adaptation of QASPER\([Dasigi et al\., 2021](https://arxiv.org/html/2610.26951#bib.bib17)\)that we callQASPER\-MP\. WhereasMultiHop\-RAG\(Appendix[D](https://arxiv.org/html/2610.26951#A4)\) tests multi\-document reasoning on news,QASPER\-MPtests it on scientific papers, where the gold question–answer–evidence triples were written assuming a single known paper\. To make the task non\-trivial under retrieval, we \(i\) place each gold paper alongside ten distractor papers, \(ii\) rewrite the question so it identifies the target paper by a realistic short reference \(an author name or a 2–3 word topic cue\), and \(iii\) evaluate in both LC and RAG conditions over the same 50\-query stratified pilot\.

The headline finding is that the aggregate score \(Answer\-F1 or LLM\-judge, averaged over all questions\) makes LC and RAG look roughly tied, but this tie hides an answerability asymmetry:*LC EBG beats RAG by 5–22 judge points on answerable questions for every frontier model tested*, while RAG “wins” on unanswerable questions because it over\-refuses\. The over\-refusal mechanically matches the QASPERUnanswerablegold label on the 24% of pilot questions where that is the correct answer, pulling the aggregate up\.

### E\.1Setup

##### Dataset\.

We use the QASPER dev split \(N=1,005N=1\{,\}005questions over 281 papers\)\. Each question is annotated with an*answer type*\(extractive,abstractive,boolean, ornone\- meaning unanswerable from the paper\) and an*evidence set*: the list of paragraphs from the paper that support the answer\. Up to three annotators contribute per question; we follow the official QASPER evaluator and take the max F1 across annotators\.

##### 50\-query pilot\.

We sample 50 questions, stratified by answer type, so that the composition matches the dev\-split distribution: 13 boolean, 13 abstractive, 12 extractive, 12 unanswerable\. This split \(38 answerable, 12 unanswerable\) is the denominator for every condition below\.

##### Multi\-paper haystacks \(LC condition\)\.

For each pilot query we build an 11\-paper haystack consisting of the gold paper plus 10 distractor papers drawn uniformly from the dev split\. Each paper is rendered with explicit document boundaries:

> `<paper\>` `<title\>`Paper title`</title\>` `<abstract\>`Paper abstract`</abstract\>` `<section\>`Section name`</section\>` Paragraph 1 of that section Paragraph 2 of that section … `</paper\>`

Lines are concatenated and numbered`i: <line\>`so the model can cite specific lines as evidence \(consistent with our EBG protocol\)\. The median LC haystack size is∼\\sim32K characters; the largest is∼\\sim56K\. The gold\-evidence paragraph\(s\) for each query are re\-mapped to line indices in the assembled haystack\.

##### RAG condition\.

We embed every QASPER dev paragraph and every \(disambiguated\) query with OpenAItext\-embedding\-3\-large, and retrieve the top\-KKparagraphs by cosine similarity\. We evaluate four retrieval variants, factored as\{flat,struct\}×\{pool11,full\}\\\{\\text\{flat\},\\,\\text\{struct\}\\\}\\times\\\{\\text\{pool11\},\\,\\text\{full\}\\\}:

flat\_pool11\.Top\-KKparagraphs retrieved from the same 11\-paper pool as the LC haystack \(pool\-matched candidate corpus\)\. The reader sees a flat list of retrieved paragraphs with a one\-line`\[CHUNK <rank\> \| <title\>\]`marker before each\.

struct\_pool11\.Same retrieval, but paragraphs are regrouped by their source paper and wrapped in the same XML layout used for LC\. Within a paper, paragraphs are sorted by their original index, and discontinuities \(gaps where intermediate paragraphs were not retrieved\) are marked with a single\.\.\.line\.

flat\_full / struct\_full\.Same as above but retrieving from the full dev corpus \(N≈6,000N\\approx 6\{,\}000paragraphs\) rather than the 11\-paper pool\. This is the*realistic*open\-corpus RAG setting: the retriever must locate the target paper among hundreds of similar NLP papers using only the short cue in the rewritten question\.

##### Models\.

We evaluate five frontier reasoning models:GPT\-5,GPT\-5\.5,Claude Opus 4\.6,Claude Sonnet 4\.5, andGemini 3 Flash\(preview\)\. All runs use the same query set, the same retrieval indices, and the same EBG prompt template\. Citation rules in the prompt forbid citing structural lines \(`<title\>`,`<abstract\>`,`<paper\>`,\[CHUNK, or`\.\.\.`\); these lines are present only as context\.

##### Question disambiguation\.

QASPER questions were written to be unambiguous given the paper; many become ambiguous in a multi\-paper haystack \(e\.g\., “What evaluation metrics were used?” becomes pathological with 11 candidate papers\)\. We disambiguate each pilot question by inserting a short*paper cue*, chosen deterministically 50/50 from two cue types:

- •Author cue\(n=24n=24\):*“In Smith et al\., …”*, with surname\(s\) drawn from the arXiv metadata of the gold paper\. Conventions: 1 author→\\to*“Smith”*, 2 authors→\\to*“Smith and Doe”*,≥3\\geq 3authors→\\to*“Smith et al\.”*\.
- •Title cue\(n=26n=26\):*“In the⟨\\langlecue⟩\\ranglepaper, …”*, where the cue is a verbatim 2–3 word substring of the paper title \(e\.g\.,*“Vietnamese Word Segmentation”*,*“Winograd Schemas”*,*“TF\-IDF Features”*\)\.

Cues are intentionally short and realistic, matching what a researcher would type into a search box\. This makes the retrieval problem genuinely harder than indexing on full titles or descriptive paraphrases\.

### E\.2Aggregate Performance: LC≈\\approxRAG \(and why this is misleading\)

Table[8](https://arxiv.org/html/2610.26951#A5.T8)reports per\-condition Answer\-F1, Evidence\-F1, and LLM\-judge \(an automated 0/0\.5/1 grader using GPT\-5; see Appendix[E\.5](https://arxiv.org/html/2610.26951#A5.SS5)\)\. Aggregating over all 50 pilot questions:

- •LC andflat\_pool11@3are within±\\pm0\.06 judge points for every model\.
- •struct\_pool11@3beats LC for GPT\-5 and matches it for Gemini under the LLM judge\.

On its face, this is the standard “LC is comparable to RAG” result\.

Table 8:QASPER\-MP aggregate scores\(50 pilot questions, EBG mode\)\. Best condition per model inbold\. Judge scores use the GPT\-5 judge described in Appendix[E\.5](https://arxiv.org/html/2610.26951#A5.SS5);nnreports how many queries returned valid output\. Higher \(↑\\uparrow\) values are better\.†Gemini hit transient API errors on this condition; the remaining 25 responses are used\.

##### The aggregate hides an answerability asymmetry\.

Splitting the 50 queries into the 38 answerable and 12 unanswerable subsets tells a sharper story \(Table[9](https://arxiv.org/html/2610.26951#A5.T9)\)\.

Table 9:QASPER\-MP results split by answerability\(EBG mode, LLM\-judge\)\. “Refusal” is the fraction of model outputs that started withUnanswerable\. Best LC vs\. best RAG variant per model inboldwithin each subset\. Higher \(↑\\uparrow\) values are better\.Three observations:

1. 1\.LC EBG wins answerable questions cleanly\.For all five models, LC has the highest Judge score on the 38\-question answerable subset, with absolute gains over the best RAG variant of\+0\.05\+0\.05\(GPT\-5\),\+0\.13\+0\.13\(GPT\-5\.5\),\+0\.13\+0\.13\(Opus\-4\.6\),\+0\.08\+0\.08\(Sonnet\-4\.5\), and\+0\.12\+0\.12\(Gemini\-3\-Flash\)\. These gaps are robust to the citation\-rule and discontinuity\-marker changes described in Appendix[E\.4](https://arxiv.org/html/2610.26951#A5.SS4)\.
2. 2\.RAG over\-refuses, and the refusals pay off on the unanswerable subset\.flat\_pool11@3refuses 29–40% of answerable questions \(vs\. 16–18% for LC\) because when retrieval misses the gold paragraph the reader sees only distractor content and sensibly says “Unanswerable\.” Those same refusal\-prone retrieval failures happen to be the right answer on unanswerable queries \(where “Unanswerable”*is*gold\), so RAG wins that subset by 0\.17–0\.25 judge points\.
3. 3\.Realistic open\-corpus retrieval makes LC win by a margin the aggregate cannot hide\.Theflat\_full@10row in Table[8](https://arxiv.org/html/2610.26951#A5.T8)shows LC outperforming full\-corpus RAG by 5–13 judge points on the aggregate already; the answerable\-only gap is larger again \(not shown\)\. When the retriever must distinguish the gold paper from∼\\sim280 distractors using a 2–3 word cue, paper\-level retrieval coverage drops to 48% \(all\-needles@10\), and downstream answer quality follows\.

The aggregate score thus rewards RAG’s high refusal rate\. Once we condition on answerability, the LC\>\>RAG gap on*actually answerable*questions is consistent across models and retrieval variants\.

### E\.3Disambiguation Details and Cue Length Sensitivity

We initially experimented with longer “descriptive” cues generated by GPT\-5 from each paper’s title and abstract \(e\.g\.,*“In the cross\-lingual zero\-shot NMT paper”*\)\. These cues are unrealistically informative: they often include domain phrases that do not appear in the title at all, which makes retrieval implausibly easy\. Embedding\-based retrieval coverage with descriptive cues was 60\.0% \(all\-needles@10, full corpus\)\. Switching to the short, realistic cues described in Appendix[E\.1](https://arxiv.org/html/2610.26951#A5.SS1)dropped coverage to 48\.0% \- a 12\-point hit \- reflecting the fact that*“In Smith et al\.”*or*“In the BERTSUM paper”*is a much weaker retrieval signal than a paraphrased domain summary\.

We use the short, realistic cues throughout this appendix because they correspond to how researchers actually query systems in practice; the\+\+12pp coverage gain on descriptive cues would inflate RAG numbers relative to what users would experience\.

### E\.4Structured RAG, Citation Eligibility, and Discontinuity Markers

A first iteration of structured RAG produced a counterintuitive result:struct\_pool11@3*underperformed*flat\_pool11@3on Evidence\-F1 across models \(gaps of 0\.07–0\.12\), despite carrying*more*information than the flat baseline \(abstract per paper\)\. On inspection,∼\\sim44% of structured\-RAG citations pointed at`<title\>`or`<abstract\>`lines: when an abstract paraphrases the answer, the model cites it \- but QASPER gold evidence excludes abstracts, so abstract citations*cannot*match gold\.

We therefore made two changes to the structured layout and prompt used throughout the main results:

1. 1\.Citation eligibility\.The EBG prompt forbids citing structural lines \(`<title\>`,`<abstract\>`,`<paper\>`,`</paper\>`,\[CHUNK, and\.\.\.\)\. Models must cite at least one body paragraph line for any non\-refusal answer\. Titles and abstracts remain visible as context; they are simply not valid evidence\.
2. 2\.Discontinuity markers\.Within each paper, retrieved paragraphs are sorted by their original paragraph index, and any gap \(the abstract followed by paragraph 2, or paragraph 5 followed by paragraph 9\) is marked with a single`\.\.\.`line\. This gives the reader explicit signal that some material is missing between the visible paragraphs\.

Under this revised protocol \(Table[8](https://arxiv.org/html/2610.26951#A5.T8)\), structured RAG recovers: Evidence\-F1 jumps by\+\+0\.04 to\+\+0\.15 per model andstruct\_pool11@3beats or matchesflat\_pool11@3for two of five models\. The intuition that paper\-level structure is useful is correct \- it had been masked by a metric artifact\.

### E\.5LLM\-Judge Evaluation

To control for token\-F1 brittleness on paraphrases, we additionally score outputs with an LLM judge:

- •Judge model:GPT\-5\.
- •Inputs:the original question, all annotator gold answers, and the model prediction\.
- •Output:a single number in\{0,0\.5,1\}\\\{0,\\,0\.5,\\,1\\\}\. The judge is instructed to be lenient about formatting and elaboration but strict about content; partial credit \(0\.5\) is reserved for answers that capture some but not all required information\. “Unanswerable” vs\. “Unanswerable” scores 1\.0; “Unanswerable” vs\. a real answer \(or the reverse\) scores 0\.0\.

Judgments are cached by content hash so repeated scoring is free\. Across all conditions, Judge scores are systematically higher than Answer\-F1 \(the judge is lenient about phrasing\), but the*ordering*of conditions is preserved\. The LC vs\. RAG gap on answerable questions is larger under the judge than under Answer\-F1, because the judge does not penalize Sonnet\-4\.5 and Gemini\-3\-Flash for their characteristic boolean elaborations\.

### E\.6Generalization to open\-weight models

We repeat the QASPER\-MP comparison with three open\-weight models served through Fireworks\.ai \-GPT\-OSS\-120B,GLM\-5\.2, andNemotron\-3\.5\-Lightning\-30B\-A3B\- using the same retrieval indices, EBG prompt, and GPT\-5 judge \(Appendix[E\.5](https://arxiv.org/html/2610.26951#A5.SS5)\), scaled to a larger200200\-query stratified sample \(n=150n\{=\}150answerable,n=50n\{=\}50unanswerable\) for tighter estimates\. Table[10](https://arxiv.org/html/2610.26951#A5.T10)splits the results by answerability, exactly as Table[9](https://arxiv.org/html/2610.26951#A5.T9)does for the frontier models\.

Table 10:QASPER\-MP open\-weight results split by answerability\(EBG mode, GPT\-5 judge\), over a200200\-query stratified sample\. Answerable subset isn=150n\{=\}150and unanswerablen=50n\{=\}50; a handful of cells return 1–2 fewer queries where a provider error dropped a row \(†GLM\-5\.2flat\_full@10returnedn=148/49n\{=\}148/49\)\. Best LC vs\. RAG variant per model shown inboldwithin the answerable subset\. “Refuse” is the fraction of outputs starting withUnanswerable\. Higher \(↑\\uparrow\) values are better\.The answerability asymmetry we documented for frontier models reproduces on every open\-weight reader\.LC EBG wins the answerable subset cleanly: the LC judge score is highest for all three models, beating the best RAG variant by\+0\.16\+0\.16\(GPT\-OSS\-120B\),\+0\.24\+0\.24\(GLM\-5\.2\), and\+0\.07\+0\.07\(Nemotron\) \- gaps as large as, or larger than, the\+0\.05\+0\.05to\+0\.13\+0\.13we observe for the frontier models\.RAG over\-refuses for the same mechanistic reason:flat\_pool11@3refuses2121–33%33\\%of answerable questions \(vs\.77–12%12\\%for LC\), because a missed retrieval leaves the reader with only distractor content; those refusals then mechanically boost RAG on the unanswerable subset, where “Unanswerable” is gold\.Realistic full\-corpus retrieval widens the LC advantage:flat\_full@10answerable judge scores drop to0\.430\.43–0\.500\.50, trailing LC by0\.090\.09to0\.240\.24judge points, as the retriever must find the gold paper among∼\\sim280 distractors from a 2–3 word cue\. The open\-weight results thus corroborate the central QASPER\-MP finding: once conditioned on answerability, long\-context EBG beats every RAG variant, and the aggregate “tie” is an artifact of RAG’s refusal behavior\.

### E\.7Strengthening RAG: metadata\-in\-chunk retrieval

The full\-corpus RAG deficit above is partly a*retrieval*failure rather than solely a limitation of the downstream reader\. After v2 disambiguation, each query carries only a short, typically22–33word cue identifying the relevant paper\. In the baseline index, however, candidate chunks are embedded as raw paragraph text, without any explicit representation of document identity\. The retriever must therefore infer the connection between a sparse paper\-level cue in the query and a paragraph whose local content may never mention the paper title or otherwise reveal its source\. This mismatch is especially consequential in the full\-corpus setting, where evidence from the gold paper competes with paragraphs drawn from the entire∼\\sim281\-paper dev corpus\. Even when the query contains enough information to identify the intended document, the embedding space has no direct document\-level feature with which to exploit that signal, making it easy for the gold paper \- and consequently the relevant evidence within it \- to fall outside the retrieval budget\.

A natural way to give RAG a fairer chance is therefore to expose document identity directly to the retriever by injecting metadata into the indexed chunks\. We rebuild the retrieval index so that every paragraph is embedded together with its source paper’s title, using the representation “title\\n\\nparagraph”\. This modification creates an explicit lexical and semantic bridge between the short paper cue supplied by v2 disambiguation and every paragraph originating from that paper, while preserving the paragraph text needed to rank evidence within the document\. We then re\-run the hardest retrieval condition,flat\_full@10, atn=200n\{=\}200for all five readers\. Importantly, this intervention is confined to retrieval: generation\-side rendering, prompting, context construction, and reader settings are unchanged\. Thus, any downstream improvement can be attributed to changing the representation used for ranking rather than to providing the reader with additional metadata or altering the generation procedure\.

Metadata\-in\-chunk produces a large improvement in evidence retrieval\. On the200200\-query set, the fraction of queries for which the*full*gold\-evidence set is recovered within the top\-1010increases from43\.5%43\.5\\%to61\.0%61\.0\\%, a gain of17\.517\.5percentage points\. Gold\-paragraph recall rises even more substantially in relative terms, from30\.6%30\.6\\%to50\.3%50\.3\\%\(\+19\.7\+19\.7pp\)\. These gains indicate that the original full\-corpus deficit was not simply caused by an insufficient top\-kkbudget: a substantial portion arose because the retriever lacked a representation of the very document\-level signal introduced during query disambiguation\. Prepending the title allows that signal to influence ranking directly, increasing both the probability that the correct paper is represented among the retrieved chunks and the amount of gold evidence that reaches the reader\.

The retrieval gains propagate consistently to answer quality \(Table[11](https://arxiv.org/html/2610.26951#A5.T11)\)\. On the answerable subset, the judge score improves for*every*model\. The four strongest readers \- GPT\-5, Sonnet\-4\.5, GPT\-OSS\-120B, and GLM\-5\.2 \- gain between\+0\.10\+0\.10and\+0\.13\+0\.13, while Nemotron improves by\+0\.03\+0\.03\. The answerable\-question refusal rate also falls by55–88,pp, consistent with fewer queries reaching the reader with distractor or insufficient context\. That the same retrieval\-only intervention benefits readers with different capabilities strengthens the interpretation that evidence availability is a shared bottleneck across otherwise diverse reader architectures and levels of baseline answer quality\. Part of the reader\-level degradation under full\-corpus RAG is therefore upstream: when document identity is represented during retrieval, more evidence survives the top\-1010bottleneck, giving all five readers a better opportunity to answer\. This shows that chunk representation can materially affect end\-to\-end RAG performance over a large, heterogeneous corpus\.

Table 11:Metadata\-in\-chunk retrieval onflat\_full@10\(QASPER\-MP, EBG, GPT\-5 judge, answerable subset,n=200n\{=\}200\)\. “meta” prepends the paper title to each chunk before embedding\. Judge improves over plain retrieval for every model \(bold\); “Ref%” is the answerable\-question refusal rate\. LC \(full document neighborhood\) is shown for reference\. The95%95\\%bootstrap CI half\-width on the judge scores is≈\\approx0\.070\.07\.Crucially,a stronger retriever narrows but does not close the LC gap\. With metadata\-in\-chunk,flat\_full@10trails LC on the answerable subset for all models \- by0\.040\.04\(Sonnet\-4\.5\),0\.060\.06\(Nemotron\),0\.080\.08\(GPT\-OSS\-120B\),0\.140\.14\(GLM\-5\.2\), and0\.150\.15\(GPT\-5\) judge points\. Roughly half the full\-corpus LC advantage is recoverable by fixing retrieval, but a residual remains: even when the right paragraphs are retrieved, removing document context costs answer quality relative to reading the full neighborhood\. This points beyond evidence discovery: retrieval preserves passages but discards surrounding structure\. That context supplies definitions, antecedents, methodological details, and cross\-paragraph relationships needed to interpret evidence\. The central finding remains robust to better RAG \- LC EBG still wins\.

![Refer to caption](https://arxiv.org/html/2610.26951v1/synthetic_ans_vs_ans_only_heatmap.png)\(a\)
![Refer to caption](https://arxiv.org/html/2610.26951v1/synthetic_delta_ans_star_and_ans_heatmap.png)\(b\)

Figure 4:Synthetic two\-hop benchmark comparingAns∗\(answer\-only prompting\) andAns\(underEBGprompting\)for three frontier models across 10K–1M characters\. \(a\) AbsoluteAns∗andAnsaccuracies\. Higher \(↑\\uparrow\) values are better\. \(b\) Accuracy difference between the two prompting settings\. Cells report percent accuracy averaged over≥50\\geq 50trials per context length\. Sonnet\-4\.5’s 1M entry is markedN/Abecause the input exceeds its 200K\-token context window\. The two prompting settings produce nearly identical performance on this controlled synthetic benchmark, indicating that the citation requirement neither substantially helps nor hurts answer accuracy\. Larger \(↑\\uparrow\) positive difference indicates effectiveness ofEBGoveranswer\-onlyprompting\.
### E\.8Summary

QASPER\-MPcorroborates the picture in Appendix[D](https://arxiv.org/html/2610.26951#A4): under the pool\-matchedpool11condition, long\-context EBG and RAG end up at comparable aggregate scores, but the mechanism is asymmetric\. RAG’s higher refusal rate inflates its score on unanswerable questions; on the answerable subset \- where retrieval quality actually matters \- LC EBG beats every RAG variant for every one of the five frontier models we tested\. The gap is largest for the strongest readers \(Opus\-4\.6 and GPT\-5\.5 each show\+0\.13\+0\.13judge points LC over best RAG on answerable questions\), suggesting that LC EBG benefits disproportionately from stronger long\-context comprehension\. The gap widens further when the retrieval corpus expands from the apples\-to\-apples 11\-paper pool to the realistic full\-corpus setting, where LC outperforms the best RAG configuration by 5–13 judge points\.

## Appendix FLong\-context EBG: Synthetic and NoLiMa\-based Datasets

This appendix collects the controlled long\-context EBG results that support RQ2–RQ4: the synthetic two\-hop benchmark \(§[F\.1](https://arxiv.org/html/2610.26951#A6.SS1)\) and the NoLiMa\-based benchmark \(§[F\.2](https://arxiv.org/html/2610.26951#A6.SS2)\)\.

### F\.1Ans\-only vs EBG on Synthetic Dataset

##### Observations\.

Fig\.[4](https://arxiv.org/html/2610.26951#A5.F4)compares evidence\-based generation \(EBG\) againstanswer\-onlyprompting on the synthetic two\-hop benchmark, highlighting both the advantages and limitations of requiring explicit evidence attribution\.

- •Performance rapidly saturates for all frontier models\.Across nearly the entire 10K–1M character range, all models operate close to ceiling performance, with accuracies clustering near 100% under both prompt settings\. This indicates that the synthetic two\-hop benchmark is largely solved by current frontier systems once the relevant evidence is successfully retrieved, and that increasing context length alone does not substantially reduce answer accuracy for the strongest models\.
- •Answer\-only and EBG prompting achieve nearly identical performance across all context lengths\.ForGPT\-5andGemini\-3\-Fl,Ans∗\(answer\-only prompting\) andAns\(answer with evidence prompting\) remain effectively indistinguishable from 10K to 1M characters, with almost all cells at or near 100% accuracy\. The difference heatmap confirms that the vast majority of entries have zero difference, indicating that adding citation requirements does not materially affect answer correctness on this synthetic benchmark\.

\(a\)Performance from 10K to 100K\.
\(b\)Performance from 200K to 700K\.
\(c\)Performance from 800K to 1M\.

Table 12:Performance of models across different context lengths, for one\-hop and two\-hop questions, averaged acrossn=16n=16\(4 needles×\\times4 depths\) cases,comparingAns∗\(answer\-only\) withAns\(EBG\)\.Δ\\Deltadenotes the differenceAns−Ans∗\\texttt\{Ans\}\-\\texttt\{Ans\}^\{\*\}; positive values are shown in boldblue, negative values in boldred\. Higher \(↑\\uparrow\) values are better\. Positive values indicate thatEBGprompting outperformsanswer\-onlyprompting, while negative values indicate thatanswer\-onlyprompting improves overEBGprompting\. Zero indicates no difference between the two prompting strategies\. “N/A” denotes unavailable entries due to unsupported context lengths\. Here we present scores correct upto two decimal places\.![Refer to caption](https://arxiv.org/html/2610.26951v1/combined_accuracy_heatmap_extended.png)Figure 5:Performance of different models onAns\+Evid \(EBG\)accuracy forone\-hop and two\-hopquestions across varying context lengths, averaged over needle depths\. Cells marked with “N/A” indicate no available results for that model\-context combination\. Higher \(↑\\uparrow\) values are better\)\.

### F\.2Ans\-only vs EBG on NoLiMa\-based Dataset

##### Observations\.

As shown in Table[12](https://arxiv.org/html/2610.26951#A6.T12),EBGprompting is more beneficial for multi\-hop queries than for simple one\-hop queries\. In one\-hop settings, many strong models already perform near ceiling under answer\-only prompting, leaving little room for explicit evidence organization to help\. In some cases, the additional structure introduced byEBGeven hurts performance, suggesting that it may interfere with models that already perform strong implicit retrieval or that are sensitive to extra reasoning overhead\. The benefits ofEBGbecome clearer in the two\-hop setting, where models must retrieve and combine evidence across multiple pieces of context\. The best long\-context performers areGPT\-5andGemini\-3\-Fl, but for different reasons\.GPT\-5is highly robust regardless of prompting strategy, whileGemini\-3\-Flgains substantially fromEBGin two\-hop settings\.

The strongest evidence in favor ofEBGcomes from theGemini\-3\-Fltwo\-hop results, whereEBGimproves performance consistently across almost all context lengths, including the 1M setting\. TheSonnetmodels also show frequent gains fromEBGin two\-hop tasks, further supporting the view that explicit evidence organization is most useful when reasoning requires multiple retrieval steps\. In contrast, the strongest evidence against universalEBGuse comes fromGPT\-4o, whereEBGsubstantially lowers performance in both one\-hop and two\-hop tasks\.

Overall, the table suggests thatEBGis especially valuable for long\-context multi\-hop reasoning in models, but it may be unnecessary for simple one\-hop retrieval\.

### F\.3Strict EBG on NoLiMa\-based Dataset

##### Observations\.

Fig\.[5](https://arxiv.org/html/2610.26951#A6.F5)reports the combinedAns\+Evidaccuracy across models, context lengths, and reasoning types\. Since this metric requires both the final answer and the supporting evidence to be correct, it is astrictermeasure than answer accuracy alone\. The results show clear differences in how well models remain both accurate and grounded as context length increases\.

##### One\-hop performance\.

In the one\-hop setting, the reported jointAns\+Evidmetric measures answer correctness together with citation of the primary contextual anchor, rather than a complete context\-only reasoning chain, because for one\-hop, part of the inference relies on*bridge knowledge*not stated in the haystack \(such as*veganism*implying avoidance of*fish*or*lactose*intolerance implying avoidance of*milk*\. Here, we observe that most frontier models perform very strongly at short and medium context lengths\.GPT\-5,Sonnet\-4,Sonnet\-4\.5,Gemini\-2\.5\-Fl, andGemini\-3\-Flfrequently achieve near\-perfect or perfect accuracy across many context lengths\. This suggests that when only one relevant piece of evidence needs to be retrieved, strong long\-context models can usually identify the evidence and produce the correct answer reliably\.

Gemini\-3\-Flis the strongest one\-hop model overall, maintaining 100% accuracy across all evaluated context lengths, including 1M characters\.GPT\-5is also highly robust, achieving 100% accuracy across most context lengths and only dropping slightly to 94% at 1M\.Gemini\-2\.5\-Flalso remains strong, although it declines at longer contexts, reaching 81% at 600K and 700K before recovering to 94% at 1M\.

GPT\-4\.1performs well in one\-hop reasoning, but its performance decreases at the largest context length, reaching 81% at 1M\.GPT\-4oando3\-minishow much weaker robustness\.GPT\-4odrops substantially after 100K characters, whileo3\-minidegrades sharply beyond 200K characters, falling to 31% at 500K and 19% at 700K\.

##### Two\-hop performance\.

The two\-hop setting is significantly harder\. Even models that perform very well in one\-hop reasoning show larger performance drops when they must retrieve and combine multiple pieces of evidence\. This indicates that long\-context multi\-hop reasoning requires more than simply locating one relevant span; the model must maintain multiple pieces of information and reason over their relationship\.

Gemini\-3\-Flis the strongest model in the two\-hop setting\. It maintains high accuracy across the context range and achieves 93% at 1M characters\.GPT\-5is strong from 10K to 700K, but drops to 81% at 1M\.GPT\-4\.1performs well through mid\-range contexts but declines at 1M, reaching 62%\.

### F\.4TN vs\. CTN on NoLiMa\-based Dataset: Adherence to Contextual Evidence

We conduct an experiment on the NoLiMa\-based dataset to evaluate whether LLMs remain faithful to contextual evidence in two\-hop reasoning questions when presented with either factual secondary evidence that aligns with world knowledge \(TN\) or counterfactual secondary evidence that conflicts with world knowledge \(CTN; e\.g\., “black coffee has milk”\)\.

##### Experiment\.

Under TN and CTN, each query is associated with two needles\. We insert the primary needle at the middle of the haystack \(depth0\.50\.5\) and the secondary needle at one of the sampled depths\{0\.0,0\.33,0\.67,1\.0\}\\\{0\.0,0\.33,0\.67,1\.0\\\}\. Since questions must be answered strictly from the provided context, success in CTN requires relying on the contradictory contextual evidence rather than parametric memory\. We evaluate the performance of TN and CTN usingAnsandAns\+Evidaccuracy metrics\.

##### Observations\.

- •Contradiction reliably hurts performance\.Averaged across models, CTN reducesAnsaccuracy by≈\\approx7\.0 pp\(76\.2%→\\to69\.2%\) andAns\+Evidby≈\\approx5\.9 pp\(49\.9%→\\to44\.0%\)\.
- •The Ans\+Evid accuracy metric is much harder than Ans\.Ans\+Evidis far belowAnsin bothTNandCTN, indicating many correct\-answer cases still fail to produce a complete evidence trail\.
- •Robustness varies sharply by model\.Gemini\-3\-FlandGPT\-5remain most stable; the largestAns\+Eviddrops underCTNoccur forGPT\-4\.1\(≈\\approx\-13 pp\),Gemini\-2\.5\-Fl\(≈\\approx\-10 pp\), ando3\-mini\(≈\\approx\-8\.6 pp\)\.
- •Depth sensitivity persists under contradiction,with the CTN penalty tending to be larger when the secondary needle is deeper, consistent with degraded multi\-evidence retrieval under harder placements, i\.e\., mostly at depths 0\.33 and 0\.67\.
- •CTN remains below TN across context lengths, indicating contradiction amplifies long\-context brittleness as both conditions degrade with length\.
- •Contradiction weakens evidence grounding, not just answers\.CTN lowers citation rates for both needles \(primary≈\\approx\-7\.4 ppand secondary≈\\approx\-4\.6 pp\), pointing to provenance failure as a primary failure mode under conflicting information\.
- •The answer dimension degrades too, but less than the joint metric\.Figure[6](https://arxiv.org/html/2610.26951#A6.F6)reportsAnsaccuracy for the same runs\. Every model exceptGemini\-3\-Flloses answer accuracy under contradiction, with the largest drops forGPT\-OSS\-120B\(≈\\approx\-17 pp\),Sonnet\-4\.5\(≈\\approx\-16 pp\), andGemini\-2\.5\-FlandSonnet\-4\(≈\\approx\-12 ppeach\)\.

Figure 6:TN vs\. CTNAns accuracy under EBG prompting\. We report an accuracy for the same runs whoseAns\+Evidaccuracy appears in Figure[3](https://arxiv.org/html/2610.26951#S5.F3)\. Higher \(↑\\uparrow\) values indicate better performance\.

### F\.5EBG vs\. CoT Prompting on NoLiMa\-based Dataset

##### Experiment\.

We compare answer accuracy under standard EBG and CoT prompting separately for one\-hop and two\-hop questions\. We defineΔ=Ans−Ansc\\Delta=\\texttt\{Ans\}\-\\texttt\{Ans\}^\{c\}, whereAnsdenotes answer accuracy under EBG prompting andAnsc\\texttt\{Ans\}^\{c\}denotes answer accuracy under CoT prompting\. Positive values ofΔ\\Deltatherefore indicate an advantage for EBG, whereas negative values indicate an advantage for CoT\.

![Refer to caption](https://arxiv.org/html/2610.26951v1/ebg_vs_cot_delta.png)\(a\)
![Refer to caption](https://arxiv.org/html/2610.26951v1/ebg_vs_cot_ebg_delta_ans.png)\(b\)

Figure 7:Comparison of answer\-accuracy differences between EBG vs\. CoT and EBG vs\. CoT\-EBG promptingon the NoLiMa\-based dataset, with results reported separately for one\-hop and two\-hop questions\.\(a\)EBG vs\. CoT, where each cell reportsΔ=Ans−Ansc\\Delta=\\texttt\{Ans\}\-\\texttt\{Ans\}^\{c\}in percentage points; positive values favor EBG and negative values favor CoT\.\(b\)EBG vs\. CoT\-EBG,where each cell reportsΔ=Ans−Ansc​e\\Delta=\\texttt\{Ans\}\-\\texttt\{Ans\}^\{ce\}; positive values favor EBG and negative values favor CoT\-EBG\. Cells markedN/Aindicate that no results are available for the corresponding model–context combination\.
##### Observations\.

As shown in Fig\.[7\(a\)](https://arxiv.org/html/2610.26951#A6.F7.sf1), the effect of CoT is substantially larger for two\-hop questions than for one\-hop questions\. On a model\-macro average, CoT improves answer accuracy by approximately8\.58\.5pp for one\-hop questions and27\.227\.2pp for two\-hop questions\. At the model level, CoT never reduces aggregate one\-hop accuracy: it improves accuracy for eight models and matches EBG forGPT\-5andGemini\-3\-Fl\. The largest one\-hop gains occur forGPT\-OSS\-120B\(\+23\.6\+23\.6pp\),GPT\-4o\(\+20\.0\+20\.0pp\), ando3\-mini\(\+19\.3\+19\.3pp\)\.

For two\-hop questions, CoT achieves higher aggregate answer accuracy for every evaluated model\. The improvements range from5\.95\.9pp forGPT\-5to68\.868\.8pp forGPT\-OSS\-120B, with large gains also observed forGLM\-5\.2\(\+42\.3\+42\.3pp\),o3\-mini\(\+33\.3\+33\.3pp\),Gemini\-2\.5\-Fl\(\+31\.3\+31\.3pp\), andGPT\-4o\(\+30\.8\+30\.8pp\)\. This larger two\-hop effect suggests that explicitly eliciting a short reasoning step is particularly useful when answering requires an additional inference beyond direct needle retrieval\. Because vanilla CoT also removes the citation and JSON\-output requirements, however, this comparison should be interpreted as a comparison between complete prompting strategies rather than as an isolated estimate of the effect of reasoning alone\.

### F\.6EBG vs\. CoT\-EBG Prompting on NoLiMa\-based Dataset

##### Setup\.

To examine whether explicit reasoning remains beneficial when evidence citation is retained, we compare standard EBG with CoT\-EBG\. We reportAnsso that this comparison is directly commensurable with the vanilla\-CoT comparison in Appendix[F\.5](https://arxiv.org/html/2610.26951#A6.SS5), which can only be scored on answer accuracy because the plain CoT prompt requests no citations\. Holding the metric fixed across the two experiments isolates the effect of the citation requirement itself\. We defineΔ=Ans−Ansc​e\\Delta=\\texttt\{Ans\}\-\\texttt\{Ans\}^\{ce\}, wherec​ecedenotes CoT\-EBG; negative values therefore favor CoT\-EBG\.

##### Observations\.

Fig\.[7\(b\)](https://arxiv.org/html/2610.26951#A6.F7.sf2)shows a smaller and more model\-dependent effect for one\-hop questions\. CoT\-EBG improves aggregate answer accuracy for five models, matches EBG for two, and lowers it for three, with a model\-macro average of2\.22\.2pp in favor of CoT\-EBG\. This mixed result is consistent with the near\-ceiling performance of several models on direct one\-hop retrieval, where an additional reasoning instruction has limited room to help and may introduce unnecessary output complexity\. The two\-hop result is more consistent: CoT\-EBG improves aggregateAnsaccuracy for every evaluated model, by12\.312\.3pp on a model\-macro average\. The gains range from2\.42\.4pp forGPT\-4\.1to25\.525\.5pp forGemini\-2\.5\-Fl\. Substantial improvements are also observed forGPT\-OSS\-120BandGLM\-5\.2\(\+23\.6\+23\.6pp each\),o3\-mini\(\+12\.0\+12\.0pp\), andGemini\-3\-Fl\(\+10\.4\+10\.4pp\)\. Thus, explicitly eliciting a brief reasoning step remains beneficial when the evidence\-citation requirement and output schema are retained, particularly for multi\-hop questions\.

Taken together, the two experiments suggest that the gains from structured prompting are not attributable to citation formatting alone\.Explicit reasoning improves answer accuracy both under vanilla CoT and when the EBG citation requirement is retained, with the largest and most consistent effects appearing in the two\-hop setting\.

## Appendix GRAG on NoLiMa\-based Dataset

We usetext\-embedding\-3\-smallto generate embeddings for both the NoLiMa haystack lines \(with inserted needles\) and the corresponding queries, and perform retrieval using cosine similarity\.

Table 13:Retrieval analysis ofRAGunder NoLiMa, showing average total vector count; minimum full\-coverage retrieval depth for one\-hop and two\-hop queries \(i\.e\., the smallest top\-kksuch that all required needles/evidence facts are contained within the retrieved set for every sample\) and corresponding retrieved character budget across varying haystack context lengths, averaged over three books\. Lower \(↓\\downarrow\) values are better\.### G\.1Retrieval Analysis

Table[13](https://arxiv.org/html/2610.26951#A7.T13)summarizes how the retrieval requirements of a RAG system changes as the haystack context length grows from10​K10\\mathrm\{K\}to1​M1\\mathrm\{M\}\. The first row reports the average number of vectors, or indexed chunks, produced from the full context\. As expected, this number increases with context length: for example, the retriever searches over only6666vectors at10​K10\\mathrm\{K\}context, but over5,7305\{,\}730vectors at1​M1\\mathrm\{M\}context\.

The remaining rows measure the retrieval depth required for*full\-coverage*\. Here, full coverage means that the retrieved top\-kkset contains all evidence facts needed to answer every query\. Thus, the reported number of vectors is the smallest value ofkksuch that no required evidence is missed\.

For one\-hop queries, where the answer depends on a single evidence fact, the required retrieval depth grows moderately with context length\. At10​K10\\mathrm\{K\}context, retrieving only22vectors, corresponding to350350characters, is sufficient for full coverage\. However, at1​M1\\mathrm\{M\}context, full coverage requires retrieving9696vectors, corresponding to22,91022\{,\}910characters\. This shows that even relatively simple queries require a larger retrieval budget as the haystack becomes longer\.

For two\-hop queries, where the answer depends on combining multiple evidence facts, the retrieval requirement grows much more sharply\. At10​K10\\mathrm\{K\}context, full coverage requires retrieving1010vectors, or1,9561\{,\}956characters\. At1​M1\\mathrm\{M\}context, however, it requires retrieving890890vectors, or202,412202\{,\}412characters\. This indicates that multi\-hop evidence is substantially harder to retrieve in long\-context settings, since the relevant pieces of information may be distributed across many more chunks\.

Several observations follow from these results\. First, the total number of indexed vectors grows approximately linearly with context length, increasing from6666vectors at10​K10\\mathrm\{K\}to5,7305\{,\}730vectors at1​M1\\mathrm\{M\}\. Second, the retrieval depth required for full coverage is much smaller than the total number of indexed vectors, but it still grows substantially as the context becomes longer\. Third, two\-hop queries are consistently more demanding than one\-hop queries: at1​M1\\mathrm\{M\}context, two\-hop full coverage requires890890vectors, compared with only9696vectors for one\-hop full coverage\. This is more than a9×9\\timesincrease in the number of retrieved vectors:

89096≈9\.27\.\\frac\{890\}\{96\}\\approx 9\.27\.
![Refer to caption](https://arxiv.org/html/2610.26951v1/needle_topk_requirement_heatmap.png)Figure 8:Minimumfull\-coverage retrieval depth \(kk\)\(for bookB1cB^\{c\}\_\{1\}underNoLiMa\)\. Lower \(↓\\downarrow\) values are better\.Fourth, the retrieved character budget also grows significantly\. At1​M1\\mathrm\{M\}context, one\-hop full coverage requires22,91022\{,\}910characters, whereas two\-hop full coverage requires202,412202\{,\}412characters\. Thus, two\-hop retrieval requires nearly nine times more retrieved text:

202,41222,910≈8\.83\.\\frac\{202\{,\}412\}\{22\{,\}910\}\\approx 8\.83\.
Another important observation is that standard small retrieval budgets, such as top\-55or top\-1010, are unlikely to provide full coverage in long\-context settings\. For instance, at100​K100\\mathrm\{K\}context, one\-hop full coverage already requires top\-1010, while two\-hop full coverage requires top\-9292\. At1​M1\\mathrm\{M\}context, the required two\-hop retrieval depth increases to top\-890890, which is far beyond typical RAG configurations\.

Overall, the table shows that standard small top\-kkretrieval settings may be insufficient for long\-context RAG, especially for multi\-hop queries\. While one\-hop retrieval remains relatively manageable, two\-hop retrieval at1​M1\\mathrm\{M\}context requires retrieving a large fraction of the indexed chunks:

8905730≈15\.5%\.\\frac\{890\}\{5730\}\\approx 15\.5\\%\.Therefore, as context length increases, guaranteeing full evidence coverage can require a very large retrieved character budget, reducing the efficiency advantage usually associated with RAG\.

### G\.2Character and Pronoun Sensitivity in Needle Retrieval

##### Motivation\.

During retrieval analysis, we observed that the0405needle family requires substantially larger top\-kkbudgets to achieve reliable retrieval compared to the0402family \(Figure[8](https://arxiv.org/html/2610.26951#A7.F8)\)\. This suggests that retrieval difficulty may be influenced not only by reasoning complexity but also by the*surface form of the needle itself*\. The default0405template contains a gendered pronoun:

> “Then\{CHAR\}mentioned that he has been vegan for years\.”

Because the pronounheis independent of the inserted character name, it may introduce additional semantic signals in the embedding representation that are not directly relevant to the query\.

##### Character sensitivity analysis\.

To investigate this effect, we compute cosine similarity between the query embedding and needle sentences instantiated with different character names drawn from the NoLiMa corpus\. Figure[9](https://arxiv.org/html/2610.26951#A7.F9)shows the top retrieved variants of the needle and their similarity scores for the query“Which character cannot eat Brandade?”\.

Figure 9:Ranking of the0405needle instantiated with 10 different character namesbased on cosine similarity to a one\-hop test query\. Although the sentence semantics remain unchanged, similarity scores vary across character tokens\. Higher \(↑\\uparrow\) values are better\.Although the sentence semantics remain unchanged, we observe noticeable variation in similarity scores across character substitutions\. This indicates that character tokens themselves influence the embedding representation and can affect retrieval ranking\.

##### Effect of pronoun removal\.

To isolate the role of the gendered pronoun, we construct a rephrased variant of the needle \(0405\-R, Table[2](https://arxiv.org/html/2610.26951#A2.T2)\) that removes the pronoun while preserving the meaning:

> “Then\{CHAR\}mentioned being vegan for years\.”

We recompute similarity scores for both the original and rephrased needles across multiple character substitutions\. Figure[10](https://arxiv.org/html/2610.26951#A7.F10)compares cosine similarity between the two formulations\.

Figure 10:Cosine similaritycomparison between the original0405needle and the rephrased0405\-Rvariant across 10 characters\. Removing the pronoun consistently increases similarity scores across different characters\. Higher positive difference \(↑\\uparrow\) indicates greater effectiveness of the rephrased version compared to the original\.Across most characters, the rephrased template yields slightly higher similarity scores\. While the improvement is modest for individual instances, it is systematic across characters, suggesting that removing the pronoun reduces semantic noise in the embedding space\.

##### Implications for RAG retrieval\.

These results help explain the higher retrieval budget required for the0405needle family observed earlier\. When the needle contains additional syntactic elements that are not directly aligned with the query semantics, embedding similarity becomes less discriminative\. In large haystacks, even small similarity differences can significantly affect ranking, pushing the correct needle lower in the retrieval list and increasing the required top\-kkbudget\.

Table 14:Retrieval analysis of three RAG pipelines under NoLiMa, showing total vector count; minimum full\-coverage retrieval depth for one\-hop and two\-hop queries \(i\.e\., the smallest top\-kksuch that all required needles/evidence facts are contained within the retrieved set for every sample\) and corresponding retrieved character budget across varying haystack context lengths for BM25\-RAG, Embedding\-RAG, and Reranker\-RAG\. Here all the results are computed only on Book 1 \(B1cB^\{c\}\_\{1\}\)\. Lower \(↓\\downarrow\) values are better\.

### G\.3Retrieval Analysis under Embedding, BM25, and Reranker\-RAG Pipelines

Motivated by the results and observations in Table[13](https://arxiv.org/html/2610.26951#A7.T13), we further investigate whether the observed retrieval bottleneck is specific to embedding\-based semantic retrieval or persists across alternative retrieval and ranking strategies\.

##### Setup of Embedding\-RAG\.

We first consider the embedding\-based RAG pipeline from the preceding experiment \(§[5\.1\.3](https://arxiv.org/html/2610.26951#S5.SS1.SSS3)\), restricting the evaluation to Book 1 \(B1cB^\{c\}\_\{1\}\)\. The pipeline follows the same retrieval setup as before: the haystack is divided into chunks, represented using dense vector embeddings, and ranked by semantic similarity to the query\. We retain the original chunking, retrieval, and full\-coverage evaluation protocol, and report the minimum top\-kkrequired to retrieve all evidence for one\-hop and two\-hop queries, together with the corresponding retrieved character budget\. Thus, the Embedding\-RAG results in Table[14](https://arxiv.org/html/2610.26951#A7.T14)serve as the semantic\-retrieval baseline against which Okapi BM25 and Qwen3\-Reranker\-8B are compared\.

##### Setup of BM25\-RAG\.

To examine whether the retrieval limitations observed with embedding\-based RAG are specific to semantic similarity search, we replace the vector embedding\-based retriever with Okapi BM25222https://pypi\.org/project/rank\-bm25/, a lexical retrieval method based on term matching and term\-frequency statistics\. We keep the evaluation protocol unchanged and measure the minimum retrieval depth required to achieve full evidence coverage for both one\-hop and two\-hop queries across increasing NoLiMa context lengths\. As in our primary analysis, full coverage is defined as the smallest top\-kkfor which all required evidence facts are present in the retrieved set for every sample\. We report both this retrieval depth and its corresponding character budget in Table[14](https://arxiv.org/html/2610.26951#A7.T14)\.

##### Setup of Reranker\-based RAG\.

We further evaluate whether a stronger learned ranking stage can reduce the retrieval depth required for complete evidence coverage\. Specifically, we useQwen3\-Reranker\-8B, a fine\-tuned reranking model, to reorder candidate passages \(retrieved by the embedding\-based RAG retriever\) according to their relevance to the query\. We then evaluate the resulting ranking using the same full\-coverage criterion as the BM25\-RAG and Embedding\-RAG settings\. This experiment tests the extent to which a learned reranker can move required evidence toward the top of the ranking and consequently reduce the context passed to the downstream model\.

##### Observations\.

We observe that BM25\-RAG consistently requires substantially larger retrieval budgets than Embedding\-RAG, with the difference becoming more pronounced as context length increases\. At 1M characters, full coverage of one\-hop queries requires 307 vectors \(102,475 characters\) with BM25\-RAG, compared with 104 vectors \(23,589 characters\) for Embedding\-RAG\. The same pattern holds for two\-hop queries, where BM25\-RAG requires 1,061 vectors \(265,785 characters\), compared with 920 vectors \(217,152 characters\) for Embedding\-RAG\. Thus, replacing semantic retrieval with Okapi BM25 does not remove the full\-coverage bottleneck; instead, lexical retrieval generally requires retrieving more of the haystack to guarantee that all necessary evidence is available\. This gap suggests that semantic similarity provides a stronger ranking signal for isolating relevant evidence as the distractor pool expands\. Moreover, the persistent two\-hop overhead across both retrievers indicates that jointly recovering multiple evidence pieces remains intrinsically more demanding than single\-evidence retrieval\.

As shown in Table[14](https://arxiv.org/html/2610.26951#A7.T14), Qwen3\-Reranker\-8B improves retrieval efficiency over BM25\-RAG and Embedding\-RAG\. For one\-hop queries, evidence ranks first across all context lengths, yielding full coverage at top\-11with 62 characters\. For two\-hop queries, depth grows with context length: at 1M characters, Reranker\-RAG reaches full coverage with 149 vectors \(21,372 characters\), compared with 920 vectors \(217,152 characters\) for Embedding\-RAG and 1,061 vectors \(265,785 characters\) for BM25\-RAG\. Thus, a fine\-tuned reranker dramatically reduces the retrieval budget for full evidence coverage\. Nevertheless, increasing top\-kkfor two\-hop queries shows that stronger reranking does not eliminate the difficulty of jointly surfacing evidence as the haystack grows\. The one\-hop–two\-hop contrast suggests that evidence multiplicity, not context length alone, drives the remaining burden: joint coverage becomes harder when multiple passages must survive the cutoff\.

### G\.4Randomly Sampled Haystacks vs\. Top\-kkRanked Retrieval

##### Setup\.

We examine whether LLMs answer more accurately when we give them a randomly sampled long\-context haystack or a compact context formed by top\-kkranked retrieval\. In the long\-context setting, we provide the model with the Book 1 haystack at a target character length and insert the needle at predefined depths\. This setting corresponds to randomly sampled haystacks: the long\-context construction procedure embeds the relevant evidence within a larger context and determines the surrounding distractors and needle positions\. In contrast, the retrieval setting replaces the full randomly sampled haystack with a retrieval\-ranked context\. For each query, we compute cosine similarity between the query embedding and each chunk embedding in the corresponding vector database, rank all chunks by similarity score, and select the minimum top\-kkprefix that contains all gold needle evidence for that condition\. We call this value the best top\-kk\.

Because long\-context and retrieval inputs differ substantially in structure, we compare them under approximately matched character budgets rather than matched nominal context labels\. For each Book 1 haystack, we first compute the total character count of the best top\-kkretrieved chunks\. We then select the long\-context condition whose haystack character count most closely matches that retrieval character count\. For example, the best top\-kkretrieved context for the 1M\-character haystack contains 217,152 characters; we therefore compare this retrieval condition against the 200K\-character long\-context condition, since the two inputs match most closely in character budget\. Throughout, “context length” denotes character count rather than token count\.

Table 15:Character\-budget matching procedure\. For each long\-context \(LC\) haystack used during the retrieval stage on bookB1cB^\{c\}\_\{1\}, we compute the minimum top\-kksuch that the retrieved set contains all required needles and evidence facts for every sample\. We then measure the character count of this retrieved set and match it to the LC condition with the closest character count\. The columns report, from left to right, the original LC haystack size, the minimum top\-kk\(in vectors\), its character count, and the matched LC budget used for comparison\.We use the same answer\+evidence prompt as the long\-context EBG setting, constructed from the selected top\-kkchunks and gold evidence metadata\. Models produce an answer with cited evidence\. We denote retrieval and matched long\-context answer accuracy asAnsRAG\\mathrm\{Ans\}\_\{\\mathrm\{RAG\}\}andAnsLC\\mathrm\{Ans\}\_\{\\mathrm\{LC\}\}, respectively\. The primary comparison is the accuracy delta:

Δ=AnsRAG−AnsLC\.\\Delta=\\mathrm\{Ans\}\_\{\\mathrm\{RAG\}\}\-\\mathrm\{Ans\}\_\{\\mathrm\{LC\}\}\.A positive value means that top\-kkranked retrieval outperforms randomly sampled long\-context prompting under the matched character budget, while a negative value means that randomly sampled long\-context prompting outperforms top\-kkranked retrieval\.

Table[15](https://arxiv.org/html/2610.26951#A7.T15)summarizes the character\-budget matching procedure\. The table reports, for each original Book 1 haystack length, the best top\-kkvalue, the character count of the corresponding retrieved context, and the matched long\-context character budget we use for comparison\.

![Refer to caption](https://arxiv.org/html/2610.26951v1/rag_answer_heatmap.png)Figure 11:Retrieval answer accuracy under the EBG prompting for the matched top\-kkretrieval contexts\. Columns denote the long\-context character budgets we use for comparison after matching each retrieved top\-kkcontext to the nearest long\-context haystack length\. Rows denote the evaluated models\. Each cell reports answer accuracy,AnsRAG\\mathrm\{Ans\}\_\{\\mathrm\{RAG\}\}, after we condition the model on the retrieved top\-kkchunks rather than the randomly sampled long\-context haystack\. Higher \(↑\\uparrow\) values are better\.![Refer to caption](https://arxiv.org/html/2610.26951v1/rag_lc_delta_heatmap.png)Figure 12:Answer accuracy difference between RAG\-EBG and LC\-EBG, compared under approximately matched character budgets \(Table[15](https://arxiv.org/html/2610.26951#A7.T15)\)\. Positive values indicate that RAG\-EBG outperforms LC\-EBG; negative values indicate that LC\-EBG outperforms RAG\-EBG\. Higher \(↑\\uparrow\) positive values are better for RAG\.Figure[11](https://arxiv.org/html/2610.26951#A7.F11)reports retrieval answer accuracy under the answer\+evidence prompt, and Figure[12](https://arxiv.org/html/2610.26951#A7.F12)reports the resultingΔ\\Deltadifference values across models and matched context lengths\.

##### Observations\.

Figure[12](https://arxiv.org/html/2610.26951#A7.F12)shows that the relative advantage of top\-kkranked retrieval versus randomly sampled long\-context prompting depends strongly on the model\. Since each cell reportsΔ=AnsRAG−AnsLC\\Delta=\\mathrm\{Ans\}\_\{\\mathrm\{RAG\}\}\-\\mathrm\{Ans\}\_\{\\mathrm\{LC\}\}, positive values indicate that the top\-kkretrieved context performs better, whereas negative values indicate that the randomly sampled long\-context haystack performs better\.

Overall, top\-kkranked retrieval competes strongly with long\-context prompting under the matched character\-budget setting\. For GPT\-4o, retrieval outperforms long\-context prompting in four of the five matched budgets: 50K \(\+31\+31pp\), 70K \(\+22\+22pp\), 100K \(\+28\+28pp\), and 200K \(\+9\+9pp\)\. GPT\-4o therefore benefits substantially when we replace the larger randomly sampled haystack with a compact retrieval\-ranked context\. GPT\-5 also shows near parity between the two settings, with deltas close to zero across all budgets \(\+3\+3,00,\+3\+3,\+6\+6, and00pp\)\. For GPT\-5, once the retrieved top\-kkcontext contains the relevant evidence, the additional surrounding haystack context provides little or no answer\-accuracy benefit\.

In contrast, GPT\-4\.1 consistently favors the randomly sampled long\-context haystack\. Its deltas remain negative at every matched budget, ranging from−9\-9pp at 100K to−34\-34pp at 90K\. GPT\-4\.1 therefore exploits information from the broader haystack context more effectively than from the compact top\-kkcontext, even when the retrieved context contains the necessary gold evidence\. The Claude models show a similar but weaker long\-context preference\. Sonnet\-4 improves under long\-context prompting at 50K, 70K, 90K, and 100K, with the largest gain at 90K \(−31\-31pp\), but retrieval performs slightly better at 200K \(\+6\+6pp\)\. Sonnet\-4\.5 also favors long\-context prompting at most matched budgets, especially 90K \(−25\-25pp\) and 100K \(−16\-16pp\), while matching retrieval at 50K and showing only a small long\-context advantage at 200K \(−6\-6pp\)\. O3\-mini swings unpredictably across budgets in both directions: long\-context prompting helps at 50K \(−12\-12pp\), retrieval helps at 70K and 200K \(\+12\+12pp and\+9\+9pp\), and the two settings tie at 90K and 100K\.

A notable pattern emerges at the 90K budget, which favors long\-context prompting most strongly\. Four models show large negative deltas: GPT\-4o \(−25\-25,pp\), GPT\-4\.1 \(−34\-34,pp\), Sonnet\-4 \(−31\-31,pp\), and Sonnet\-4\.5 \(−25\-25,pp\)\. Evidence completeness therefore does not guarantee retrieval superiority\. Even when the top\-kkcontext contains the required needles, the model may benefit from broader discourse, surrounding distractors, or positional structure of the original haystack\. Retrieval can therefore alter the contextual relationships through which evidence is interpreted, making compression effectiveness model\-dependent\. Conversely, strong positive deltas for GPT\-4o and near\-zero deltas for GPT\-5 show that compact retrieval removes distractor burden without hurting accuracy for some models\.

Overall, the results do not support a single universal winner between randomly sampled haystacks and top\-kkranked retrieval\. They instead reveal a model\-specific tradeoff\. Top\-kkretrieval outperforms long\-context prompting when compact evidence\-focused contexts help the model avoid distraction, as GPT\-4o demonstrates most clearly\. Long\-context prompting outperforms retrieval when models benefit from the broader haystack context, as GPT\-4\.1 and the Claude models demonstrate\. GPT\-5 emerges as the most retrieval\-robust model in this comparison, showing that a sufficiently strong model answers equally well from the compact retrieved context and the matched randomly sampled haystack\.

### G\.5Embedding\-RAG vs Reranker\-RAG under EBG Prompting

The preceding retrieval analysis \(Table[14](https://arxiv.org/html/2610.26951#A7.T14)\) shows that Qwen3\-Reranker\-8B can rank required evidence substantially closer to the top of the retrieved context than embedding\-based similarity search\. However, improved retrieval ranking does not necessarily translate into better end\-to\-end generation: even when the same evidence is available, its position within the retrieved context may affect a model’s ability to identify, reason over, and correctly attribute that evidence\. Motivated by this distinction, we evaluate whether the retrieval\-order improvements from reranking translate into higher jointAns\+Evidaccuracy when the amount of retrieved context is held fixed\.

Table 16:Comparison of Embedding\-RAG and Reranker\-RAGAnsaccuracy under EBG prompting\.Columns are indexed by the LC\-matched retrieved\-context budgets from Table[15](https://arxiv.org/html/2610.26951#A7.T15); for example,200Kdenotes the retrieval budget matched to the1MLC haystack\. Each cell reports answer accuracy overn=8n=8samples \(4 needles×\\timesone\-hop/two\-hop\)\. Emb: Embedding\-RAG\. Higher \(↑\\uparrow\) values indicate better performance\.Table 17:Dataset statistics\.For the synthetic benchmark, we construct 66,834 unique single\-sentence distractor facts and curate 10 two\-hop needles, evaluating 100 independently generated trials per context length\. For theNoLiMa\-based benchmark, we use four primary needle variants at four insertion depths, yielding 16 needle\-placement conditions per context length per model\. ForMultiHop\-RAG, removing 301null\_queryexamples leaves 2,255 answerable queries; we evaluate 50 queries with proprietary models and 200 with open\-weight models\. ForQASPER\-MP, experiments use the QASPER development split containing 1,005 questions over 281 papers\. Each gold paper is paired with ten distractor papers\. We evaluate 50 queries \(38 answerable and 12 unanswerable\) with proprietary models and 200 queries \(150 answerable and 50 unanswerable\) with open\-weight models\.##### Setup\.

We compare Embedding\-RAG and Reranker\-FixedTopk\-RAG across four LLMs\. For each LC\-matched budget in Table[15](https://arxiv.org/html/2610.26951#A7.T15), Embedding\-RAG retrieves full\-coverage top\-kkchunks by cosine similarity\. Reranker\-RAG uses*the same chunks*and preserves the top\-kkand character budget, but re\-orders them usingQwen3\-Reranker\-8Brelevance scores\. No chunks are added or removed; the pipelines differ only in evidence ordering\. This comparison isolates whether placing relevant evidence earlier improves generation rather than conflating ranking quality with context volume\. With evidence availability and context size fixed, differences can be attributed to ranking and reveal whether models benefit from prioritizing relevant chunks\. We evaluate both pipelines with our EBG prompt overn=8n=8examples, comprising four needles under one\-hop and two\-hop queries\. Results appear in Table[16](https://arxiv.org/html/2610.26951#A7.T16)\.

##### Observations\.

Table[16](https://arxiv.org/html/2610.26951#A7.T16)shows that reranking improves end\-to\-end answer accuracy over embedding\-based retrieval, with gains particularly pronounced at larger context budgets\. Sonnet\-4\.5 benefits consistently from reranking, reaching 100% accuracy from 50K–100K and retaining an advantage at 200K\. GPT\-OSS\-120B exhibits the largest gain at longer contexts, improving from 62\.5% to 100% at 100K and from 50% to 87\.5% at 200K\. GPT\-4\.1 and GLM\-5\.2 also improve in most settings, although reranking is not uniformly beneficial: GPT\-4\.1 drops from 87\.5% to 75% at 100K, while GLM\-5\.2 drops from 62\.5% to 50% at 50K and remains unchanged at 90K\.

Overall, across the 20 model–budget combinations, Reranker\-RAG improves over Embedding\-RAG in 15 cases, ties in three, and underperforms in only two, indicating thatimproved evidence ordering can translate into better downstream answer generation even when the retrieved context budget is fixed\.The larger gains at higher budgets suggest that retrieval quality remains important even when more context is available, particularly as irrelevant passages increasingly compete for the reader’s attention\. Because both pipelines contain identical chunks, these gains isolate ordering rather than evidence availability, suggesting that readers are sensitive to where relevant evidence appears within the retrieved context\.

Table 18:Models benchmarked in this work, along with their providers, source URLs, and licenses\.

## Appendix HDataset Source and Statistics

Table[17](https://arxiv.org/html/2610.26951#A7.T17)summarizes all datasets used in this work along with their sources and licenses\. We construct or adapt all datasets from publicly available sources\. We construct the synthetic two\-hop haystack entirely using iterative LLM prompting for distractor generation and manually curated needles, and we release it under CC BY 4\.0\. The NoLiMa\-based dataset adapts the haystack construction procedure and needle set from[Modarressi et al\. \(2025\)](https://arxiv.org/html/2610.26951#bib.bib5), which is released under the Adobe Research License\. We useMultiHop\-RAG\([Tang and Yang, 2024](https://arxiv.org/html/2610.26951#bib.bib16)\)as released under the ODC\-By license, and re\-map gold evidence facts to line\-level labels for EBG evaluation\. We deriveQASPER\-MPfrom theQASPERdev split\([Dasigi et al\., 2021](https://arxiv.org/html/2610.26951#bib.bib17)\), released under CC BY 4\.0, and extend it to a multi\-paper setting by pairing each gold paper with ten distractor papers and rewriting questions with short realistic paper cues\.

## Appendix IModel Information

We evaluate ten proprietary frontier and three open\-weight models via their APIs\. Table[18](https://arxiv.org/html/2610.26951#A7.T18)lists each model along with its provider, source URL, and license\. All models were queried at temperature 0 \(or the provider\-equivalent default\) with no additional fine\-tuning or adaptation\.

## Appendix JArtifact Use and Intended Use

We use all existing datasets and models consistently with their intended use\.MultiHop\-RAG,QASPER, andNoLiMaare used strictly for research evaluation, consistent with their release conditions\. All proprietary models are accessed via official APIs in accordance with provider terms of service\. All derived datasets introduced in this work are intended exclusively for research use and do not conflict with the access conditions of the original artifacts\.

## Appendix KDisclosure of AI Use

We used AI assistants \(large language models\) at several stages of this work\. We used them to review and debug code for our experimental pipelines and evaluation scripts, to polish and refine the writing of the manuscript by improving clarity, tightening phrasing, and catching grammatical errors, and to suggest alternative phrasings for titles, figure captions, and section headings\. We developed all research ideas, experimental designs, benchmark constructions, analyses, and conclusions ourselves\. We verified all AI\-assisted code changes through testing, reviewed and edited all AI\-assisted text, and take full responsibility for the final content of the paper\.