更好的检索器，更差的标签：不完整的相关性判断如何扭曲对话式 RAG 中的拒答评估

# How Incomplete Relevance JudgmentsDistort Abstention Evaluation in Conversational RAG
Source: [https://arxiv.org/html/2610.40966](https://arxiv.org/html/2610.40966)
## Better Retrievers, Worse Labels: How Incomplete Relevance Judgments Distort Abstention Evaluation in Conversational RAG

###### Abstract

Conversational RAG benchmarks such as MTRAG and RETECO evaluate retrieval and generation end to end: each system answers from its own retrieval, so abstention must be scored against whether each retrieved context is sufficient\. A convenient shortcut reads sufficiency off the relevance judgments and treats a context without gold documents as unanswerable\. We re\-run the official baselines exactly and test this shortcut on RECOR \(RETECO track 2\) with four retrievers \(nDCG@10 from \.18 to \.59\) and three Qwen3 readers\. Two open LLM judges and a panel of four small open models label each context by content, and a model of another family audits 250 contexts blind, with adjudication\. The stronger the retriever, the larger the share of no\-gold contexts that still contain the answer \(28% to 72%\) and the lower the shortcut’s precision \(\.68 to \.10; \.70 to \.21 under the audit\)\. A reader agrees: Qwen3\-4B answers 62–67% of these false misses correctly, against 23–30% of no\-gold contexts judged insufficient and 1% with no documents\. In most false misses, the answer comes from a hard negative mined for the same question, another turn’s gold or a neighbouring chunk of the gold page\. On MTRAG, whose judgments come from annotators, precision drops from \.72 to \.50\. Under the shortcut, answerability AUROC is up to \.19 too low, and the best sufficiency detector appears to be a retrieval score that never reads the documents\. We release code, runs and labels, and recommend scoring refusals against content labels, not the presence of gold\.

## 1Introduction

A retrieval\-augmented reader should answer when the retrieved documents support an answer and abstain when they do not[Joren et al\. \(2025\)](https://arxiv.org/html/2610.40966#bib.bib8);[Song et al\. \(2025\)](https://arxiv.org/html/2610.40966#bib.bib9)\. In conversational search this decision is frequent, because turns are elliptical and first\-stage retrieval often misses\. Recent benchmarks therefore evaluate retrieval and generation end to end, with each system answering from what it retrieved: MTRAG[Katsis et al\. \(2025\)](https://arxiv.org/html/2610.40966#bib.bib13), MTRAGEval at SemEval\-2026[Rosenthal et al\. \(2026b\)](https://arxiv.org/html/2610.40966#bib.bib14)and RETECO at SemEval\-2027[Abdallah et al\. \(2026\)](https://arxiv.org/html/2610.40966#bib.bib17)\. RETECO pairs two settings on the same turns: answering from the gold documents \(task 2b\) and from one’s own retrieval \(task 2c\)\.

Scoring abstention end to end requires knowing, for each retrieved context, whether it suffices\. Labelling every context of every system by hand is infeasible, so the convenient protocol reads sufficiency off the relevance judgments: a context that contains none of the turn’s gold documents is*insufficient*\. At the question level, MTRAG and MTRAGEval call a turn unanswerable when annotators found no relevant passage[Katsis et al\. \(2025\)](https://arxiv.org/html/2610.40966#bib.bib13);[Rosenthal et al\. \(2026b\)](https://arxiv.org/html/2610.40966#bib.bib14), and unanswerability is created by deleting gold passages from the index[Chandradevan et al\. \(2026\)](https://arxiv.org/html/2610.40966#bib.bib12)\. At the context level, insufficiency detectors are trained and tested on whether the annotated passages are present in the retrieved set[Chen et al\. \(2026b\)](https://arxiv.org/html/2610.40966#bib.bib10)\. The 2b/2c pairing invites the same reading: 2b is sufficient by construction, and a 2c context without gold looks like natural unanswerability\.

Figure 1:A false miss of the Fusion retriever \(medical sciences\)\. The retrieved document contains the answer but is a negative in the qrels, so the two labels grade the same reader behaviour in opposite ways\.Relevance judgments are incomplete, and stronger systems retrieve more unjudged relevant documents[Arabzadeh et al\. \(2022\)](https://arxiv.org/html/2610.40966#bib.bib21);[Weller et al\. \(2025\)](https://arxiv.org/html/2610.40966#bib.bib24)\. Fig\.[1](https://arxiv.org/html/2610.40966#S1.F1)shows the consequence: the retriever finds the answer in a document the qrels do not credit, and the protocol rewards the reader that refuses\. No earlier use measures how its validity changes with the retriever\. In this reproducibility paper we re\-run the official RETECO baselines exactly, reproduce the context\-level label on RECOR[Ali et al\. \(2026\)](https://arxiv.org/html/2610.40966#bib.bib16)\(different team, different setup\), and generalise it along what earlier uses held fixed: the retriever, the source of the label and the benchmark\. We ask:

1. RQ1Does the qrels\-derived sufficiency label stay valid as the retriever improves?
2. RQ2Are the content\-based labels that contradict it trustworthy?
3. RQ3Does the choice of label change conclusions about abstention?
4. RQ4Does the effect generalise to MTRAG, whose qrels come from annotators and whose corpus has no mined negatives?

The protocol’s precision falls from \.68 to \.10 as nDCG@10 rises from \.18 to \.59 \(§[4\.1](https://arxiv.org/html/2610.40966#S4.SS1)\)\. Independent checks support the content labels: a judge\-free count of gold copies, a blind audit by a model of another family, a negative control, human judgments on MTRAG and a reader’s own answers; four small open judges reproduce the decline but are more lenient \(§[4\.2](https://arxiv.org/html/2610.40966#S4.SS2)\)\. The label changes which readers and detectors look good \(§[4\.3](https://arxiv.org/html/2610.40966#S4.SS3)\), and the decline replicates on MTRAG \(§[4\.4](https://arxiv.org/html/2610.40966#S4.SS4)\)\. We argue that the protocol can reverse conclusions whenever retrievers of different strength are compared, and that abstention in end\-to\-end benchmarks should be scored against content\-based labels of each retrieved context \(§[5](https://arxiv.org/html/2610.40966#S5)\)\. Code, runs, model outputs, labels and audit annotations are available\.111[https://github\.com/iskochergin/reteco\-2026](https://github.com/iskochergin/reteco-2026)

## 2Related Work

#### Sufficiency and abstention in RAG\.

Abstention has been studied for reading comprehension with unanswerable questions[Rajpurkar et al\. \(2018\)](https://arxiv.org/html/2610.40966#bib.bib1);[Trivedi et al\. \(2022\)](https://arxiv.org/html/2610.40966#bib.bib2), for LLMs in general[Wen et al\. \(2025\)](https://arxiv.org/html/2610.40966#bib.bib5);[Kirichenko et al\. \(2025\)](https://arxiv.org/html/2610.40966#bib.bib6), and for RAG, where the reader must notice that the retrieved text does not support an answer[Chen et al\. \(2024\)](https://arxiv.org/html/2610.40966#bib.bib4);[Peng et al\. \(2025\)](https://arxiv.org/html/2610.40966#bib.bib7)\.[Joren et al\. \(2025\)](https://arxiv.org/html/2610.40966#bib.bib8)define*sufficient context*by content and label it with an LLM autorater;[Song et al\. \(2025\)](https://arxiv.org/html/2610.40966#bib.bib9)score groundedness and refusal per context against gold claims\. Refusal after a retrieval miss has been tabulated per retriever[von Däniken et al\. \(2026\)](https://arxiv.org/html/2610.40966#bib.bib11)\. None of these compares content labels with qrels\-derived labels across retrievers\.

#### Unanswerability in shared tasks\.

The MTRAG and SemEval\-2026 Task 8 organizers acknowledge likely missed relevant passages and a likely bias toward the ELSER retriever used to build the data[Katsis et al\. \(2025\)](https://arxiv.org/html/2610.40966#bib.bib13);[Rosenthal et al\. \(2026a\)](https://arxiv.org/html/2610.40966#bib.bib39), and the winning retrieval system attributes part of ELSER’s advantage to this bias[Athanasiou et al\. \(2026\)](https://arxiv.org/html/2610.40966#bib.bib41)\. The task overview notes one unanswerable task for which 27 systems retrieved relevant passages[Rosenthal et al\. \(2026b\)](https://arxiv.org/html/2610.40966#bib.bib14)\. Participants report that all unanswerable generation items come with empty reference passages, a shortcut for abstention[Bondarenko et al\. \(2026\)](https://arxiv.org/html/2610.40966#bib.bib15), and that IDK conditioning penalises hedged grounded answers[Dutra et al\. \(2026\)](https://arxiv.org/html/2610.40966#bib.bib42)\. Outside MTRAG, CLAPNQ, whose corpus is one MTRAG domain, attributes its drop in unanswerable accuracy under retrieval to questions that become answerable from the retrieved passages[Rosenthal et al\. \(2025\)](https://arxiv.org/html/2610.40966#bib.bib40); in TyDi QA, retrieval misses account for 50–74% of the annotated unanswerable questions[Asai and Choi \(2021\)](https://arxiv.org/html/2610.40966#bib.bib3); CRAG scores a missing answer as zero whether or not the retrieved pages contained it[Yang et al\. \(2024\)](https://arxiv.org/html/2610.40966#bib.bib44); and LiveRAG notes that reference answers built from one or two documents can penalise correct answers found elsewhere[Carmel et al\. \(2025\)](https://arxiv.org/html/2610.40966#bib.bib46)\. These observations concern single items or ranking metrics; none measures how often a context without gold contains the answer as retrievers improve\.

#### Incomplete relevance judgments\.

Pooling leaves relevant documents unjudged and biases evaluation against systems that retrieve them[Zobel \(1998\)](https://arxiv.org/html/2610.40966#bib.bib18);[Buckley and Voorhees \(2004\)](https://arxiv.org/html/2610.40966#bib.bib19);[Yilmaz and Aslam \(2006\)](https://arxiv.org/html/2610.40966#bib.bib20)\. Shallow judgments miss many relevant passages in MS MARCO and BEIR, where dense retrievers suffer most[Arabzadeh et al\. \(2022\)](https://arxiv.org/html/2610.40966#bib.bib21);[Thakur et al\. \(2021\)](https://arxiv.org/html/2610.40966#bib.bib22), and mined hard negatives are often relevant[Qu et al\. \(2021\)](https://arxiv.org/html/2610.40966#bib.bib23);[Moreira et al\. \(2024\)](https://arxiv.org/html/2610.40966#bib.bib45);[Thakur et al\. \(2025\)](https://arxiv.org/html/2610.40966#bib.bib43); on TREC DL19, the share of unjudged top\-10 documents that are relevant rises from 50% for MonoT5 to 86–88% for reasoning rerankers[Weller et al\. \(2025\)](https://arxiv.org/html/2610.40966#bib.bib24)\. BRIGHT corpora contain unlabelled copies of gold documents that penalise stronger retrievers more[Sharifymoghaddam et al\. \(2026\)](https://arxiv.org/html/2610.40966#bib.bib25)\. LLM judgments can fill such holes[Thomas et al\. \(2024\)](https://arxiv.org/html/2610.40966#bib.bib26);[Upadhyay et al\. \(2024\)](https://arxiv.org/html/2610.40966#bib.bib27), with risks of circularity[Clarke and Dietz \(2025\)](https://arxiv.org/html/2610.40966#bib.bib28)\. We show that the same incompleteness becomes a retriever\-dependent bias in a binary label that decides which abstentions count as correct\.

## 3Reproduction Setup

#### Data\.

RETECO track 2 uses RECOR[Ali et al\. \(2026\)](https://arxiv.org/html/2610.40966#bib.bib16)\. Each dialogue expands a question from StackExchange \(drones, politics, law, medical sciences, hardware\) or from BRIGHT[Su et al\. \(2025\)](https://arxiv.org/html/2610.40966#bib.bib34)\(economics, psychology, biology, sustainable living, robotics, earth science\) into 3–5 short, context\-dependent turns\. Every turn has gold documents and a reference answer generated by GPT\-4\.1 from verified atomic facts; each domain corpus \(16k–121k documents\) includes negatives mined for each question\. We use the 858 dev turns with gold \(211 dialogues\)\. As the task allows, the official dialogue history contains the reference answers of earlier turns\.

#### The protocol\.

Theqrels labelcounts the gold documents in a top\-kkcontext \(k=5k\{=\}5\):all,partornone; the protocol treatsnoneas insufficient\. Re\-running the official starter kit reproduces the published nDCG@10 of both BM25 baselines \(turn only, and turn plus the official history\) in all 11 domains to four decimals \(dev macro \.183 and \.438\)\. We then build a ladder of first stages that the protocol has not been tested on:BM25\-turnsearches with the current turn only;BM25\-histis the official baseline;Fusionmin\-max fuses BM25 runs over three LLM query expansions \(a Query2doc passage[Wang et al\. \(2023\)](https://arxiv.org/html/2610.40966#bib.bib32), a rewrite with a passage, and a rewrite with a HyDE passage[Gao et al\. \(2023\)](https://arxiv.org/html/2610.40966#bib.bib33)\);Hybridadds a dense retriever \(reason\-embed\-qwen3\-4b[Chen et al\. \(2026a\)](https://arxiv.org/html/2610.40966#bib.bib35)\) to Fusion\. All weights were chosen on train\. Dev nDCG@10 is \.183, \.438, \.550 and \.591, and the share of turns with at least one gold document in the top\-5 is \.28, \.65, \.77 and \.82\.

#### Readers\.

Qwen3\-1\.7B, Qwen3\-4B\-Instruct\-2507 and Qwen3\-8B[Yang et al\. \(2025\)](https://arxiv.org/html/2610.40966#bib.bib36)answer from the numbered documents and put an explicit status,ANSWERABLEorUNANSWERABLE, on the first line; we readp⁡\(answerable\)p\(\\textsc\{answerable\}\)from the first\-token distribution\. Each reader answers every turn from the gold documents \(2b\), from the top\-5 of each retriever \(2c\), and without documents\. A refusal is anUNANSWERABLEstatus; responses without a status line \(at most 75 of 858 per context set\) are excluded from refusal rates and AUROCs\. We use vLLM[Kwon et al\. \(2023\)](https://arxiv.org/html/2610.40966#bib.bib38)\.

#### Content labels\.

A judge reads the dialogue, the latest question, the reference answer and the five numbered documents \(without IDs\), and decides whether the documents support the core of the reference answer:yes,partial\(some but not all central points\) orno\. It also names the supporting documents\. Two open judges of different families label all858×5=4,290858\\times 5=4\{,\}290contexts \(the top\-5 of each retriever and the gold contexts\): phi\-4[Abdin et al\. \(2024\)](https://arxiv.org/html/2610.40966#bib.bib37)\(primary\) and Qwen3\-14B[Yang et al\. \(2025\)](https://arxiv.org/html/2610.40966#bib.bib36)\. To test whether the decline depends on judge size or family, four small open models from four families \(two of them shared with the large judges\) label the same contexts with the same prompt: Qwen3\-4B\-Instruct\-2507, Phi\-4\-mini\-instruct \(3\.8B\), SmolLM3\-3B and Granite\-3\.3\-8B\-instruct; their*panel*label is the majority, with ties counted aspartial\. Finally, two runs of Claude Opus 5\.5 annotate 250 contexts blind \(100 Fusion contexts, halfnone, and 50nonecontexts from each other retriever\), shuffled and without IDs, so that the annotator sees neither the retriever nor the qrels label\. The two runs follow different procedures \(holistic versus claim by claim\) and agree on 97% of items \(self\-consistency of one model, not inter\-annotator agreement\)\. A third run adjudicates\. Binary metrics compareyeswithnoand droppartial\.

#### Statistics\.

Confidence intervals are 95% percentile intervals from 1,000 bootstrap resamples of turns \(seed 0\); differences are paired\. Resampling dialogues instead moves interval bounds by at most \.03\.

#### Scope and process of the reproduction\.

Re\-run exactly: the official BM25 baselines\. Reproduced: the context\-level qrels label as used for insufficiency detection[Chen et al\. \(2026b\)](https://arxiv.org/html/2610.40966#bib.bib10), a deterministic function of the qrels and the run; RETECO itself does not score abstention, so we apply the label to its 2b/2c pairing\. Generalised: the first\-stage retriever, the source of the sufficiency label, and the benchmark \(MTRAG\)\. Everything needed was public \(starter kit, qrels, MTRAG annotations\), so no contact with the original authors was required\. Two difficulties arose: the kit’s format checker needs an explicit document key for track 2, and the query expansions behind Fusion and the hybrid came from a closed model, Claude Sonnet 5 \(regenerated with an open model, the Query2doc passages give \.453 instead of \.546 with BM25\)\. LLM inference ran on single A100 or L4 GPUs in bfloat16 with greedy decoding, and dense corpus embeddings in float16 \(partly on T4\); prompts, model identifiers and settings are released with the artifacts, and every number in the paper is recomputed by the provided scripts from saved outputs, including the query expansions and audit annotations\.

## 4Results

### 4\.1RQ1: The qrels label degrades with retrieval quality

Figure 2:\(a\) Share of top\-5 contexts labelled insufficient by the qrels label \(no gold\) and by the two content judges \(no;partialcounted as sufficient\), against first\-stage nDCG@10\. \(b\) Precision of “no gold” as a detector of content insufficiency under each labeller; the audit covers 50nonecontexts per retriever; grey: the four small judges of Table[1](https://arxiv.org/html/2610.40966#S4.T1)\. Bars: 95% bootstrap CIs\.Withpartialcounted as sufficient, the qrels label overstates insufficiency, and the overstatement grows along the ladder \(Fig\.[2](https://arxiv.org/html/2610.40966#S4.F2)a\): 71\.6% of BM25\-turn contexts lack gold and 45\.0% lack the answer according to phi\-4 \(1\.6×\\times\), while for the hybrid the figures are 18\.4% and 4\.9% \(3\.8×\\times\)\. Read as a detector of insufficient context, “no gold” has precision \.68 \[\.65, \.72\] on BM25\-turn, \.44 on BM25\-hist, \.18 on Fusion and \.10 \[\.05, \.16\] on the hybrid \(Fig\.[2](https://arxiv.org/html/2610.40966#S4.F2)b\)\. Every consecutive step is significant, including Fusion to the hybrid \(−\.08\-\.08\[−\-\.14,−\-\.02\]\), and precision falls in all 11 domains\. The share ofnonecontexts that a judge finds sufficient \(*false misses*\) rises from 28% to 45%, 68% and 72%\.

Much of the drop comes from the base rate:nocontexts become rare on strong retrievers \(386, 157, 68 and 42\), so false misses outnumber them\. But abstention evaluation depends on precision, because it sets how many sufficient contexts sit in the class against which refusals are scored\. The label also becomes less informative beyond the base rate: recall minus the false\-alarm rate \(Youden’sJJ\) falls from \.52 to \.15 \(difference−\.37\-\.37\[−\-\.51,−\-\.22\]\)\. The overstatement grows only under this convention; precision falls under either: withpartialcounted as insufficient, the ratio is 1\.07–1\.30×\\timesunder phi\-4 with no trend and below 1 under Qwen3\-14B on the three stronger retrievers, while precision still falls from \.72 to \.28 and the false\-miss share, now includingpartial, rises from 39% to 92%\.

### 4\.2RQ2: Are the content labels right?

A skeptic can object that LLM judges credit on\-topic documents too readily, and that this leniency, rather than missing qrels, makes strong retrievers look sufficient\. We test the content labels in five ways\.

#### Without any judge\.

Some false misses need no judgment: the context contains a near\-verbatim copy of a gold document of the turn \(at least 80% of its word 8\-grams, or character 5\-gram Jaccard≥\.7\{\\geq\}\.7\), so it is relevant by the qrels’ own standard and is missed only because the copy has another ID\. Such contexts make up 1\.6% \[0\.7, 2\.8\] ofnonecontexts on BM25\-turn, 6\.0% on BM25\-hist, 16\.4% \[11\.2, 21\.7\] on Fusion and 9\.5% on the hybrid: a judge\-free lower bound on how often “no gold” is wrong, ten times higher on Fusion than on BM25\-turn\.

#### Other labellers\.

Table[1](https://arxiv.org/html/2610.40966#S4.T1)gives the precision of “no gold” under every labeller\. Qwen3\-14B marks more contexts insufficient than phi\-4, but its precision falls in the same way \(\.84 to \.21\), and so does the blind audit’s \(\.70, \.62, \.23, \.21\)\. Each of the four small judges, and their panel, reproduces the decline with a confidence interval that excludes zero \(−\.32\-\.32to−\.55\-\.55\)\. The small judges are, however, more lenient: they label 7–31% of the negative\-control contextsyes\(Table[1](https://arxiv.org/html/2610.40966#S4.T1), last column\), three of them have lower precision than both large judges at every stage, and they agree with each other and with the audit only fairly \(Fleiss’κ\\kappa\.26 on three labels;κ\\kappa\.18–\.25 with the audit\)\. They confirm the direction of the effect, not its size\. Among the larger labellers agreement is fair to moderate: Cohen’sκ\\kappa[Cohen \(1960\)](https://arxiv.org/html/2610.40966#bib.bib29)is \.44 between phi\-4 and Qwen3\-14B on all 4,290 contexts, and \.38 and \.49 between the audit and the two judges; on the binaryyes/nodecision it is \.74, \.59 and \.79, and raw three\-label agreement is \.70, \.64 and \.68\. The large judges are not uniformly lenient: phi\-4 labels 59 of the 858 gold contextsno\(Qwen3\-14B 56\) and names as evidence only about half of the near\-verbatim copies of gold that Fusion retrieves\. On the stronger retrieversκ\\kappafalls, but raw agreement and Gwet’s AC1[Gwet \(2008\)](https://arxiv.org/html/2610.40966#bib.bib30)do not, consistent with the rising share ofyes\. We therefore draw conclusions from stage\-level trends, on which all seven labellers agree\.

Table 1:Precision of “no gold” as a detector of content insufficiency \(no;partialexcluded\) under each labeller; its change from BM25\-turn to the hybrid with 95% CI; agreement \(κ\\kappa, three labels\) with the blind audit on its 250 contexts; and the share ofyesin the negative control \(contexts of other dialogues\)\. Panel: majority of the four small judges, ties aspartial\. Unparsable outputs \(at most 27 of 858 per context set\) are dropped from precision; control shares are over all 858 contexts\.
#### Negative control\.

Each turn is paired with the hybrid’s top\-5 for a turn of another dialogue in the same domain, with the turn’s own gold removed\. Qwen3\-14B labels none of the 858 contextsyes, phi\-4 labels 35 \(4\.1%\), and the small judges 6\.8–30\.7% \(Table[1](https://arxiv.org/html/2610.40966#S4.T1)\)\. Correcting for phi\-4’s leniency changes the false\-miss rates little \(28% to 25% on BM25\-turn, 72% to 70% on the hybrid\)\. This rules out indiscriminate leniency, but not leniency toward documents about the same question; the next two checks address that\.

#### A reader answers from false\-miss contexts\.

If false misses really contain the answer, a reader should answer them correctly more often than othernonecontexts, and should not manage without the documents\. Two LLM answer judges \(phi\-4 and Qwen3\-8B\) compare each response with the reference answer without seeing any sufficiency label; an answer counts as correct if the reader answered and correctness is at least 4 of 5\. On BM25\-turn, BM25\-hist and Fusion \(answer judges were not run on the hybrid\), Qwen3\-4B answers 62–67% of false misses correctly, nearly always faithfully to the documents, against 23–30% of true misses \(nonecontexts that phi\-4 judges insufficient\) and 1% of the same turns without documents \(paired difference\+\.61\+\.61to\+\.66\+\.66, every lower bound above\+\.53\+\.53\)\. The Qwen3\-8B answer judge, which never labels sufficiency, gives the same picture \(Table[2](https://arxiv.org/html/2610.40966#S4.T2)\)\. Since Qwen3\-4B abstains on 99% of turns without documents, as instructed, this control shows that its correct answers come from the retrieved documents, not that it lacks the knowledge\.

Table 2:Share of correct Qwen3\-4B answers \(answered, correctness≥4\\geq 4of 5 by the phi\-4 answer judge\) onnonecontexts that phi\-4 judges sufficient \(false misses\) or insufficient \(true misses\), and on the false\-miss turns without documents\.nn: false misses / true misses\. With the Qwen3\-8B answer judge every cell is within \.04\.
#### Human judgments on MTRAG\.

MTRAG records which passages its annotators saw and rejected for a turn\. Shown one passage at a time, phi\-4 labels 59% of gold passagesyesand only 8% \[6, 11\] of rejected passages; 1 of 109 false misses of the dense retriever rests only on rejected passages \(§[4\.4](https://arxiv.org/html/2610.40966#S4.SS4)\)\.

#### What a false miss contains\.

For each false miss we classify the best evidence document named by the judge by its relation to the turn’s gold\. On both strong retrievers, 86% of false misses are tied to the dialogue\. On Fusion, 41% rest on a*same\-thread negative*, a document mined as a hard negative for the same question, 14% on a near\-duplicate of a gold document \(word 8\-gram containment≥\.8\{\\geq\}\.8\), 19% on the gold of another turn and 13% on another chunk of the gold page\. In the StackExchange domains, where each dialogue has its own pool of mined negatives, such negatives carry the answer in 63–69% of false misses; in the BRIGHT domains the evidence is more often another turn’s gold or a neighbouring chunk\. The blind audit’s own false misses fall into the same classes \(90% and 100% tied to the dialogue\)\. Dialogue\-tied documents make up 72–74% of all documents innonecontexts, so this shows where false misses sit rather than why; discounting every dialogue\-tied false miss, precision still falls from \.89 to \.45\.

### 4\.3RQ3: The label changes conclusions about abstention

Figure 3:AUROC of each reader’sp⁡\(answerable\)p\(\\textsc\{answerable\}\)for separating sufficient from insufficient contexts, under the qrels label \(allvs\.none\) and the two content labels \(yesvs\.no\)\. Gold contexts are all sufficient under the qrels label\. Numbers: content \(phi\-4\) minus qrels, paired over bootstrap resamples; bold if the 95% CI excludes zero\.#### Detector quality is understated\.

Under the phi\-4 label, the AUROC ofp⁡\(answerable\)p\(\\textsc\{answerable\}\)is significantly higher than under the qrels label for every reader at every retriever \(Fig\.[3](https://arxiv.org/html/2610.40966#S4.F3)\)\. For Qwen3\-4B the gap widens with the retriever \(\+\.10, \+\.13, \+\.14, \+\.19\)\. On the hybrid the qrels label puts all three readers near chance \(\.50–\.54\), while under the content label Qwen3\-4B reaches \.72 and Qwen3\-8B \.68\. Restricted to contexts that both labels keep, the Qwen3\-4B gap is still \+\.08 to \+\.16, so most of it comes from the label rather than from which contexts are dropped\.

#### The qrels label hides the 2b/2c contrast\.

The question behind the 2b/2c pairing is whether a reader refuses more when its retrieval failed than when it had the gold\. Under the qrels label this contrast is indistinguishable from zero already on BM25\-hist for Qwen3\-1\.7B and on Fusion for Qwen3\-4B\. The content label keeps it significant for Qwen3\-1\.7B on BM25\-hist and for Qwen3\-4B on Fusion \(\+\.20; not significant with refusals read from the Qwen3\-8B answer judge\) and the hybrid \(\+\.31\); for Qwen3\-8B on Fusion the content contrast is not significant and the qrels contrast is significantly negative\. Compared within turns, the content\-label contrast stays significant on BM25\-hist for all three readers, where the qrels contrast is not; on Fusion and the hybrid it is not significant for any reader \(17–30 insufficient contexts per cell\)\. The qrels contrast turns significantly negative for two readers on Fusion and all three on the hybrid, as if they refused less after a retrieval miss than with the gold of the same turn\.

#### The wrong detector wins\.

Under the qrels label, a retrieval score that never reads the documents \(the top\-1 score, or the top\-1 margin on Fusion\) is the best insufficiency detector at every retriever: on BM25\-hist the top\-1 score reaches \.73 \(\.71 when z\-normalised within domain\) against \.63 for the best reader\. Under the content label, the reader’s ownp⁡\(answerable\)p\(\\textsc\{answerable\}\)is best at every retriever \(Qwen3\-4B \.74 on BM25\-hist; on Fusion and the hybrid its lead is not significant\), and the difference between the domain\-normalised top\-1 score and Qwen3\-4B changes sign on BM25\-turn \(\+\.07 to−\.07\-\.07\) and BM25\-hist \(\+\.11 to−\.17\-\.17\), with all four intervals excluding zero\. This is expected: the qrels label records whether the retriever found the gold, and the retriever’s own confidence predicts that best\.

#### How many conclusions change\.

We enumerate the comparisons an end\-to\-end study would report: refusal rates and AUROC between retrievers and between readers; refusal gaps between insufficient and sufficient contexts and between retrieved insufficient contexts and the gold \(the 2b/2c contrast\); AUROC against chance; and, for each change of retriever, the composition and rate terms of a Kitagawa decomposition of the refusal change[Kitagawa \(1955\)](https://arxiv.org/html/2610.40966#bib.bib31)\. Of these 258 comparisons, 75 change sign or significance when the qrels label is replaced by the phi\-4 label, 13 of them by sign; swapping one content judge for the other changes 25, none by sign\. Of the 7 refusal\-based changes that an answer judge can check, 4 survive with Qwen3\-8B as answer judge and 6 with phi\-4\.

### 4\.4RQ4: Replication on MTRAG

Table 3:MTRAG \(777 answerable and partially answerable turns\): the qrels label against the phi\-4 content label for five first\-stage retrievers, ordered by nDCG@10\. Qs: BM25 over all user questions; Last: BM25 over the last turn; Rewr\.: BM25 over MTRAG’s rewritten question; Dense: Qwen3\-Embedding\-0\.6B over the rewrite; Hybrid: equal\-weight fusion of Rewr\. and Dense\. Bold: content\-label AUROC whose difference from the qrels\-label AUROC has a 95% CI excluding zero\.QsLastRewr\.HybridDensenDCG@10\.18\.22\.25\.40\.42no gold \(%\)7164614037contentno\(%\)4642372018false misses \(%\)2323274038precision of “no gold”\.72\.72\.67\.50\.50Qwen3\-4B AUROC: qrels\.73\.76\.70\.67\.68content\.74\.80\.78\.77\.75MTRAG[Katsis et al\. \(2025\)](https://arxiv.org/html/2610.40966#bib.bib13)builds its qrels differently\. Annotators held each conversation with a live RAG agent backed by an ELSER retriever, checked the passages it retrieved and could search for others; the passages they marked relevant are the qrels\. There are no mined negatives, and the corpora are split into overlapping 512\-token chunks\. We use five first stages on the 777 turns that have qrels; with no training split, all settings were fixed in advance\. phi\-4 labels every top\-5 context as on RECOR\.

The decline replicates on a shorter ladder \(Table[3](https://arxiv.org/html/2610.40966#S4.T3)\): precision falls from \.72 to \.50 \(dense minus BM25 over all questions:−\.22\-\.22\[−\-\.29,−\-\.16\]\), and false misses rise from 23% to 38–40%\. Of the dense retriever’s 109 false misses, 29 cite the gold of another turn of the same conversation, 21 another chunk of a gold passage’s source document, and 56 only passages that the annotators never saw for that turn\. The consequences replicate as well: Qwen3\-4B’s answerability AUROC is understated by the qrels label on the three strongest retrievers \(\+\.07 to \+\.10\), and on the hybrid its within\-turn 2b/2c contrast is significant under the content label but not under the qrels label \(\+\.22 \[\.06, \.38\] on 58 turns against \+\.01 \[−\-\.06, \.09\]; Qwen3\-4B gives no status line on 13–23% of MTRAG contexts, and these are excluded as on RECOR\)\.

## 5Discussion and Recommendations

#### The protocol can give wrong conclusions\.

The reproduction overturns the qrels\-derived sufficiency label\. With a weak retriever most no\-gold contexts are indeed insufficient \(precision \.68 on BM25\-turn\), but already on the official baseline 24–45% of itsnonecontexts contain the answer, depending on the labeller \(precision \.44 under phi\-4, \.62 under the audit, \.69 under Qwen3\-14B\)\. End\-to\-end benchmarks exist to compare systems with different retrievers, and that is exactly where the label fails\. Readers that answer false misses correctly are penalised, and a detector that only tracks the retriever’s confidence wins\. The failure is not specific to RECOR’s mined negatives: on MTRAG, with human qrels and no mined negatives, 38–40% of the strong retrievers’ “insufficient” contexts are sufficient \(precision \.50\)\. We expect the cause to be general: qrels are collected per turn and per chunk around one retrieval setup, while the answer is spread over duplicates, sibling chunks and other turns’ gold\.

#### Recommendations\.

\(1\) Do not compare abstention across retrievers, or select sufficiency detectors, under a qrels\-derived label; if it must be used, report its precision on an audited sample for each retriever\. \(2\) Label sufficiency by content for each retrieved context, with judges from more than one model family, and validate each judge with a negative control first: small open models reproduce the trend but are too lenient for absolute rates\. Human judgments, where they exist, should check the judge \(§[4\.2](https://arxiv.org/html/2610.40966#S4.SS2)\)\. \(3\) Before using mined negatives as negatives, check them for the answer, not only for duplication against gold, and re\-judge the gold of other turns and sibling chunks of gold pages as candidates\. \(4\) Report abstention conditionally on sufficiency and state the retriever\. For RETECO 2c and similar tasks, this means scoring refusals against a content label of each submitted context rather than against the presence of gold\.

#### Limitations\.

Sufficiency on RECOR is labelled by LLMs; the human evidence comes from MTRAG, and the judge\-free copy count covers only part of the false misses\. Item\-level agreement is fair to moderate between the large judges and the audit \(κ\\kappa\.38–\.49\) and only fair for the small judges, so we rely on stage\-level trends\. All labellers see the reference answer, so a shared leniency toward on\-topic documents would inflate the content label on strong retrievers; the negative control, the generation check and MTRAG’s rejected passages bound this, but passages that no annotator saw, where most false misses lie, have no human check\. The query expansions of Fusion and the hybrid were generated by Claude Sonnet 5, of the same family as the auditor\. The readers are three Qwen3 models of 1\.7B–8B parameters, and RECOR dev has 211 dialogues\. We report many comparisons without correction; the counts of changed conclusions indicate scale, not significance\.

## 6Conclusion

We reproduced the qrels label for scoring abstention on RECOR, starting from an exact re\-run of its official baselines\. RQ1: no; the precision of “no gold” falls from \.68 to \.10\. RQ2: labellers from five model families agree on the decline, and Qwen3\-4B answers correctly from false misses\. RQ3: the label understates answerability AUROC, hides the 2b/2c contrast and favours retrieval\-score detectors\. RQ4: the decline replicates on MTRAG\.

## References

- Abdallahet al\.\(2026\)A\. Abdallah, M\. Ali, M\. Abdul\-Mageed, K\. Duh, and A\. JatowtRETECO: reasoning\-oriented retrieval \(SemEval\-2027 task 1\)\.Note:[https://datascienceuibk\.github\.io/RETECO/](https://datascienceuibk.github.io/RETECO/)Shared task website; data at[https://huggingface\.co/datasets/DataScience\-UIBK/RETECO\-SemEval2027](https://huggingface.co/datasets/DataScience-UIBK/RETECO-SemEval2027)\. Accessed 2026\-10\-10Cited by:[§1](https://arxiv.org/html/2610.40966#S1.p1.1)\.
- Abdinet al\.\(2024\)M\. Abdin, J\. Aneja, H\. Behl, S\. Bubeck, R\. Eldan, S\. Gunasekar, M\. Harrison, R\. J\. Hewett, M\. Javaheripi, P\. Kauffmann, J\. R\. Lee, Y\. T\. Lee, Y\. Li, W\. Liu, C\. C\. T\. Mendes, A\. Nguyen, E\. Price, G\. de Rosa, O\. Saarikivi, A\. Salim, S\. Shah, X\. Wang, R\. Ward, Y\. Wu, D\. Yu, C\. Zhang, and Y\. ZhangPhi\-4 technical report\.External Links:2412\.08905,[Link](https://arxiv.org/abs/2412.08905)Cited by:[§3](https://arxiv.org/html/2610.40966#S3.SS0.SSS0.Px4.p1.1)\.
- Aliet al\.\(2026\)M\. Ali, A\. Abdallah, A\. Agarwal, H\. L\. Patel, and A\. JatowtRECOR: reasoning\-focused multi\-turn conversational retrieval benchmark\.InFindings of the Association for Computational Linguistics: ACL 2026,San Diego, California, United States,pp\. 2688–2723\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.129),[Link](https://aclanthology.org/2026.findings-acl.129/)Cited by:[§1](https://arxiv.org/html/2610.40966#S1.p3.1),[§3](https://arxiv.org/html/2610.40966#S3.SS0.SSS0.Px1.p1.1)\.
- Arabzadehet al\.\(2022\)N\. Arabzadeh, A\. Vtyurina, X\. Yan, and C\. L\. A\. ClarkeShallow pooling for sparse labels\.Information Retrieval Journal25\(4\),pp\. 365–385\.External Links:[Document](https://dx.doi.org/10.1007/s10791-022-09411-0)Cited by:[§1](https://arxiv.org/html/2610.40966#S1.p3.1),[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px3.p1.1)\.
- Asai and Choi \(2021\)A\. Asai and E\. ChoiChallenges in information\-seeking QA: unanswerable questions and paragraph retrieval\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),Online,pp\. 1492–1504\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.118),[Link](https://aclanthology.org/2021.acl-long.118/)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px2.p1.1)\.
- Athanasiouet al\.\(2026\)D\. Athanasiou, M\. Lymperaiou, G\. Filandrianos, A\. Voulodimos, and G\. StamouAILS\-NTUA at SemEval\-2026 task 8: query diversity via nested reciprocal rank fusion and evidence\-guided agentic generation for multi\-turn RAG\.InProceedings of the 20th International Workshop on Semantic Evaluation \(SemEval\-2026\),pp\. 1340–1365\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.semeval-1.175)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px2.p1.1)\.
- Bondarenkoet al\.\(2026\)I\. Bondarenko, R\. Derunets, O\. Sedukhin, M\. Komarov, I\. Chernov, and M\. KulakovRaguTeam at SemEval\-2026 task 8: meno and Friends in a judge\-orchestrated LLM ensemble for faithful multi\-turn response generation\.InProceedings of the 20th International Workshop on Semantic Evaluation \(SemEval\-2026\),San Diego, California, USA,pp\. 1678–1694\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.semeval-1.216),[Link](https://aclanthology.org/2026.semeval-1.216/)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px2.p1.1)\.
- Buckley and Voorhees \(2004\)C\. Buckley and E\. M\. VoorheesRetrieval evaluation with incomplete information\.InProceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 25–32\.External Links:[Document](https://dx.doi.org/10.1145/1008992.1009000)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px3.p1.1)\.
- Carmelet al\.\(2025\)D\. Carmel, S\. Filice, G\. Horowitz, Y\. Maarek, A\. Shtoff, O\. Somekh, and R\. TavoryLiveRAG: a diverse Q&A dataset with varying difficulty level for RAG evaluation\.Note:arXiv:2511\.14531External Links:2511\.14531Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px2.p1.1)\.
- Chandradevanet al\.\(2026\)R\. Chandradevan, S\. Ghosh, and V\. SelvendranSearch engines never say no: how frozen agents react when the retrieval tool refuses\.External Links:2610\.05348,[Link](https://arxiv.org/abs/2610.05348)Cited by:[§1](https://arxiv.org/html/2610.40966#S1.p2.1)\.
- Chenet al\.\(2026a\)J\. Chen, J\. Lan, C\. Li, D\. Lian, and Z\. LiuReasonEmbed: enhanced text embeddings for reasoning\-intensive document retrieval\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),San Diego, California, United States,pp\. 1203–1221\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.54),[Link](https://aclanthology.org/2026.acl-long.54/)Cited by:[§3](https://arxiv.org/html/2610.40966#S3.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2024\)J\. Chen, H\. Lin, X\. Han, and L\. SunBenchmarking large language models in retrieval\-augmented generation\.Proceedings of the AAAI Conference on Artificial Intelligence38\(16\),pp\. 17754–17762\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v38i16.29728)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2026b\)S\. Chen, P\. Hua, and Y\. XiaoRelevance is not sufficient evidence: detecting evidence gaps before generation in RAG\.External Links:2609\.37469,[Link](https://arxiv.org/abs/2609.37469)Cited by:[§1](https://arxiv.org/html/2610.40966#S1.p2.1),[§3](https://arxiv.org/html/2610.40966#S3.SS0.SSS0.Px6.p1.1)\.
- Clarke and Dietz \(2025\)C\. L\. A\. Clarke and L\. DietzLLM\-based relevance assessment still can’t replace human relevance assessment\.InThe 11th International Workshop on Evaluating Information Access \(EVIA 2025\),External Links:[Document](https://dx.doi.org/10.20736/0002002105),[Link](https://arxiv.org/abs/2412.17156)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px3.p1.1)\.
- Cohen \(1960\)J\. CohenA coefficient of agreement for nominal scales\.Educational and Psychological Measurement20\(1\),pp\. 37–46\.External Links:[Document](https://dx.doi.org/10.1177/001316446002000104)Cited by:[§4\.2](https://arxiv.org/html/2610.40966#S4.SS2.SSS0.Px2.p1.1)\.
- Dutraet al\.\(2026\)G\. Dutra, A\. F\. Caraíba, N\. F\. Da Silva, P\. Dos Santos, D\. S\. Fernandes, and S\. S\. De OliveiraNLP\-CEIA\-UFG at SemEval\-2026 task 8: iterative retrieval with notes\-guided query refinement for multi\-turn RAG\.InProceedings of the 20th International Workshop on Semantic Evaluation \(SemEval\-2026\),pp\. 2771–2778\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.semeval-1.349)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px2.p1.1)\.
- Gaoet al\.\(2023\)L\. Gao, X\. Ma, J\. Lin, and J\. CallanPrecise zero\-shot dense retrieval without relevance labels\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Toronto, Canada,pp\. 1762–1777\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.99),[Link](https://aclanthology.org/2023.acl-long.99/)Cited by:[§3](https://arxiv.org/html/2610.40966#S3.SS0.SSS0.Px2.p1.1)\.
- Gwet \(2008\)K\. L\. GwetComputing inter\-rater reliability and its variance in the presence of high agreement\.British Journal of Mathematical and Statistical Psychology61\(1\),pp\. 29–48\.External Links:[Document](https://dx.doi.org/10.1348/000711006X126600)Cited by:[§4\.2](https://arxiv.org/html/2610.40966#S4.SS2.SSS0.Px2.p1.1)\.
- Jorenet al\.\(2025\)H\. Joren, J\. Zhang, C\. Ferng, D\. Juan, A\. Taly, and C\. RashtchianSufficient context: a new lens on retrieval augmented generation systems\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=Jjr2Odj8DJ)Cited by:[§1](https://arxiv.org/html/2610.40966#S1.p1.1),[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px1.p1.1)\.
- Katsiset al\.\(2025\)Y\. Katsis, S\. Rosenthal, K\. Fadnis, C\. Gunasekara, Y\. Lee, L\. Popa, V\. Shah, H\. Zhu, D\. Contractor, and M\. DanilevskyMTRAG: a multi\-turn conversational benchmark for evaluating retrieval\-augmented generation systems\.Transactions of the Association for Computational Linguistics13,pp\. 784–808\.External Links:[Document](https://dx.doi.org/10.1162/tacl.a.19),[Link](https://aclanthology.org/2025.tacl-1.36/)Cited by:[§1](https://arxiv.org/html/2610.40966#S1.p1.1),[§1](https://arxiv.org/html/2610.40966#S1.p2.1),[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px2.p1.1),[§4\.4](https://arxiv.org/html/2610.40966#S4.SS4.p1.1)\.
- Kirichenkoet al\.\(2025\)P\. Kirichenko, M\. Ibrahim, K\. Chaudhuri, and S\. J\. BellAbstentionBench: reasoning LLMs fail on unanswerable questions\.InAdvances in Neural Information Processing Systems 38 \(NeurIPS 2025\), Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=OkHC30LLpO)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px1.p1.1)\.
- Kitagawa \(1955\)E\. M\. KitagawaComponents of a difference between two rates\.Journal of the American Statistical Association50\(272\),pp\. 1168–1194\.External Links:[Document](https://dx.doi.org/10.1080/01621459.1955.10501299)Cited by:[§4\.3](https://arxiv.org/html/2610.40966#S4.SS3.SSS0.Px4.p1.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with PagedAttention\.InProceedings of the 29th Symposium on Operating Systems Principles \(SOSP ’23\),pp\. 611–626\.External Links:[Document](https://dx.doi.org/10.1145/3600006.3613165)Cited by:[§3](https://arxiv.org/html/2610.40966#S3.SS0.SSS0.Px3.p1.1)\.
- Moreiraet al\.\(2024\)G\. d\. S\. P\. Moreira, R\. Osmulski, M\. Xu, R\. Ak, B\. Schifferer, and E\. OldridgeNV\-Retriever: improving text embedding models with effective hard\-negative mining\.Note:arXiv:2407\.15831External Links:2407\.15831Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px3.p1.1)\.
- Penget al\.\(2025\)X\. Peng, P\. K\. Choubey, C\. Xiong, and C\. WuUnanswerability evaluation for retrieval augmented generation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 8452–8472\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.415),[Link](https://aclanthology.org/2025.acl-long.415/)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px1.p1.1)\.
- Quet al\.\(2021\)Y\. Qu, Y\. Ding, J\. Liu, K\. Liu, R\. Ren, W\. X\. Zhao, D\. Dong, H\. Wu, and H\. WangRocketQA: an optimized training approach to dense passage retrieval for open\-domain question answering\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,Online,pp\. 5835–5847\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.466),[Link](https://aclanthology.org/2021.naacl-main.466/)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px3.p1.1)\.
- Rajpurkaret al\.\(2018\)P\. Rajpurkar, R\. Jia, and P\. LiangKnow what you don’t know: unanswerable questions for SQuAD\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),Melbourne, Australia,pp\. 784–789\.External Links:[Document](https://dx.doi.org/10.18653/v1/P18-2124),[Link](https://aclanthology.org/P18-2124/)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px1.p1.1)\.
- Rosenthalet al\.\(2026a\)S\. Rosenthal, Y\. Katsis, V\. Shah, L\. He, L\. Popa, and M\. DanilevskyMTRAG\-UN: a benchmark for open challenges in multi\-turn RAG conversations\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 10363–10369\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.503)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px2.p1.1)\.
- Rosenthalet al\.\(2026b\)S\. Rosenthal, V\. Shah, Y\. Katsis, and M\. DanilevskySemEval\-2026 task 8: MTRAGEval: evaluating multi\-turn RAG conversations\.InProceedings of the 20th International Workshop on Semantic Evaluation \(SemEval\-2026\),San Diego, California, USA,pp\. 3673–3690\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.semeval-1.447),[Link](https://aclanthology.org/2026.semeval-1.447/)Cited by:[§1](https://arxiv.org/html/2610.40966#S1.p1.1),[§1](https://arxiv.org/html/2610.40966#S1.p2.1),[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px2.p1.1)\.
- Rosenthalet al\.\(2025\)S\. Rosenthal, A\. Sil, R\. Florian, and S\. RoukosCLAPNQ: cohesive long\-form answers from passages in natural questions for RAG systems\.Transactions of the Association for Computational Linguistics13,pp\. 53–72\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00729)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px2.p1.1)\.
- Sharifymoghaddamet al\.\(2026\)S\. Sharifymoghaddam, Y\. Ge, R\. Vasudeva, and J\. LinLighting the way for BRIGHT: reproducible baselines with Anserini, Pyserini, and RankLLM\.InProceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 3059–3069\.External Links:[Document](https://dx.doi.org/10.1145/3805712.3808570)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px3.p1.1)\.
- Songet al\.\(2025\)M\. Song, S\. H\. Sim, R\. Bhardwaj, H\. L\. Chieu, N\. Majumder, and S\. PoriaMeasuring and enhancing trustworthiness of LLMs in RAG through grounded attributions and learning to refuse\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=Iyrtb9EJBp)Cited by:[§1](https://arxiv.org/html/2610.40966#S1.p1.1),[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px1.p1.1)\.
- Suet al\.\(2025\)H\. Su, H\. Yen, M\. Xia, W\. Shi, N\. Muennighoff, H\. Wang, H\. Liu, Q\. Shi, Z\. S\. Siegel, M\. Tang, R\. Sun, J\. Yoon, S\. O\. Arik, D\. Chen, and T\. YuBRIGHT: a realistic and challenging benchmark for reasoning\-intensive retrieval\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=ykuc5q381b)Cited by:[§3](https://arxiv.org/html/2610.40966#S3.SS0.SSS0.Px1.p1.1)\.
- Thakuret al\.\(2021\)N\. Thakur, N\. Reimers, A\. Rücklé, A\. Srivastava, and I\. GurevychBEIR: a heterogeneous benchmark for zero\-shot evaluation of information retrieval models\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks,Vol\.1\.External Links:[Link](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/65b9eea6e1cc6bb9f0cd2a47751a186f-Abstract-round2.html)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px3.p1.1)\.
- Thakuret al\.\(2025\)N\. Thakur, C\. Zhang, X\. Ma, and J\. LinHard negatives, hard lessons: revisiting training data quality for robust information retrieval with LLMs\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 9064–9083\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.481)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px3.p1.1)\.
- Thomaset al\.\(2024\)P\. Thomas, S\. Spielman, N\. Craswell, and B\. MitraLarge language models can accurately predict searcher preferences\.InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 1930–1940\.External Links:[Document](https://dx.doi.org/10.1145/3626772.3657707)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px3.p1.1)\.
- Trivediet al\.\(2022\)H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. SabharwalMuSiQue: multihop questions via single\-hop question composition\.Transactions of the Association for Computational Linguistics10,pp\. 539–554\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00475),[Link](https://aclanthology.org/2022.tacl-1.31/)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px1.p1.1)\.
- Upadhyayet al\.\(2024\)S\. Upadhyay, R\. Pradeep, N\. Thakur, N\. Craswell, and J\. LinUMBRELA: UMbrela is the \(open\-source reproduction of the\) Bing RELevance assessor\.External Links:2406\.06519,[Link](https://arxiv.org/abs/2406.06519)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px3.p1.1)\.
- von Dänikenet al\.\(2026\)P\. von Däniken, F\. M\. Saaro, M\. Cieliebak, and J\. DeriuThe RAT: a unified Bayesian model for RAG evaluation\.External Links:2608\.24753,[Link](https://arxiv.org/abs/2608.24753)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2023\)L\. Wang, N\. Yang, and F\. WeiQuery2doc: query expansion with large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 9414–9423\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.585),[Link](https://aclanthology.org/2023.emnlp-main.585/)Cited by:[§3](https://arxiv.org/html/2610.40966#S3.SS0.SSS0.Px2.p1.1)\.
- Welleret al\.\(2025\)O\. Weller, K\. Ricci, E\. Yang, A\. Yates, D\. Lawrie, and B\. Van DurmeRank1: test\-time compute for reranking in information retrieval\.InSecond Conference on Language Modeling \(COLM\),External Links:[Link](https://openreview.net/forum?id=Pg0PAvbhGv)Cited by:[§1](https://arxiv.org/html/2610.40966#S1.p3.1),[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px3.p1.1)\.
- Wenet al\.\(2025\)B\. Wen, J\. Yao, S\. Feng, C\. Xu, Y\. Tsvetkov, B\. Howe, and L\. L\. WangKnow your limits: a survey of abstention in large language models\.Transactions of the Association for Computational Linguistics13,pp\. 529–556\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00754),[Link](https://aclanthology.org/2025.tacl-1.26/)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§3](https://arxiv.org/html/2610.40966#S3.SS0.SSS0.Px3.p1.1),[§3](https://arxiv.org/html/2610.40966#S3.SS0.SSS0.Px4.p1.1)\.
- Yanget al\.\(2024\)X\. Yang, K\. Sun, H\. Xin, Y\. Sun, N\. Bhalla, X\. Chen, S\. Choudhary, R\. D\. Gui, Z\. W\. Jiang, Z\. Jiang, L\. Kong, B\. Moran, J\. Wang, Y\. E\. Xu, A\. Yan, C\. Yang, E\. Yuan, H\. Zha, N\. Tang, L\. Chen, N\. Scheffer, Y\. Liu, N\. Shah, R\. Wanga, A\. Kumar, W\. Yih, and X\. L\. DongCRAG – comprehensive RAG benchmark\.InAdvances in Neural Information Processing Systems 37 \(Datasets and Benchmarks Track\),pp\. 10470–10490\.Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px2.p1.1)\.
- Yilmaz and Aslam \(2006\)E\. Yilmaz and J\. A\. AslamEstimating average precision with incomplete and imperfect judgments\.InProceedings of the 15th ACM International Conference on Information and Knowledge Management \(CIKM ’06\),Arlington, Virginia, USA,pp\. 102–111\.External Links:[Document](https://dx.doi.org/10.1145/1183614.1183633)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px3.p1.1)\.
- Zobel \(1998\)J\. ZobelHow reliable are the results of large\-scale information retrieval experiments?\.InProceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 307–314\.External Links:[Document](https://dx.doi.org/10.1145/290941.291014)Cited by:[§2](https://arxiv.org/html/2610.40966#S2.SS0.SSS0.Px3.p1.1)\.