MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks

# MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks
Source: [https://arxiv.org/html/2605.20729](https://arxiv.org/html/2605.20729)
Junhao Ruan1,2,Abudukeyumu Abudula1,Bei Li2,Yongjing Yin2, Xinyu Liu1,Kechen Jiao4,Xin Chen2,Jingang Wang2, Xunliang Cai2,Tong Xiao1,322footnotemark:2,Jingbo Zhu1,3

1School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China 2Meituan Inc\.,3NiuTrans Research, Shenyang, China 4Tsinghua University, Beijing, China rangehow@outlook\.com

###### Abstract

Accurate evaluation of conversational retrieval is pivotal for advancing Retrieval\-Augmented Generation \(RAG\) systems\. However, existing conversational retrieval benchmarks suffer from costly, sparse human annotation or rigid, unnatural automated heuristics\. To address these challenges, we introduceMTR\-Suite, a unified framework for auditing, synthesizing, and benchmarking retrieval\. It features: \(1\)MTR\-Eval, an LLM\-based auditor quantifying alignment gaps in previous benchmarks; \(2\)MTR\-Pipeline, a multi\-agent system using greedy traversal clustering to generate high\-fidelity dialogues at 1/400th human cost; and \(3\)MTR\-Bench, a rigorous general\-domain benchmark\.MTR\-Benchmimics production\-style challenges \(hard topic switching, verbosity\), offering superior discriminative power\. We make our code and data publicly available to facilitate future research\.111[https://github\.com/rangehow/mtr\-suite](https://github.com/rangehow/mtr-suite)

MTR\-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks

Junhao Ruan1,2††thanks:Work done during internship at Meituan\., Abudukeyumu Abudula1, Bei Li2††thanks:Corresponding authors\., Yongjing Yin2,Xinyu Liu1,Kechen Jiao4,Xin Chen2,Jingang Wang2,Xunliang Cai2,Tong Xiao1,322footnotemark:2,Jingbo Zhu1,31School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China2Meituan Inc\.,3NiuTrans Research, Shenyang, China4Tsinghua University, Beijing, Chinarangehow@outlook\.com

![Refer to caption](https://arxiv.org/html/2605.20729v1/figure/intro.png)Figure 1:Top: Symbol definitions\. Second Row: Illustration of the humancognitive boundary\. Third Row: TheMTR\-Evalauditing process\. Bottom: TheMTR\-Pipeline, which synthesize high\-quality benchmarks\.\(a\)

Evidence CompletenessQuery\-EvidenceAnswer FaithfulAnswer Quality2\.02\.03\.03\.04\.04\.05\.05\.03\.73\.73\.93\.94\.14\.14\.04\.03\.83\.83\.73\.73\.43\.42\.42\.43\.33\.33\.53\.53\.93\.93\.13\.12\.72\.73\.23\.23\.63\.63\.43\.42\.12\.13\.33\.33\.13\.13\.43\.42\.52\.52\.22\.22\.22\.23\.63\.64\.74\.74\.84\.84\.84\.84\.94\.9QReCCDoc2DialQuACTopiOCQAInscitCORALMTR\-Bench

\(b\)

QReCCQuACDoc2DialTopiOCQAMTR\-Bench25254040555570708585100100Large Performance GapHigh Discriminative PowerPerformancegte\-Qwen2\-7Bstella\_en\_400M\_v5gte\-ModernBert\-basebge\-Large\-en\-v1\.5Dragon\-ChatQA

Figure 2:\(a\) Average quantitative analysis results of multiple powerful open\-source LLMs on different datasets using our proposedMTR\-Evalmetrics\. Detailed score heatmaps for each model across all benchmarks are provided in Appendix[A\.1](https://arxiv.org/html/2605.20729#A1.SS1)\. \(b\) Recall@20 scores of popular open\-source retrievers on various benchmarks\. Note the larger performance gap inMTR\-Bench, highlighting its discriminative power compared to previous benchmarks\.\\phantomsubcaption

\\phantomsubcaption

## 1Introduction

Retrieval\-Augmented Generation \(RAG\) has emerged as the standard paradigm for grounding Large Language Models \(LLMs\) in verifiable, external knowledge\(Lewis and others,[2020](https://arxiv.org/html/2605.20729#bib.bib73); OpenAI,[2023](https://arxiv.org/html/2605.20729#bib.bib79); Gemini and others,[2024](https://arxiv.org/html/2605.20729#bib.bib18); DeepSeek\-AI and others,[2024](https://arxiv.org/html/2605.20729#bib.bib80)\)\. By mitigating the hallucinations inherent to static parametric memory\(Xuet al\.,[2025](https://arxiv.org/html/2605.20729#bib.bib69)\), RAG extends the capabilities of models further amplified by tool use\(Schicket al\.,[2023](https://arxiv.org/html/2605.20729#bib.bib27)\)and test\-time scaling\(OpenAI and others,[2024](https://arxiv.org/html/2605.20729#bib.bib72); DeepSeek\-AI and others,[2025](https://arxiv.org/html/2605.20729#bib.bib71)\)\. However, the efficacy of any RAG system remains bounded by the performance of its retrieval module\. If the retriever fails to locate the precise conversational context, the generation layer is rendered ineffective\.

Consequently, evaluating the conversational retrieval stage is critical\. Traditional manual annotation faces a fundamentalcognitive boundary\(Zobel,[1998](https://arxiv.org/html/2605.20729#bib.bib102); Buckley and Voorhees,[2004](https://arxiv.org/html/2605.20729#bib.bib103)\)\. Annotators typically operate with a “local view”, which formulating queries based solely on the specific document they are reading\. They lack the global perspective to know if other documents in the corpus could also answer the query \(potentially better\)\. This leads toannotation sparsity, where valid retrievals are penalized as false negatives\. Furthermore, strict privacy constraints often make manual annotation prohibitive for high\-value proprietary domains \(e\.g\., finance, legal\)\.

Ideally, automated synthesis should overcome these human limitations\. However, prior attempts like CORAL\(Chenget al\.,[2024](https://arxiv.org/html/2605.20729#bib.bib67)\)have inadvertently inherited similar challenges\. By relying on static heuristics, such as directly converting Wikipedia section headers into questions, these methods operate within a methodological boundary that mirrors the human local view\. They assume that a query generated from a specific document structure is uniquely aligned with that document\. As our auditing suggests \(Section[3](https://arxiv.org/html/2605.20729#S3)\), without global validation, this approach can result in alignment issues: generated queries may be ambiguous or better answered by unannotated documents elsewhere in the corpus\. Thus, existing automation trades the high cost of humans for efficiency, but the challenge of annotation sparsity remains to be fully addressed\.

To bridge this gap, we argue that high\-fidelity benchmarking requires a transition from local heuristics to Global\-Aware Automated Annotation\. We introduceMTR\-Suite\(Figure[1](https://arxiv.org/html/2605.20729#S0.F1)\), a unified framework designed to audit, synthesize, and benchmark conversational retrieval systems\.MTR\-Suiteprovides a holistic solution through three integrated contributions:

- •Diagnostic Auditing \(MTR\-Eval\):We propose the first fine\-grained evaluation method designed to scientifically quantify the quality of retrieval benchmarks\. Our experiments reveal that while automated baselines suffer from linguistic degradation, even human\-annotated datasets exhibit significant sparsity\-induced noise\.MTR\-Evalcorrelates highly with human judgments, offering a reliable metric for dataset integrity \(Figure[2](https://arxiv.org/html/2605.20729#S0.F2)\)\.
- •Multi\-Agent Synthesis \(MTR\-Pipeline\):We introduce a fully automated framework for constructing multi\-turn retrieval datasets\. Unlike rule\-based predecessors, our pipeline utilizes a sophisticated multi\-agent architecture driven by a greedy traversal clustering algorithm\. This approach ensures query uniqueness and linguistic naturalness, achieving annotation quality that surpasses human standards at a fraction of the cost \(1/4001/400th\)\.
- •A Rigorous Benchmark \(MTR\-Bench\):We open\-sourceMTR\-Bench, a general\-domain dataset designed to stress\-test modern retrievers\. Distinguished by realistic conversational phenomena often ignored by previous datasets, including hard topic switching, long\-context ambiguity, and production\-style verbosity, it provides significantly higher discriminative power for evaluating mainstream retrieval modules \(Figure[2](https://arxiv.org/html/2605.20729#S0.F2)\)\.

## 2Related Work

The evolution of conversational search benchmarks reflects a continuous balancing act between annotation quality, scalability, and cost\. We categorize prior efforts into three main paradigms\.

### 2\.1Manual and Hybrid Annotation

Foundational datasets like CoQA\(Reddyet al\.,[2019](https://arxiv.org/html/2605.20729#bib.bib47)\)and QuAC\(Choiet al\.,[2018](https://arxiv.org/html/2605.20729#bib.bib61)\)established the standard for multi\-turn interactions, while subsequent works targeted specific domains, such as goal\-oriented dialogues in Doc2Dial\(Fenget al\.,[2020](https://arxiv.org/html/2605.20729#bib.bib59)\)and topic shifts in TopiocQA\(Adlakhaet al\.,[2022](https://arxiv.org/html/2605.20729#bib.bib94)\)\. To mitigate annotation costs, hybrid semi\-synthetic approaches like TREC CAsT\(Daltonet al\.,[2020](https://arxiv.org/html/2605.20729#bib.bib58)\)and QReCC\(Ananthaet al\.,[2021](https://arxiv.org/html/2605.20729#bib.bib85)\)combine human rewrites with machine retrieval\. While improving efficiency, they inherit the fundamental limitations of human oversight and struggle to scale effectively to private, large\-scale corpora\. Despite their status as gold standards, these datasets suffer from the human cognitive limitations discussed in introduction\. Specifically, the lack of global corpus visibility leads to significant false negatives \(retrievable evidence labeled as irrelevant\) and hallucinated queries\. As demonstrated in our case studies \(see Appendix[A\.3](https://arxiv.org/html/2605.20729#A1.SS3)\), even these manually curated datasets exhibit considerable noise and sparsity\.

### 2\.2Automated Benchmark Synthesis

The emergence of LLMs has shifted focus toward fully automated construction\. Notable examples like CORAL\(Chenget al\.,[2024](https://arxiv.org/html/2605.20729#bib.bib67)\)synthesize dialogues by leveraging the hierarchical structure of Wikipedia \(e\.g\., turning section headings into questions\)\. However, reliance on static document structures may restrict conversational naturalness, resulting in dialogues that mirror the source text’s organization rather than dynamic user intent\.Leeet al\.\([2024](https://arxiv.org/html/2605.20729#bib.bib105)\)introduced a multi\-document synthesis pipeline\. Their reliance on rigid rules to enforce specific question types can compromise conversational flow, leading to unnatural transitions such as illogical counter\-arguments to correct responses\. The unavailability of this dataset also precludes assessment of its quality\.

## 3MTR\-Eval: Auditing Benchmark Quality

Formally, a multi\-turn conversational retrieval benchmark consists of a document corpus𝒟=\{d1,d2,…,dN\}\\mathcal\{D\}=\\\{d\_\{1\},d\_\{2\},\\dots,d\_\{N\}\\\}, where eachddrepresents an individual document \(or passage\), and a set of evaluation instances\. Each instance corresponds to a conversational turnii, represented as a tuple\(Hi,qi,Gi\)\(H\_\{i\},q\_\{i\},G\_\{i\}\)\. Here,qiq\_\{i\}is the user’s current query, andHi=\{\(q1,a1\),…,\(qi−1,ai−1\)\}H\_\{i\}=\\\{\(q\_\{1\},a\_\{1\}\),\\dots,\(q\_\{i\-1\},a\_\{i\-1\}\)\\\}denotes the conversation history\. The setGi⊆𝒟G\_\{i\}\\subseteq\\mathcal\{D\}contains the ground\-truth documents annotated as relevant for answeringqiq\_\{i\}\.

### 3\.1The Quality Gap: Annotated vs\. True Evidence

The fundamental challenge in benchmarking is the discrepancy between theannotatedground\-truth setGiG\_\{i\}and thetrueset of all supporting documentsG^i\\hat\{G\}\_\{i\}existing in the corpus𝒟\\mathcal\{D\}\. A high\-quality benchmark must minimize two specific error types:

##### Annotation Noise:

WhenGi∖G^i≠∅G\_\{i\}\\setminus\\hat\{G\}\_\{i\}\\neq\\emptyset, the benchmark includes documentsddin the gold set that are actually irrelevant\. This leads to model overestimation\.

##### Annotation Sparsity :

WhenG^i∖Gi≠∅\\hat\{G\}\_\{i\}\\setminus G\_\{i\}\\neq\\emptyset, valid supporting documentsddexist in the corpus but are missing from the annotation\. This causes valid retrievals to be penalized as false negatives\.

### 3\.2The Evaluation Metrics

To quantify these discrepancies without requiring exhaustive manual re\-annotation,MTR\-Evalemploys an heterogeneous LLM\-as\-a\-Judge approach across four fine\-grained dimensions:

##### Query\-Evidence Alignment:

This metric assesses whether a specific annotated documentd∈Gid\\in G\_\{i\}actually contains the answer toqiq\_\{i\}\. The LLM is presented with the query and the gold document; ifddfails to support the query, it indicatesAnnotation Noise, meaning the benchmark contains false positives\.

##### Evidence Completeness:

MeasuringAnnotation Sparsityis difficult because enumerating all relevant documents in a large corpus is intractable\. We propose a proxy task: Discriminability Testing\. We present the LLM with a candidate pool containing the gold documentdg​o​l​d∈Gid\_\{gold\}\\in G\_\{i\}and several retrieved hard negatives\. The LLM is asked to identify the most relevant document\. If the LLM consistently selects a non\-gold documentdo​t​h​e​r∉Gid\_\{other\}\\notin G\_\{i\}as beingmorerelevant thandg​o​l​dd\_\{gold\}, it strongly suggests the existence of valid evidence that was missed during annotation\. This metric penalizes benchmarks where the “gold” document is not uniquely or clearly the best match\.

##### Answer\-Evidence Faithfulness:

This evaluates hallucinations during annotation\. We verify if the answeraia\_\{i\}is fully grounded in its corresponding document setGiG\_\{i\}\.

##### Answer Quality:

Independent of evidence, this metric evaluates the linguistic quality of the responseaia\_\{i\}\(e\.g\., coherence, helpfulness forqiq\_\{i\}\)\.

Collectively, these metrics act as a reliability coefficient for the dataset\. A critical implication of our methodology is that achieving high performance with a retriever is only significant on benchmarks that possess highMTR\-Evalscores\. Conversely, high scores \(e\.g\., Recall or mrr\) on benchmarks with lowMTR\-Evalratings hold limited value, as the results are likely conflated by annotation noise and sparsity\.

### 3\.3Validation

##### Verification of Missing Evidence\.

To verify the detected sparsity, we manually examined 300 sampled cases where the LLM identified a document as relevant despite its absence from the ground truth\. Human review confirmed that 98% of these were indeed valid gold\(under 0\.92 Fleiss’ Kappa score\) documents sufficient to fully answer the user’s query, validating the existence of the cognitive boundary\. More cases are available in appendix[A\.3](https://arxiv.org/html/2605.20729#A1.SS3)\.

##### Human Correlations\.

We conducted a parallel human annotation study\. Critically, our method minimizes biases through a multi\-LLM ensemble \(reducing self\-preference bias\) and a pointwise scoring design \(eliminating position bias\)\. In multi\-document scenarios like Discriminability Testing, document positions are further randomized\. These design choices resulted in automated scores that align strongly with human judgments, achieving Pearson correlations of0\.820\.82for Query\-Evidence Relevance,0\.890\.89for Answer Faithfulness, and0\.740\.74for Answer Quality \(allp≪0\.001p\\ll 0\.001\)\. This high alignment confirms that MTR\-EVAL serves as a reliable proxy for human perception, decoupled from specific model preferences\. Further details are provided in Appendix[A\.4](https://arxiv.org/html/2605.20729#A1.SS4)\.

## 4MTR\-Pipeline

To generate high\-quality data at scale, we proposeMTR\-Pipeline\. We decompose the complex task of benchmark construction into three streamlined stages: Curation, Clustering, and Multi Agent Generation\. For comprehensive engineering implementation details regarding recursive chunking guidelines, clustering case study, agent model selection, and cost estimation breakdowns, please refer to Appendix[A\.5](https://arxiv.org/html/2605.20729#A1.SS5)\.

### 4\.1Knowledge Base Curation

The quality of a benchmark is strictly bounded by its source data\. To ensure only high\-quality segments serve as reference documents, we process the raw corpus through a streamlined pipeline\. We first remove non\-textual elements and apply recursive chunking to optimize context window usage, followed by MinHash\-LSHBroder \([1997](https://arxiv.org/html/2605.20729#bib.bib104)\)to eliminate near\-duplicate chunks\. Finally, we employ a hybrid quality filter that utilizes an NVIDIA quality classifier for fluency and the FineWeb\-EDU scorer\(Penedoet al\.,[2024](https://arxiv.org/html/2605.20729#bib.bib74)\)for educational value\. We retain only those chunks that exhibit high information density and score as “High Quality\.”

### 4\.2Greedy Clustering: Scalable Semantic Trajectories

To simulate natural topic\-switching behaviors, such as a user following a chain of hyperlinks, the pipeline must feed the LLM sequences of semantically related documents\. However, conventional clustering algorithms like K\-means or DBSCAN are ill\-suited for this task, as they produce clusters of uncontrollable sizes that often exceed the LLM’s context window\. Furthermore, a naive approach of selecting neighbors for each document based on a similarity threshold is also problematic: it leads to high document overlap across different clusters, causing the LLM to encounter the same content repeatedly during training and evaluation, which introduces significant evaluation bias\.

To address these limitations, we propose a Greedy Traversal Clustering strategy\. While greedy nearest\-neighbor search is a classical heuristic for the Traveling Salesperson Problem \(TSP\), its application as a clustering mechanism for “soft topic switching” is highly novel\. Instead of static grouping, we repurpose this traditional logic to construct a single, continuous “semantic path” by iteratively selecting the nearest unvisited neighbor from a random starting node\. This path is then segmented everykknodes\.

This approach offers three distinct advantages: \(1\) it guarantees a fixed cluster sizekkto maximize context utilization; \(2\) by ensuring each document is visited exactly once, it eliminates the redundancy and bias inherent in threshold\-based methods; and \(3\) it creates a smooth semantic gradient that naturally mimics a user’s browsing trajectory, providing a more realistic foundation for simulating fluid topic transitions than any conventional clustering method\.

### 4\.3Multi\-Agent Dialogue Generation

To simulate realistic conversational dynamics, we employ a three\-agent system\. Decomposing the generation process into specific roles enhances instruction adherence and data quality:

##### The Questioner \(User Simulator\):

Formulates queries based on the document cluster and conversation history\. It decides when to switch topics or drill down into specific details\.

##### The Responder \(RAG Simulator\):

Generates answers strictly grounded in the designated gold document\. This agent mimics the behavior of an ideal RAG system\.

##### The Polisher \(Refiner\):

Post\-processes the dialogue to enhance linguistic naturalness, injecting conversational phenomena such as coreference \(e\.g\., “What abouthim?”\) and ellipsis to mimic human speech patterns\.

##### Scalability and Cost Efficiency\.

Beyond linguistic quality, a critical advantage of our automated framework is its economic viability\. Based on current API pricing \(e\.g\., DeepSeek\-V3\.2\), we estimate the average cost per synthesized dialogue to be approximately $0\.005\. This represents a dramatic reduction \(roughly 1/400\) compared to the $1\.50–$2\.00 per dialogue reported in crowd sourced benchmarks like Doc2Dial\(Fenget al\.,[2020](https://arxiv.org/html/2605.20729#bib.bib59)\)\. This efficiency enables the scalable construction of large\-scale, domain\-specific datasets\.

Table 1:Comparison ofMTR\-Benchwith previous benchmarks\. Settings that more closely align with practical application scenarios are highlighted inbold\. The abbreviations are as follows: KC \(Knowledge Cutoff\), RL \(Response Length in tokens\), and CT \(Conversation Turns\)\. The design choices for items marked with†\\daggerare based on an analysis of large\-scale, real user\-agent dialogues from ShareGPT\. The choice for the item marked with‡\\ddaggeris informed by prior research from LlamaIndex\. For KC, dates marked with ‘≤\\leq‘ were not explicitly specified in the original papers and are our estimations based on the paper release dates or their documented data sources\.Table 2:Statistical overview of the MTR dataset\. ‘Dev’ refers to MTR\-dev, which can be partitioned from the training set for model debugging and development, but is not used for evaluation in this paper\. ‘Test’ refers to the primary evaluation benchmarkMTR\-Bench\. Note: we publicly release an improved 12\-turn version synthesized by Qwen3\.5 at our repository, which we recommend for future research\.![Refer to caption](https://arxiv.org/html/2605.20729v1/x1.png)Figure 3:Domain Distribution inMTR\-Bench\. The dataset covers a diverse range of subjects, rigorous testing generalization\.

## 5MTR\-Bench: A Realistic Conversational Benchmark

Leveraging theMTR\-Pipeline, we synthesizedMTR\-Bench, a large\-scale, general\-domain conversational retrieval benchmark\. Unlike previous datasets restricted by human cognitive bottlenecks or rigid heuristic rules,MTR\-Benchis explicitly engineered to stress\-test modern RAG systems against the complexities of real\-world production environments\. In this section, we analyze the statistical characteristics of the benchmark and detail the design principles that contribute to its high discriminative difficulty\. Table[1](https://arxiv.org/html/2605.20729#S4.T1)lists the key parameters ofMTR\-Benchand compares them with previous benchmarks\. In the following sections, we will elaborate on how these parameters influence the relevance of the benchmark to real\-world RAG scenarios\. We also provide engineering guidelines to assist users in selecting these parameters for their specific scenarios, thereby enhancing the effectiveness of evaluations using our pipeline \(see Appendix[A\.8](https://arxiv.org/html/2605.20729#A1.SS8)\)\.

### 5\.1Dataset Overview and Statistics

MTR\-Benchis constructed from the Wikipedia 2025\-01 dump, ensuring temporal relevance\. The dataset is split into a training set \(MTR\-train\) and an evaluation benchmark \(MTR\-Bench\)\. As shown in Table[2](https://arxiv.org/html/2605.20729#S4.T2), the benchmark comprises 1,000 conversations with 8,000 turns, featuring an average of 5\.6 distinct topics per conversation, which indicates high informational density\.

##### Domain Diversity

To ensure robust evaluation, a general\-purpose benchmark must cover a wide spectrum of knowledge\. We analyzed the semantic distribution ofMTR\-Benchusing a domain classifier\. As shown in Figure[3](https://arxiv.org/html/2605.20729#S4.F3), the dataset achieves broad coverage across Humanities, STEM, Social Sciences, and Vital Statistics\. This diversity is critical for testing a retriever’s ability to generalize across distinct contexts, rather than overfitting to specific domains\.

##### Topic Flow Dynamics

A key feature ofMTR\-Benchis its complex topic evolution\. We visualize the inter\-turn topical transitions using a Sankey diagram in Figure[6](https://arxiv.org/html/2605.20729#A1.F6)\. The visualization reveals that conversations do not merely drift linearly; they exhibit realistic recursive patterns where users revert to previously discussed subjects or branch into distinct sub\-topics\. This non\-linear flow challenges a retriever’s ability to maintain long\-range context dependence\.

### 5\.2Why isMTR\-BenchHard?

We explicitly engineered three factors that increase difficulty:

#### 5\.2\.1Realistic Topic Switching \(Hard vs\. Soft\)

Human information\-seeking is rarely linear\. Research indicates that users switch topics approximately every 2\.11 turns\(Spinket al\.,[2002](https://arxiv.org/html/2605.20729#bib.bib52)\)\. Most existing benchmarks capture only smooth transitions\.MTR\-Benchimplements two distinct switching modes:

##### Soft Topic Switching:

Simulated via Wikipedia hyperlinks, these transitions represent natural exploratory behavior where a user navigates between semantically related concepts\.

##### Hard Topic Switching:

This mode simulates abrupt context shifts, akin to a user initiating a completely new query without clearing the chat history\. This creates severe contextual interference, forcing the retriever to distinguish between the current query’s intent and irrelevant global history\.

#### 5\.2\.2Production\-Grade Response Characteristics

A major discrepancy exists between academic datasets \(often annotated by humans who prefer brevity\) and production environments \(dominated by verbose LLMs\)\.

##### Long\-Context Responses

LLMs driven by RLHF\(Ouyanget al\.,[2022](https://arxiv.org/html/2605.20729#bib.bib57)\), tend to be verbose\. Analysis of real\-world ShareGPT data reveals an average assistant response length of∼\\sim464 tokens, whereas traditional benchmarks average far fewer\.MTR\-Benchmirrors this reality \(see Table[2](https://arxiv.org/html/2605.20729#S4.T2)\), forcing retrievers to process substantially more noise within the dialogue history\.

##### Ambiguous Decision Style

Traditional datasets often handle ambiguity through clarifying questions\. This approach lowers retrieval difficulty because the subsequent user turn is typically a simple confirmation \(e\.g\., yes or no\), allowing the system to retain the previous gold document\. In contrast,MTR\-Benchmimics production LLMs, which tend to proactively guess the user’s intent to provide an immediate answer\. This eliminates low\-entropy clarification turns and instead fills the history with verbose, speculative content, forcing the retriever to filter through dense noise rather than relying on a static context\.

#### 5\.2\.3Industrial\-Scale Knowledge

##### Knowledge Cutoff \(2025\.01\)

Pre\-trained models often rely on internal parametric memory for older facts\. To strictly evaluate retrieval capability rather than memorization,MTR\-Benchutilizes a very recent Wikipedia dump \(2025\.01\)\. This targets “long\-tail” and recent knowledge\(Kandpalet al\.,[2023](https://arxiv.org/html/2605.20729#bib.bib84)\)that models have not seen during training\.

##### Optimized Document Granularity

We set a target document length of∼\\sim1024 characters\. This choice is informed by LlamaIndex research, balancing the need for sufficient context \(avoiding fragmentation\) with the limits of embedding model fidelity\(Zhuet al\.,[2024](https://arxiv.org/html/2605.20729#bib.bib89)\)\. This granularity ensures that the challenge stems from semantic matching rather than arbitrary chunking artifacts\.

## 6Experiments

### 6\.1Models

We selected a diverse set of dense retrievers for evaluation, the BERT\-sized SOTA shared encoderstella\_en\_400m\_v5\(Zhanget al\.,[2025](https://arxiv.org/html/2605.20729#bib.bib83)\), the community\-popularbge\-large\-en\-v1\.5\(Xiaoet al\.,[2023](https://arxiv.org/html/2605.20729#bib.bib44)\), andgte\-modernbert\-base, which has demonstrated strong performance on the novel ModernBERT\(Warneret al\.,[2024](https://arxiv.org/html/2605.20729#bib.bib92)\)architecture\. In addition to these single\-turn retrievers, we incorporate existing SOTA Conversational Dense Retrievers \(CDRs\) specifically designed to handle multi\-turn dialogue context, includingDragon\-DocChat222[https://huggingface\.co/cerebras/Dragon\-DocChat\-Context\-Encoder](https://huggingface.co/cerebras/Dragon-DocChat-Context-Encoder)andDragon\-ChatQA\(Liuet al\.,[2024](https://arxiv.org/html/2605.20729#bib.bib99)\)\. These models were trained on a synthetic dataset derived fromChatQA\(Liuet al\.,[2024](https://arxiv.org/html/2605.20729#bib.bib99)\)onDragon\+\(Linet al\.,[2023](https://arxiv.org/html/2605.20729#bib.bib43)\)with different implementations\.

### 6\.2Metrics

We employed the commonly used retrieval performance evaluation metrics, Recall \(@5, @20\), NDCG@20, and MRR@20\. Recall only considers whether the golden document is included within a given retrieval budget \(@k\) but is highly interpretable\. NDCG and MRR build upon this by also being sensitive to the golden document’s position in the retrieved set, offering a more fine\-grained representation\. We provide the NDCG and MRR scores in Table[11](https://arxiv.org/html/2605.20729#A1.T11)\.

Table 3:Recall \(R@k\) performance of various retrievers across multiple conversational retrieval benchmarks\. We use R as a general shorthand for Recall\.
### 6\.3Results

For each evaluation instance in the benchmark, the input context is constructed by serializing the conversation history and prepending it to the current query\. The history is formatted by alternating “User:” and “Agent:” prefixes for each turn\. We present our primary experimental findings in Table[3](https://arxiv.org/html/2605.20729#S6.T3)\. The results lead to the following key observations:

##### MTR\-Benchoffers substantial headroom for retriever improvement\.

A consistent observation is thatMTR\-Benchpresents a more challenging evaluation landscape compared to many existing conversational retrieval benchmarks\. While leading models achieve Recall@20 scores exceeding 90 points on several prior datasets, their average Recall@20 on these benchmarks is 43\.54 points higher than their average performance onMTR\-Bench\. This significant performance differential suggests thatMTR\-Benchis less susceptible to score saturation and provides greater capacity to differentiate the capabilities of current and future advanced retrieval systems\.

##### MTR\-Benchprobes deeper retrieval capabilities\.

The benchmark’s challenging nature is further validated by the marginal performance gains when increasing the recall budget\. On average, expanding the retrieval window from the top 5 to the top 20 documents \(Recall@5 vs\. Recall@20\) yields a 15\.06 point improvement in Recall@20 on previous benchmarks\. In contrast, the same expansion onMTR\-Benchresults in a more modest average improvement of only 8\.68 points\. This suggests thatMTR\-Benchdemands more precise and robust retrieval, as simple increases in the number of retrieved candidates provide diminishing returns, indicating a more complex relevance landscape\.

## 7Analysis

Table 4:Validation on the industrial\-scale MTR\-FINANCE dataset\. Top row: The full pipeline generates high\-quality benchmarks that remain challenging for retrievers \(low Recall\)\. Bottom row: Removing the quality filter results in higher recall and evaluation scores, indicating the inclusion of simpler, less discriminative documents\.### 7\.1Robustness Across Domains

A critical requirement for automated benchmark synthesis is transferability to domain\-specific or proprietary knowledge bases\. To validate this, we appliedMTR\-Pipelineto an internal industrial\-scale financial corpus\. Crucially, this underlying knowledge base is highly complex and heterogeneous, comprising a dense mixture of unstructured emails, formal regulations, legal charters, and granular transaction records\. Due to the proprietary nature of the source data, the raw corpus remains private; however, we plan to partially release the generated dialogues to serve as a reference in future work\.

As shown in the first row of Table[4](https://arxiv.org/html/2605.20729#S7.T4), the synthetic financial benchmark maintains high annotation quality, with an Answer Quality score of 4\.91 and Answer\-Evidence Faithfulness of 4\.70\. Conversely, the retrieval difficulty remains substantial; the BGE retriever achieves a Recall@5 of only 0\.37 onMTR\-Bench\-finance, compared to much higher scores on simpler datasets\. This low retrieval score is attributable to the intricate nature of the mixed data sources\. This demonstrates thatMTR\-Suitesuccessfully generates challenging, high\-quality evaluation instances for specialized domains, confirming its viability for auditing enterprise RAG systems beyond general\-domain Wikipedia data\.

### 7\.2Ablation Study

We performed an ablation study to quantify the contributions of theFilteringandPolishermodules, usingMTR\-Bench\-financeas the testbed\.

##### Impact of Filtering\.

We compared the full pipeline against a version where the quality filter was removed\. As shown in Table[4](https://arxiv.org/html/2605.20729#S7.T4)\(w/o Filter\), removing the filter resulted in increased retrieval recall \(R@5 rose from 0\.37 to 0\.45\) and slightly higher automated evaluation scores\. While higher scores might initially seem positive, qualitative analysis reveals that the unfiltered corpus contains simpler, less information\-dense passages that are trivial to retrieve and question\. The filtering module is therefore essential for enforcing a high difficulty standard and ensuring the benchmark targets complex, educational content\.

##### Impact of Polisher\.

To evaluate the Polisher, we utilized the human evaluation setup described in Section 7\.2\. When the Polisher module was removed from the pipeline, the accuracy of human annotators in identifying machine\-generated questions increased significantly from 62% to 79%\. This indicates that the Polisher plays a crucial role in linguistic refinement, smoothing out the structural rigidity of the raw Questioner agent to produce natural, human\-like dialogue\.

### 7\.3Disentangling Knowledge Base Recency from Dialogue Complexity

SinceMTR\-Benchuses a recent Wikipedia dump \(2025\.01\) unseen during retriever pre\-training, a natural concern is whether the observed performance degradation stems primarily from unfamiliar knowledge or from the linguistic complexity of conversational queries\. We provide two pieces of evidence to disentangle these factors\.

##### Oracle Query Rewriting\.

Our LLM\-based query rewriting analysis \(Figure[7](https://arxiv.org/html/2605.20729#A1.F7)\) reveals a 20%–40% R@5 improvement when raw conversational queries are rewritten into explicit, self\-contained forms\. Crucially, the rewritten queries achieve high absolute recallon the same 2025\.01 knowledge base, demonstrating that gold documents are readily retrievable when query intent is made explicit\. This confirms that the difficulty is not primarily caused by the knowledge base being unfamiliar to the retrievers, but rather by the linguistic complexity of the raw conversational queries, ellipsis, coreference, topic switching, and verbose history\.

##### Cross\-Domain Consistency\.

WhenMTR\-Pipelineis applied to a completely different corpus \(the financial domain in Section[5\.1](https://arxiv.org/html/2605.20729#S5.SS1)\), retrievers exhibit similarly low performance despite the corpus having its own temporal characteristics unrelated to Wikipedia recency\. This further supports that the pipeline’s dialogue design is the primary driver of retrieval difficulty\.

Together, these results confirm thatMTR\-Bench’s challenge arises predominantly from genuine conversational complexity rather than knowledge base artifacts\.

### 7\.4Extended Recall@k Analysis

To assess retrieval performance under large candidate budgets—relevant for RAG pipelines using long\-context reader models that can consume hundreds or thousands of documents—we compute extended Recall@k scores onMTR\-Bench\. Results are presented in Table[5](https://arxiv.org/html/2605.20729#S7.T5)\.

Table 5:Extended Recall@k onMTR\-Bench\. Even atk=1000k\{=\}1000, no model achieves full recall, confirming that a substantial fraction of gold documents are entirely outside the retriever’s representational reach\.Several observations emerge\. First,MTR\-Benchremains challenging even at very largekk: the best model \(gte\-Qwen2\-7B\) reaches only 82\.2% at R@1000, while bge\-large saturates at 47\.0%\. Second, model\-dependent scaling behavior is evident: the 7B\-parameter model benefits substantially from increasingkk\(R@20→\\toR@1000:\+\+29\.0 points\), suggesting it places relevant documents in the broader candidate set but struggles with precise ranking\. In contrast, bge\-large shows diminishing returns \(\+\+12\.0 points\), indicating many gold documents are entirely outside its representational capacity\. Third, for RAG pipelines using a long\-context reader over the top 100 retrieved documents, even the best retriever still misses∼\\sim34% of relevant evidence, a hard ceiling that no amount of reader sophistication can overcome\.

## 8Conclusion

This paper presentsMTR\-Suite, a scalable framework for high\-fidelity conversational retrieval benchmarking\. By anchoringMTR\-Evalto human\-validated metrics and constrainingMTR\-Pipelinewith greedy traversal clustering, we decouple data complexity from model\-specific biases\. Crucially, in enterprise environments where knowledge bases evolve rapidly, manual annotation often serves as a lagging indicator\.MTR\-Suiteaddresses this by enabling continuous, on\-demand benchmarking synchronized with data updates\. Our experiments show that whileMTR\-Evalconfirms the linguistic quality of our data, the significant performance drop in state\-of\-the\-art retrievers proves thatMTR\-Benchintroduces genuine semantic challenges rather than artifacts of LLM favoritism\.MTR\-Suitethus provides a cost\-effective, rigorous standard for evaluating RAG systems without the risk of circular reasoning\.

## 9Acknowledgement

This work was supported in part by the National Natural Science Foundation of China \(Nos\. U24A20334 and 62276056\), the Yunnan Fundamental Research Projects \(No\. 202401BC070021\), the Yunnan Science and Technology Major Project \(No\. 202502AD080014\), the Fundamental Research Funds for the Central Universities \(Nos\. N25BSS054 and N25BSS094\), and the Program of Introducing Talents of Discipline to Universities, Plan 111 \(No\. B16009\)\.

## 10Limitations

MTR\-Benchfocuses explicitly on the retrieval component rather than end\-to\-end \(E2E\) generation\. This is a deliberate choice to preserve conversational realism: standard E2E metrics \(e\.g\., Exact Match, BLEU\) require rigid formatting or short ground\-truth answers, which contradict the verbose and explanatory nature of real\-world RAG responses\. By avoiding these artificial constraints, we ensure the benchmark reflects actual application utility\. Given that retrieval quality dictates the factual upper bound of RAG systems, we prioritize a precise diagnostic of the retriever over a potentially noisy and constrained E2E evaluation\.

## 11Ethical Statement

As we releaseMTR\-Benchto the research community, ensuring the safety, compliance, and ethical integrity of the dataset is paramount\. The harmlessness of the generated data is guaranteed through a three\-tiered framework:

##### Source Data Hygiene:

Our document corpus is derived exclusively from Wikipedia\. While Wikipedia inherently adheres to strict community guidelines that prohibit explicit sexual or violent content, we implement an additional layer of safety during preprocessing\. As detailed in our pipeline description, all documents pass through a rigorous educational quality filter\. This step automatically identifies and removes any residual entries containing sensitive, offensive, or non\-educational material, ensuring that the underlying knowledge base remains strictly knowledge\-centric and “clean” before it enters the generation phase\.

##### Model Alignment:

The synthetic data generation is powered by state\-of\-the\-art Large Language Models \(LLMs\) that have undergone extensive safety alignment \(e\.g\., RLHF\) to reject harmful instructions\. Furthermore, our prompt engineering is meticulously designed to be task\-specific\. By focusing the agents strictly on information retrieval and reasoning tasks, we minimize the risk of "jailbreaking" the models or inducing the generation of toxic, biased, or inappropriate content\.

##### Constrained Generation:

Unlike open\-ended chitchat systems, the generation process inMTR\-Pipelineis not free\-form\. The interactions are strictly grounded in the prstyovided reference documents\. The agents are instructed to formulate questions and answers solely based on the filtered evidence present in the corpus\. This constraint acts as a final safeguard, preventing the models from hallucinating harmful content or introducing external biases unrelated to the source text\.

Collectively, these measures ensure thatMTR\-Benchis compliant with ethical standards and safe for broad academic use\.

## References

- V\. Adlakha, S\. Dhuliawala, K\. Suleman, H\. de Vries, and S\. Reddy \(2022\)TopiOCQA: open\-domain conversational question answering with topic switching\.Transactions of the Association for Computational Linguistics10,pp\. 468–483\.External Links:[Link](https://aclanthology.org/2022.tacl-1.27/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00471)Cited by:[§2\.1](https://arxiv.org/html/2605.20729#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2605.20729#S4.T1.10.13.2.1)\.
- R\. Anantha, S\. Vakulenko, Z\. Tu, S\. Longpre, S\. Pulman, and S\. Chappidi \(2021\)Open\-domain question answering goes conversational via question rewriting\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,K\. Toutanova, A\. Rumshisky, L\. Zettlemoyer, D\. Hakkani\-Tur, I\. Beltagy, S\. Bethard, R\. Cotterell, T\. Chakraborty, and Y\. Zhou \(Eds\.\),Online,pp\. 520–534\.External Links:[Link](https://aclanthology.org/2021.naacl-main.44/),[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.44)Cited by:[§A\.5\.4](https://arxiv.org/html/2605.20729#A1.SS5.SSS4.p1.1),[§2\.1](https://arxiv.org/html/2605.20729#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2605.20729#S4.T1.10.12.1.1)\.
- A\. Z\. Broder \(1997\)On the resemblance and containment of documents\.InCompression and Complexity of Sequences 1997\. Proceedings,pp\. 21–29\.Cited by:[§4\.1](https://arxiv.org/html/2605.20729#S4.SS1.p1.1)\.
- C\. Buckley and E\. M\. Voorhees \(2004\)Retrieval evaluation with incomplete information\.SIGIR\.Cited by:[§1](https://arxiv.org/html/2605.20729#S1.p2.1)\.
- Y\. Cheng, K\. Mao, Z\. Zhao, G\. Dong, H\. Qian, Y\. Wu, T\. Sakai, J\. Wen, and Z\. Dou \(2024\)CORAL: benchmarking multi\-turn conversational retrieval\-augmentation generation\.External Links:2410\.23090,[Link](https://arxiv.org/abs/2410.23090)Cited by:[§1](https://arxiv.org/html/2605.20729#S1.p3.1),[§2\.2](https://arxiv.org/html/2605.20729#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2605.20729#S4.T1.7.7.5)\.
- W\. Chiang, L\. Zheng, Y\. Sheng, A\. N\. Angelopoulos, T\. Li, D\. Li, H\. Zhang, B\. Zhu, M\. Jordan, J\. E\. Gonzalez, and I\. Stoica \(2024\)Chatbot arena: an open platform for evaluating llms by human preference\.External Links:2403\.04132,[Link](https://arxiv.org/abs/2403.04132)Cited by:[§A\.5\.2](https://arxiv.org/html/2605.20729#A1.SS5.SSS2.p1.1)\.
- E\. Choi, H\. He, M\. Iyyer, M\. Yatskar, W\. Yih, Y\. Choi, P\. Liang, and L\. Zettlemoyer \(2018\)QuAC: question answering in context\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),Brussels, Belgium,pp\. 2174–2184\.External Links:[Link](https://aclanthology.org/D18-1241/),[Document](https://dx.doi.org/10.18653/v1/D18-1241)Cited by:[§2\.1](https://arxiv.org/html/2605.20729#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2605.20729#S4.T1.1.1.2)\.
- T\. Cohereet al\.\(2025\)Command a: an enterprise\-ready large language model\.External Links:2504\.00698,[Link](https://arxiv.org/abs/2504.00698)Cited by:[2nd item](https://arxiv.org/html/2605.20729#A1.I3.i2.p1.1)\.
- J\. Dalton, C\. Xiong, and J\. Callan \(2020\)TREC cast 2019: the conversational assistance track overview\.External Links:2003\.13624,[Link](https://arxiv.org/abs/2003.13624)Cited by:[§2\.1](https://arxiv.org/html/2605.20729#S2.SS1.p1.1)\.
- DeepSeek\-AIet al\.\(2024\)DeepSeek\-v3 technical report\.External Links:2412\.19437,[Link](https://arxiv.org/abs/2412.19437)Cited by:[§1](https://arxiv.org/html/2605.20729#S1.p1.1)\.
- DeepSeek\-AIet al\.\(2025\)DeepSeek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.External Links:2501\.12948,[Link](https://arxiv.org/abs/2501.12948)Cited by:[§1](https://arxiv.org/html/2605.20729#S1.p1.1)\.
- M\. Douze, A\. Guzhva, C\. Deng, J\. Johnson, G\. Szilvasy, P\. Mazaré, M\. Lomeli, L\. Hosseini, and H\. Jégou \(2025\)The faiss library\.External Links:2401\.08281,[Link](https://arxiv.org/abs/2401.08281)Cited by:[§A\.5\.3](https://arxiv.org/html/2605.20729#A1.SS5.SSS3.p2.1)\.
- S\. Feng, H\. Wan, C\. Gunasekara, S\. Patel, S\. Joshi, and L\. Lastras \(2020\)Doc2dial: a goal\-oriented document\-grounded dialogue dataset\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 8118–8128\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.652/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.652)Cited by:[§A\.5\.4](https://arxiv.org/html/2605.20729#A1.SS5.SSS4.p1.1),[§2\.1](https://arxiv.org/html/2605.20729#S2.SS1.p1.1),[§4\.3](https://arxiv.org/html/2605.20729#S4.SS3.SSS0.Px4.p1.1),[Table 1](https://arxiv.org/html/2605.20729#S4.T1.3.3.3)\.
- Geminiet al\.\(2024\)Gemini 1\.5: unlocking multimodal understanding across millions of tokens of context\.External Links:2403\.05530,[Link](https://arxiv.org/abs/2403.05530)Cited by:[§1](https://arxiv.org/html/2605.20729#S1.p1.1)\.
- T\. GLMet al\.\(2024\)ChatGLM: a family of large language models from glm\-130b to glm\-4 all tools\.External Links:2406\.12793,[Link](https://arxiv.org/abs/2406.12793)Cited by:[7th item](https://arxiv.org/html/2605.20729#A1.I3.i7.p1.1)\.
- A\. Q\. Jianget al\.\(2023\)Mistral 7b\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[6th item](https://arxiv.org/html/2605.20729#A1.I3.i6.p1.1)\.
- N\. Kandpal, H\. Deng, A\. Roberts, E\. Wallace, and C\. Raffel \(2023\)Large language models struggle to learn long\-tail knowledge\.InProceedings of the 40th International Conference on Machine Learning \(ICML 2023\),External Links:[Link](https://arxiv.org/abs/2211.08411)Cited by:[§A\.8\.4](https://arxiv.org/html/2605.20729#A1.SS8.SSS4.p1.1),[§5\.2\.3](https://arxiv.org/html/2605.20729#S5.SS2.SSS3.Px1.p1.1)\.
- C\. Lee, R\. Roy, M\. Xu, J\. Raiman, M\. Shoeybi, B\. Catanzaro, and W\. Ping \(2025\)NV\-embed: improved techniques for training llms as generalist embedding models\.External Links:2405\.17428,[Link](https://arxiv.org/abs/2405.17428)Cited by:[§A\.8\.3](https://arxiv.org/html/2605.20729#A1.SS8.SSS3.p1.1)\.
- Y\. Lee, C\. Gunasekara, D\. Contractor, R\. F\. Astudillo, and R\. Florian \(2024\)Multi\-document grounded multi\-turn synthetic dialog generation\.External Links:2409\.11500,[Link](https://arxiv.org/abs/2409.11500)Cited by:[§2\.2](https://arxiv.org/html/2605.20729#S2.SS2.p1.1)\.
- Lewiset al\.\(2020\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 9459–9474\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2605.20729#S1.p1.1)\.
- C\. Li, M\. Qin, S\. Xiao, J\. Chen, K\. Luo, Y\. Shao, D\. Lian, and Z\. Liu \(2024\)Making text embedders few\-shot learners\.External Links:2409\.15700,[Link](https://arxiv.org/abs/2409.15700)Cited by:[§A\.8\.3](https://arxiv.org/html/2605.20729#A1.SS8.SSS3.p1.1)\.
- S\. Lin, A\. Asai, M\. Li, B\. Oguz, J\. Lin, Y\. Mehdad, W\. Yih, and X\. Chen \(2023\)How to train your dragon: diverse augmentation towards generalizable dense retrieval\.External Links:2302\.07452,[Link](https://arxiv.org/abs/2302.07452)Cited by:[§6\.1](https://arxiv.org/html/2605.20729#S6.SS1.p1.1)\.
- Z\. Liu, W\. Ping, R\. Roy, P\. Xu, C\. Lee, M\. Shoeybi, and B\. Catanzaro \(2024\)ChatQA: surpassing GPT\-4 on conversational QA and RAG\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=bkUvKPKafQ)Cited by:[§6\.1](https://arxiv.org/html/2605.20729#S6.SS1.p1.1)\.
- Llamaet al\.\(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[5th item](https://arxiv.org/html/2605.20729#A1.I3.i5.p1.1)\.
- Y\. Meng, M\. Xia, and D\. Chen \(2024\)SimPO: simple preference optimization with a reference\-free reward\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§A\.8\.1](https://arxiv.org/html/2605.20729#A1.SS8.SSS1.Px1.p1.1)\.
- N\. Muennighoff, H\. Su, L\. Wang, N\. Yang, F\. Wei, T\. Yu, A\. Singh, and D\. Kiela \(2024\)Generative representational instruction tuning\.External Links:2402\.09906,[Link](https://arxiv.org/abs/2402.09906)Cited by:[§A\.8\.3](https://arxiv.org/html/2605.20729#A1.SS8.SSS3.p1.1)\.
- OpenAIet al\.\(2024\)OpenAI o1 system card\.External Links:2412\.16720,[Link](https://arxiv.org/abs/2412.16720)Cited by:[§1](https://arxiv.org/html/2605.20729#S1.p1.1)\.
- OpenAI \(2023\)ChatGPT \(mar 14 version\)\.Note:Accessed: 2025\-02\-07External Links:[Link](https://openai.com/chatgpt)Cited by:[§1](https://arxiv.org/html/2605.20729#S1.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. Lowe \(2022\)Training language models to follow instructions with human feedback\.External Links:2203\.02155,[Link](https://arxiv.org/abs/2203.02155)Cited by:[§A\.8\.1](https://arxiv.org/html/2605.20729#A1.SS8.SSS1.Px1.p1.1),[§5\.2\.2](https://arxiv.org/html/2605.20729#S5.SS2.SSS2.Px1.p1.1)\.
- R\. Park, R\. Rafailov, S\. Ermon, and C\. Finn \(2024\)Disentangling length from quality in direct preference optimization\.External Links:2403\.19159,[Link](https://arxiv.org/abs/2403.19159)Cited by:[§A\.8\.1](https://arxiv.org/html/2605.20729#A1.SS8.SSS1.Px1.p1.1)\.
- G\. Penedo, H\. Kydlíček, L\. B\. allal, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. V\. Werra, and T\. Wolf \(2024\)The fineweb datasets: decanting the web for the finest text data at scale\.External Links:2406\.17557,[Link](https://arxiv.org/abs/2406.17557)Cited by:[§4\.1](https://arxiv.org/html/2605.20729#S4.SS1.p1.1)\.
- J\. Portes, A\. Trott, S\. Havens, D\. KING, A\. Venigalla, M\. Nadeem, N\. Sardana, D\. Khudia, and J\. Frankle \(2023\)MosaicBERT: a bidirectional encoder optimized for fast pretraining\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 3106–3130\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/095a6917768712b7ccc61acbeecad1d8-Paper-Conference.pdf)Cited by:[§A\.8\.3](https://arxiv.org/html/2605.20729#A1.SS8.SSS3.p1.1)\.
- Qwenet al\.\(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[4th item](https://arxiv.org/html/2605.20729#A1.I3.i4.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. Finn \(2024\)Direct preference optimization: your language model is secretly a reward model\.External Links:2305\.18290,[Link](https://arxiv.org/abs/2305.18290)Cited by:[§A\.8\.1](https://arxiv.org/html/2605.20729#A1.SS8.SSS1.Px1.p1.1)\.
- S\. Reddy, D\. Chen, and C\. D\. Manning \(2019\)CoQA: a conversational question answering challenge\.Transactions of the Association for Computational Linguistics7,pp\. 249–266\.External Links:[Link](https://aclanthology.org/Q19-1016/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00266)Cited by:[§2\.1](https://arxiv.org/html/2605.20729#S2.SS1.p1.1)\.
- T\. Schick, J\. Dwivedi\-Yu, R\. Dessi, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom \(2023\)Toolformer: language models can teach themselves to use tools\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 68539–68551\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/d842425e4bf79ba039352da0f658a906-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2605.20729#S1.p1.1)\.
- A\. Spink, H\. C\. Özmutlu, and S\. Özmutlu \(2002\)Multitasking information seeking and searching processes\.J\. Assoc\. Inf\. Sci\. Technol\.53,pp\. 639–652\.External Links:[Link](https://api.semanticscholar.org/CorpusID:653649)Cited by:[§A\.8\.2](https://arxiv.org/html/2605.20729#A1.SS8.SSS2.p1.1),[§5\.2\.1](https://arxiv.org/html/2605.20729#S5.SS2.SSS1.p1.1)\.
- G\. Teamet al\.\(2025\)Gemma 3 technical report\.External Links:2503\.19786,[Link](https://arxiv.org/abs/2503.19786)Cited by:[1st item](https://arxiv.org/html/2605.20729#A1.I3.i1.p1.1)\.
- L\. Wang, N\. Yang, X\. Huang, L\. Yang, R\. Majumder, and F\. Wei \(2024\)Improving text embeddings with large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 11897–11916\.External Links:[Link](https://aclanthology.org/2024.acl-long.642/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.642)Cited by:[§A\.8\.3](https://arxiv.org/html/2605.20729#A1.SS8.SSS3.p1.1)\.
- B\. Warner, A\. Chaffin, B\. Clavié, O\. Weller, O\. Hallström, S\. Taghadouini, A\. Gallagher, R\. Biswas, F\. Ladhak, T\. Aarsen, N\. Cooper, G\. Adams, J\. Howard, and I\. Poli \(2024\)Smarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference\.External Links:2412\.13663,[Link](https://arxiv.org/abs/2412.13663)Cited by:[§A\.8\.3](https://arxiv.org/html/2605.20729#A1.SS8.SSS3.p1.1),[§6\.1](https://arxiv.org/html/2605.20729#S6.SS1.p1.1)\.
- Z\. Wu, R\. Parish, H\. Cheng, S\. Min, P\. Ammanabrolu, M\. Ostendorf, and H\. Hajishirzi \(2023\)InSCIt: information\-seeking conversations with mixed\-initiative interactions\.Transactions of the Association for Computational Linguistics11,pp\. 453–468\.External Links:[Link](https://aclanthology.org/2023.tacl-1.27/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00559)Cited by:[Table 1](https://arxiv.org/html/2605.20729#S4.T1.10.14.3.1)\.
- S\. Xiao, Z\. Liu, P\. Zhang, and N\. Muennighoff \(2023\)C\-pack: packaged resources to advance general chinese embedding\.External Links:2309\.07597Cited by:[§6\.1](https://arxiv.org/html/2605.20729#S6.SS1.p1.1)\.
- Z\. Xu, S\. Jain, and M\. Kankanhalli \(2025\)Hallucination is inevitable: an innate limitation of large language models\.External Links:2401\.11817,[Link](https://arxiv.org/abs/2401.11817)Cited by:[§1](https://arxiv.org/html/2605.20729#S1.p1.1)\.
- D\. Zhang, J\. Li, Z\. Zeng, and F\. Wang \(2025\)Jasper and stella: distillation of sota embedding models\.External Links:2412\.19048,[Link](https://arxiv.org/abs/2412.19048)Cited by:[§6\.1](https://arxiv.org/html/2605.20729#S6.SS1.p1.1)\.
- X\. Zhang, Y\. Zhang, D\. Long, W\. Xie, Z\. Dai, J\. Tang, H\. Lin, B\. Yang, P\. Xie, F\. Huang, M\. Zhang, W\. Li, and M\. Zhang \(2024\)mGTE: generalized long\-context text representation and reranking models for multilingual text retrieval\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track,F\. Dernoncourt, D\. Preoţiuc\-Pietro, and A\. Shimorina \(Eds\.\),Miami, Florida, US,pp\. 1393–1412\.External Links:[Link](https://aclanthology.org/2024.emnlp-industry.103/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-industry.103)Cited by:[§A\.8\.3](https://arxiv.org/html/2605.20729#A1.SS8.SSS3.p1.1)\.
- D\. Zhu, L\. Wang, N\. Yang, Y\. Song, W\. Wu, F\. Wei, and S\. Li \(2024\)LongEmbed: extending embedding models for long context retrieval\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 802–816\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.47/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.47)Cited by:[§A\.8\.3](https://arxiv.org/html/2605.20729#A1.SS8.SSS3.p1.1),[§5\.2\.3](https://arxiv.org/html/2605.20729#S5.SS2.SSS3.Px2.p1.1)\.
- J\. Zobel \(1998\)How reliable are the results of large\-scale information retrieval experiments?\.SIGIR\.Cited by:[§1](https://arxiv.org/html/2605.20729#S1.p2.1)\.

## Appendix AAppendix

### A\.1Previous Benchmark Evaluation

For each historical benchmark considered in our evaluation, we elicited four distinct scores from each of the seven selected models\. This systematic process yielded four 7x6 model\-benchmark heatmaps, providing a granular view of relative performance across different tasks, as depicted in[figs\.˜4\(a\)](https://arxiv.org/html/2605.20729#A1.F4.sf1),[4\(b\)](https://arxiv.org/html/2605.20729#A1.F4.sf2),[4\(d\)](https://arxiv.org/html/2605.20729#A1.F4.sf4)and[4\(c\)](https://arxiv.org/html/2605.20729#A1.F4.sf3)\.

### A\.2Quality Audit and Limitations of the CORAL Benchmark

CORAL constructs a hierarchical title tree by leveraging the structure of Wikipedia, such as titles and subtitles\. By traversing this tree, it generates answers at various levels\. The method utilizes GPT\-4 to rewrite titles into user questions, while answers are extracted directly from the body text under the corresponding titles\. Gold documents are identified from the references cited within the text\. This pipeline relies heavily on documents possessing a clear hierarchical structure, which constrains its utility as a general\-purpose automated process\. Additionally, the approach for constructing the \(question, answer, support document\) tuples is relatively simplistic, resulting in notable quality concerns\.

Our preliminary evaluation indicated low performance on the CORAL benchmark\. A detailed case study revealed that the dataset exhibits notable annotation inconsistencies, which may affect its reliability for fine\-grained benchmarking\. The first data sample from the dataset serves as an illustration:

> U:What methods are used to predict these geodynamic regimes? A:exoplanets have been directly observed and remotely sensed ,\[15\] but due to their great distance and proximity to obscuring energy sources \(the stars they orbit\), there is little concrete knowledge of their composition and geodynamic regime\. therefore, the majority of information and conjectures made about them come from alternative sources support document:\{’document\_idx’: 15, ’ctx’: “doi it uses a managed registry \( providing social and technical infrastructure \) \. it does not assume any specific business model for the provision of identifiers or services and enables other existing services to link to it in defined ways \. several approaches for making identifiers persistent have been proposed \. the comparison of persistent identifier approaches is difficult because they are not all doing the same thing \. imprecisely referring to a set of schemes as “ identifiers ” doesn’t mean that they can be compared easily \. other “ identifier systems ” may be enabling technologies with low barriers to entry , providing an easy to use labeling mechanism that allows anyone to set up a new instance \( examples include persistent uniform resource locator , urls , globally unique identifiers , etc \. \) , but may lack some of the functionality of a registry\-controlled scheme and will usually lack accompanying metadata in a controlled scheme \. the doi system does not have this approach and should not be compared directly to such identifier schemes \. various applications using such enabling technologies with added features have been devised that meet some of the features offered by the doi system for specific sectors \( e\.g\. , ark \) \. a doi name does not depend on the object ’s location and , in this way , is similar to a uniform resource name or purl but differs from an ordinary url \."\}

The dialogue pertains to astronomy, while the support document discusses the Digital Object Identifier \(DOI\) system\. This misalignment illustrates the type of query\-evidence inconsistency that can arise from rule\-based synthesis pipelines relying on document structure\.

We communicated these findings to the CORAL team\. Following our discussion and the provision of quantitative and qualitative evidence, Following constructive discussion, the authors proactively revised their dataset and released an improved version \(CORAL v2\), demonstrating the practical utility ofMTR\-Evalas a diagnostic tool for benchmark quality assurance\. The authors later released a revised version \(hereafter CORAL v2\) and requested further testing\. However, since CORAL v2 is not the dataset associated with the original publication, we present results based on the original version\. It is worth noting that while performance on CORAL v2 improved, the scores remained lower than those achieved on preceding manually annotated datasets\.

This discussion highlights two primary conclusions:

1. 1\.Fully automated benchmark synthesis for conversational retrieval remains a challenging open problem, and quality assurance mechanisms are essential\.
2. 2\.MTR\-Evalcan serve as an effective and actionable diagnostic tool, facilitating iterative improvement of benchmark quality across research groups\.

### A\.3Case Study on Annotation Cognition Boundaries

Human annotation is inherently limited by a "Cognition Boundary\." Annotators typically formulate queries based on the specific document they are reading at that moment\. However, in a large\-scale corpus, the "Gold" document selected by the annotator is often not the unique or exclusive source of the answer\. This results inAnnotation Sparsity, where valid supporting documents are overlooked simply because the annotator did not see them\.

To illustrate this, Table[6](https://arxiv.org/html/2605.20729#A1.T6)compares the human\-annotated "Gold" documents withalternative valid documentsidentified within the corpus\. These examples clearly show that ground truth is rarely unique:

- •Same Answer in Different Documents:In datasets likeTopiOCQAandInSCIT, the information is not unique to the Gold document\. As shown in the table, the alternative documents retrieved by the model contain the exact same answer \(e\.g\., the date "1980" or the country "Brazil"\) as the human\-annotated text\. Both sources are equally correct\.
- •More Direct or Suitable Answers:In other cases, such asQReCC,Doc2Dial, andQuAC, the alternative document found in the corpus may actually address the user’s specific question more directly\. For instance, inQReCC, the user explicitly asks for "types" of heartbeats\. The human annotator selected a document discussingcauses, whereas the alternative document provides a specific list oftypes\. Similarly, inDoc2Dial, the alternative document defines the "Retirement Planner" itself, rather than the "Online Calculator" selected by the annotator\.

These cases demonstrate that the human\-annotated "Gold" document is often justone of severalvalid options, and sometimes not even the most direct one\. Therefore, strict evaluation metrics that penalize models for retrieving these valid alternatives may underreport a system’s true capabilities\.

Table 6:Case study illustrating annotation sparsity\. In each instance, we identify anAlternative Valid Documentfrom the corpus that differs from the human\-annotated Gold standard\. These alternatives often answer the query more directly or provide complementary details, highlighting that relevant evidence in large corpora is rarely unique\.
### A\.4Human Evaluation and Correlation Analysis

Table 7:Pearson correlation between various models and human annotatorsTo complement the automated benchmarks, we conducted human evaluations on the QReCC andMTR\-Benchdataset\. Five human annotators, comprised of professional staff from an independent internal team, assessed the outputs of each model while adhering to a standardized scoring rubric\. The Pearson correlation coefficients between each model’s scores and the averaged human scores are presented in Table[7](https://arxiv.org/html/2605.20729#A1.T7)\. We deliberately refrained from selecting a single model with the highest human correlation as the sole scoring proxy\. This decision stems from the recognized diversity and multifaceted nature of human preferences; leveraging an ensemble of model\-derived scores is posited to enhance the comprehensiveness and robustness of our evaluation by capturing a wider spectrum of desirable attributes\.

### A\.5Engineering Implementation Details of theMTR\-Pipeline

#### A\.5\.1Chunking Separators

We adopt the commonly used delimiters recommended by Langchain \(see Table[8](https://arxiv.org/html/2605.20729#A1.T8)\)\. The priority order of the recursive splitting algorithm is related to the sequence in which the delimiters are arranged\. Compared to crude splitting based on absolute character length, a recursive delimiter\-based splitting method can provide text chunks that are as complete as possible at the document organizational level, without generating incomplete words\. A more detailed introduction can be found in the Langchain documentation333langchain doc: Recursively split by character\.

Table 8:Text Block Separators and their Descriptions
#### A\.5\.2Model Selection

Drawing upon established benchmarks such as the ChatBot Arena leaderboard\(Chianget al\.,[2024](https://arxiv.org/html/2605.20729#bib.bib39)\)and other comprehensive evaluation results, we selected seven SOTA open\-source LLMs for our study\. Due to hardware resource limitations, models such asDeepSeek\-V3/R1andLlama\-3\.1\-405B\-Instructwere excluded from our evaluation\. To ensure model diversity and represent the most capable variant within each model family, we selected only the largest available model from each series\. Consequently, Gemma\-3\-12B\-it was excluded in favor of its larger counterpart\. The chosen models, which form the basis of our comparative analysis, are enumerated below, along with their ChatBot Arena Elo ratings \(where available\) or pertinent characteristics:

- •Gemma3\-27B\-it\(Team and others,[2025](https://arxiv.org/html/2605.20729#bib.bib21)\)\(Elo: 1341\)
- •Command\-a\-03\-2025\(Cohere and others,[2025](https://arxiv.org/html/2605.20729#bib.bib22)\)\(Formerly on ChatBot Arena; 111B parameters\)
- •
- •Qwen2\.5\-72B\-Instruct\(Qwen and others,[2025](https://arxiv.org/html/2605.20729#bib.bib16)\)\(Elo: 1257\)
- •Llama\-3\.3\-70B\-Instruct\(Llama and others,[2024](https://arxiv.org/html/2605.20729#bib.bib19)\)\(Elo: 1257\)
- •Mistral\-Large\-2411\(Jiang and others,[2023](https://arxiv.org/html/2605.20729#bib.bib41)\)\(Elo: 1249\)
- •GLM\-4\-32B\-0414\(GLM and others,[2024](https://arxiv.org/html/2605.20729#bib.bib40)\)\(Not listed on ChatBot Arena; reported benchmark performance comparable to GPT\-4o/DeepSeek\-V3 in promotional materials\)

These models were subjected to rigorous evaluation as detailed below\. The prompts we used for each score were listed from[tables˜15](https://arxiv.org/html/2605.20729#A1.T15),[16](https://arxiv.org/html/2605.20729#A1.T16),[18](https://arxiv.org/html/2605.20729#A1.T18)and[17](https://arxiv.org/html/2605.20729#A1.T17)

![Refer to caption](https://arxiv.org/html/2605.20729v1/figure/answerable_score_heatmap.png)\(a\)Query relevance heatmap visualization\.
![Refer to caption](https://arxiv.org/html/2605.20729v1/figure/quality_score_heatmap.png)\(b\)Quality Score heatmap visualization\.
![Refer to caption](https://arxiv.org/html/2605.20729v1/figure/tag_score_heatmap.png)\(c\)Annotation accuracy heatmap visualization\.
![Refer to caption](https://arxiv.org/html/2605.20729v1/figure/faithful_score_heatmap.png)\(d\)Response Faithful heatmap visualization\.

Figure 4:Summary of MTR\-eval scores for different models on previous benchmarks\.![Refer to caption](https://arxiv.org/html/2605.20729v1/figure/answerable_score_heatmap.png)\(a\)Query Relevance score
![Refer to caption](https://arxiv.org/html/2605.20729v1/figure/quality_score_heatmap.png)\(b\)Quality Score
![Refer to caption](https://arxiv.org/html/2605.20729v1/figure/tag_score_heatmap.png)\(c\)Annotation accuracy
![Refer to caption](https://arxiv.org/html/2605.20729v1/figure/faithful_score_heatmap.png)\(d\)Response Faithful

Figure 5:Evaluating the quality of benchmarks generated by various model pairings to select the best combination\.##### Model Selection for Data Synthesis

In determining the optimal pairing of models for the questioner and assistant roles in our synthetic data generation pipeline, we explored all 49 pairwise combinations of the seven selected LLMs\. Each combination was tasked with generating a trial dataset comprising 1,000 interaction rounds\. These demonstration datasets served as the empirical basis for evaluating the efficacy of each pair, ultimately informing our selection of specific open\-source models for the designated roles\. A heatmap illustrating the performance of these model combinations in the synthesis task is provided in[figs\.˜5\(a\)](https://arxiv.org/html/2605.20729#A1.F5.sf1),[5\(b\)](https://arxiv.org/html/2605.20729#A1.F5.sf2),[5\(d\)](https://arxiv.org/html/2605.20729#A1.F5.sf4)and[5\(c\)](https://arxiv.org/html/2605.20729#A1.F5.sf3)\. It is noteworthy that this model selection step for synthesis is adaptable; practitioners may opt to utilize more performant, proprietary models accessible via APIs, potentially yielding superior quality in the synthesized data\.

#### A\.5\.3Example of Greedy Clustering

In table[9](https://arxiv.org/html/2605.20729#A1.T9), We observe that the topic within the cluster undergoes a subtle shift\.

Table 9:A cluster on Indigenous Peoples and Related Topics in CanadaInitially, we convert the text into embeddings using the gte\-Qwen2\-7B model, and then retrieve the top\-k candidates for each text embedding using FaissDouzeet al\.\([2025](https://arxiv.org/html/2605.20729#bib.bib51)\)\. This process essentially creates a weighted undirected graph, where the edge weights represent the embedding distance between two documents\. We then apply a greedy algorithm, starting from a random node in the graph, to find a Traveling Salesman Problem \(TSP\) circuit, ensuring that no document is repeated\. Finally, we partition the TSP circuit based on hop distance to form clusters\. To ensure that the information within each cluster is related rather than identical, we remove duplicates from adjacent documents based on 5\-gram similarity\.

#### A\.5\.4Detailed Cost Analysis

To contextualize the economic viability of our proposed LLM\-based data generation approach, we compare its estimated costs against traditional human annotation\. Within the literature of the past five years, prominent dialogue datasets relying on manual annotation include QReCC\(Ananthaet al\.,[2021](https://arxiv.org/html/2605.20729#bib.bib85)\)and Doc2Dial\(Fenget al\.,[2020](https://arxiv.org/html/2605.20729#bib.bib59)\)\. While QReCC did not disclose specific costs, Doc2Dial utilized the Appen\.com crowdsourcing platform, reporting a cost of $1\.50–$2\.00 per annotated dialogue\. It is noteworthy that Doc2Dial dialogues are, on average, shorter and less verbose than those generated in our work\.

We estimate the generation cost using the pricing for the DeepSeek\-V3\.2 model via OpenRouter555[https://openrouter\.ai/deepseek/deepseek\-v3\.2](https://openrouter.ai/deepseek/deepseek-v3.2), which supports a cost\-saving prefix\-caching mechanism\. We model the generation of a complete dialogue averaging 8 turns\. Our estimation assumes the first turn results in a cache miss \(Input: $0\.24/1M tokens\), while the subsequent seven turns benefit from the lower\-cost cache hit \(Input: $0\.19/1M tokens\)\. Assuming each turn involves an input of approximately 3,000 tokens and an output of 80 tokens \(Output: $0\.38/1M tokens\), the total cost per dialogue is calculated as:

Cost=1×3000×$​0\.241​M⏟Input \(1 miss\)\+7×3000×$​0\.191​M⏟Input \(7 hits\)\\displaystyle=\\underbrace\{1\\times 3000\\times\\frac\{\\mathdollar 0\.24\}\{1\\text\{M\}\}\}\_\{\\text\{Input \(1 miss\)\}\}\+\\underbrace\{7\\times 3000\\times\\frac\{\\mathdollar 0\.19\}\{1\\text\{M\}\}\}\_\{\\text\{Input \(7 hits\)\}\}\+\(8×80×$​0\.381​M\)⏟Output \(8 turns\)\\displaystyle\\quad\+\\underbrace\{\(8\\times 80\\times\\frac\{\\mathdollar 0\.38\}\{1\\text\{M\}\}\)\}\_\{\\text\{Output \(8 turns\)\}\}≈$​0\.005\\displaystyle\\approx\\mathdollar 0\.005
This estimated cost is approximately 1/300th to 1/400th of the reported human annotation cost for Doc2Dial, demonstrating a substantial reduction in expenses\. While enterprises with stringent data privacy requirements may opt for on\-premise or private cloud deployment, the inference costs for such local solutions are generally comparable to calling these commercial APIs\.

### A\.6Prompts and Dialogue Example

We have listed prompts for each role in Tables from[12](https://arxiv.org/html/2605.20729#A1.T12)through[14](https://arxiv.org/html/2605.20729#A1.T14)\. Examples synthesized byMTR\-Pipelineare shown in Table[10](https://arxiv.org/html/2605.20729#A1.T10)\.

Table 10:An example ofMTR\-Pipelinesynthesized dialogue\.
### A\.7Dataset Statistics

![Refer to caption](https://arxiv.org/html/2605.20729v1/x2.png)Topic 1Topic 2Topic 3Topic 4Topic 5Topic 6Topic 7Topic 8Conversation TurnFigure 6:Topic Flow inMTR\-Bench\. This Sankey diagram illustrates the flow and evolution of discussion topics across multiple turns in a conversation\. The horizontal axis represents conversation turn, and the width of the colored bands indicates the prominence of each topic at each turn\. The diagram shows how topics emerge, persist, fade, or transition into other topics as the conversation progresses\.In this section, we provide a detailed characterization of our dataset, focusing on its statistical properties \(Table[1](https://arxiv.org/html/2605.20729#S4.T1)\) and domain distribution \(Figure[3](https://arxiv.org/html/2605.20729#S4.F3)\)\. The latter was determined using a classifier distilled from a LLM666[https://huggingface\.co/nvidia/domain\-classifier](https://huggingface.co/nvidia/domain-classifier)\. Given that our data collection methodology did not impose explicit domain constraints, we posit that the observed distribution reflects naturally occurring thematic biases within high\-quality information sources\. Notably, topics pertaining to humanities and arts collectively constitute approximately 30% of the dataset\. The remaining major categories each represent roughly 5%, indicating a diverse yet discernibly skewed distribution\.

To elucidate the inter\-turn topical transitions, we employ Sankey diagrams \(Figure[6](https://arxiv.org/html/2605.20729#A1.F6)\)\. These visualizations reveal that topics not only progress towards novel themes but also exhibit a tendency to revert to previously discussed subjects\. This observed pattern of topic recurrence is analogous to established models of human information\-seeking behavior\.

### A\.8Design Principles for Real\-World Conversational AI Evaluation

#### A\.8\.1Response from LLM

##### Longer Response Length

Chat models deployed in production environments require fine\-tuning through Reinforcement Learning from Human Feedback \(RLHF\)\(Ouyanget al\.,[2022](https://arxiv.org/html/2605.20729#bib.bib57); Rafailovet al\.,[2024](https://arxiv.org/html/2605.20729#bib.bib24)\), which introduces a preference for longer responses\(Menget al\.,[2024](https://arxiv.org/html/2605.20729#bib.bib100); Parket al\.,[2024](https://arxiv.org/html/2605.20729#bib.bib101)\)\.

Analysis of real\-world human\-assistant interactions, exemplified by data from ShareGPT777https://sharegpt\.com/, reveals an average assistant response length of approximately 464 tokens\. This significantly exceeds the typical response lengths found in many human\-annotated datasets\.

This disparity poses a substantial challenge for evaluating conversational retrieval methods as they need demonstrate robust long\-context processing capabilities\.MTR\-Benchincorporates these longer average response lengths to ensure systems are evaluated against more realistic conversational outputs\.

##### Assistant Response Style

Beyond length, the style of AI responses is crucial\. In task\-oriented dialogues, ambiguous user queries can elicit different response strategies\. Human annotators often favor concise clarifying questions, effectively shifting the communication burden\. Conversely, deployed chat assistants frequently opt for “ambiguous decisions,” providing a substantive response despite ambiguity\. This trend is often observed in prominent systems \(e\.g\., Gemini’s typical response pattern\)\.

Since real\-world assistant responses are generated by LLMs exhibiting these tendencies, benchmarks dominated by human\-preferred clarifying questions may not accurately reflect deployed system behavior\.MTR\-Benchintegrates responses mirroring these production LLM styles, thereby improving the benchmark’s alignment with real\-world interactive patterns and providing a more pertinent test of a system’s ability to handle typical AI\-generated dialogue\.

#### A\.8\.2Soft and Hard Topic Switching

According to the study bySpinket al\.\([2002](https://arxiv.org/html/2605.20729#bib.bib52)\), humans typically switch topics every 2\.11 turns during conversational search\. Therefore,MTR\-Benchhas been carefully designed with two types of topic switch\.

##### Soft Topic Switching

This approach mirrors the natural evolution of user interest during information seeking\. It is simulated by transitions between semantically related Wikipedia articles via hyperlinks, reflecting how users organically explore connected concepts through dialogue\.

##### Hard Topic Switching

Conversely, this mode simulates abrupt changes in topic, a common occurrence in real\-world usage\. Such shifts mirror scenarios where users introduce entirely new and unrelated subjects within an ongoing dialogue session, akin to initiating a new inquiry without explicitly starting a new conversation window\. This presents a rigorous test for dialogue retrieval systems, challenging their robustness in managing irrelevant contextual information, re\-orienting to new subjects, and accurately discerning emergent topic boundaries\. This capability is critical for maintaining coherence and utility in diverse, multi\-topic interactions\.

R@5R@20MRR@20NDCG@2010\.010\.030\.030\.050\.050\.070\.070\.090\.090\.0\(a\) bge\-large\-en\-v1\.5wo/ rewritew/ Qwen2\.5\-3Bw/ Qwen2\.5\-7Bw/ Qwen2\.5\-14Bw/ Qwen2\.5\-32Bw/ Qwen2\.5\-72BR@5R@20MRR@20NDCG@2010\.010\.030\.030\.050\.050\.070\.070\.090\.090\.0\(b\) gte\-Qwen2\-7B\-instruct

Figure 7:Performance of different large models as rewriting models onMTR\-Bench\.

#### A\.8\.3Optimizing Document Length and Count

Document length emerges as a crucial parameter in RAG systems, impacting both retriever and generator performance\. While advancements in BERT\-based models\(Porteset al\.,[2023](https://arxiv.org/html/2605.20729#bib.bib93); Zhanget al\.,[2024](https://arxiv.org/html/2605.20729#bib.bib17); Warneret al\.,[2024](https://arxiv.org/html/2605.20729#bib.bib92)\)and the adoption of LLMs for embedding\(Leeet al\.,[2025](https://arxiv.org/html/2605.20729#bib.bib91); Liet al\.,[2024](https://arxiv.org/html/2605.20729#bib.bib88); Wanget al\.,[2024](https://arxiv.org/html/2605.20729#bib.bib62); Muennighoffet al\.,[2024](https://arxiv.org/html/2605.20729#bib.bib86)\)have expanded the capacity to process longer documents \(up to 8192 tokens in some cases\), empirical evidence reveals that retrieval recall remains sensitive to document length variations in practical applications\(Zhuet al\.,[2024](https://arxiv.org/html/2605.20729#bib.bib89)\)\. Conversely, excessively short document splits can lead to fragmented information retrieval, where even relevant documents retrieved may only partially address the user’s query\. To navigate this delicate balance,MTR\-Benchis designed with a target document length of approximately 1024 characters according to research by LlamaIndex\. We chose a slightly longer document length, partly due to the advancements in retriever performance, and partly because this length is closer to the 512\-token limit\. Crucially,MTR\-Benchfeatures a large\-scale document corpus comprising millions of curated documents\. This industrial\-scale corpus ensures that evaluations reflect the challenges of retrieving information from extensive knowledge bases, enhancing the benchmark’s practical relevance\.

#### A\.8\.4More Recent Knowledge Cutoff Date

Language models struggle with long\-tail knowledge, particularly for pre\-trained models containing outdated information\(Kandpalet al\.,[2023](https://arxiv.org/html/2605.20729#bib.bib84)\)\. For these models, updated knowledge often constitutes long\-tail information that they have rarely encountered\. To better align with current industrial application needs, it is necessary to update the knowledge cutoff date of benchmarks\. Unlike most benchmarks that are limited to knowledge up to 2020, we use the Wikipedia 2025\.01 dump as the knowledge base for generalization question\-answer pairs, ensuring that the evaluation of baseline methods and models is more in line with contemporary application scenarios\.

### A\.9Validity and Difficulty Analysis via Query Rewriting

To assess the dataset’s quality and disentangle the source of difficulty, we conducted an oracle analysis using LLM\-based query rewriting\. As shown in Figure[7](https://arxiv.org/html/2605.20729#A1.F7), we observe a substantial performance gap \(20%–40% increase in R@5\) between raw and rewritten queries across different retrievers\. This significant gap serves as a dual validation: \(1\) High Quality: The high performance of rewritten queries acts as a relevance upper bound, confirming that the ground\-truth documents are correctly labeled and retrievable when the intent is explicit\. This rules out the possibility that low baseline scores are due to annotation noise or missing documents\. \(2\) Challenging Nature: The sharp performance drop when switching back to raw queries demonstrates that our dataset contains dense contextual dependencies \(e\.g\., ellipsis and coreference\)\. It confirms that the benchmark’s difficulty stems from genuine linguistic complexity and the need for rigorous context modeling, rather than simple keyword matching\.

Table 11:The results of MRR@20 \(M for short\) and NDCG@20 \(N for short\)\.QUERYYou will be given multiple reference documents, each begins with \[Document ID\]\.Generate ONE natural\-sounding question that:1\. Can be directly answered by ONLY ONE specific document2\. Sounds like a human question \(don’t mention the document\)3\. Starts with the corresponding \[Document ID\]Format: \[Document ID\] Your question hereHere is an example:\{SEED\}Here is the real user input:\*\*Documents:\*\*\{DOCUMENTS\}Table 12:Questioner PromptRESPONSEBased on the provided documents \(and considering previous conversation, if applicable\), think step\-by\-step and provide a detailed and complete answer to the user’s question\. Do not mention any document names or source information in your response\.\*\*Documents:\*\*\{DOCUMENTS\}\*\*Question:\*\*\{QUESTION\}Table 13:Reasoner PromptPHRASEYou are an expert in natural language processing and conversational AI\. Your task is to analyze the provided dialogue and rewrite the last user query into a version that sounds highly natural and conversational, as if it were part of a casual chat or small talk between humans\.\*\*Specific Requirements for the Rewritten Query:\*\*1\. \*\*Naturalness:\*\* It should flow smoothly and sound like spontaneous human speech\.2\. \*\*Incorporate Conversational Features:\*\*\* \*\*Coreference:\*\* Use pronouns \(e\.g\., "it," "they," "that one"\) or other referring expressions where appropriate, leveraging the context from the preceding dialogue turns\.\* \*\*Ellipsis:\*\* Omit words or phrases that are easily understood from the context \(e\.g\., "What about Paris?" instead of "What is the weather forecast for Paris?"\)\.\* Use common conversational fillers or phrasings if appropriate \(e\.g\., "How about…", "And…", "So,…"\)\.3\. \*\*Meaning Preservation:\*\* This is CRUCIAL\. The rewritten query MUST retain the exact original intent and meaning of the original query\. Do not add new information, change the core question, or introduce ambiguity that wasn’t there\. Ensure the rewritten query seeks the same information or performs the same function as the original\.\*\*Example:\*\*\[USER\]: What’s the weather like in London today?\[ASSISTANT\]: Currently, it’s partly cloudy in London with a high of 15°C\. There’s a slight chance of rain later this afternoon\.\[The last query that need to be rewrite\]: What is the weather forecast for Paris for today?\[Rewrite Query\]:And how about Paris?\*\*Real input:\*\*\{MESSAGES\}Table 14:Polisher PromptJUDGE QUERYER TAG DOCUMENT CORRECTYou are evaluating the answerability of a question generated by a large language model based on a given document\. Please read the document and the question carefully\. Then, rate the question based on the following criteria:\*\*Example Scenario:\*\*\*\*Document:\*\*Singapore, officially the Republic of Singapore, is a sovereign island city\-state in maritime Southeast Asia\. It lies about one degree of latitude \(137 kilometres or 85 miles\) north of the equator, off the southern tip of the Malay Peninsula, bordering the Strait of Malacca to the west, the Singapore Strait to the south, the South China Sea to the east, and the Straits of Johor to the north\. Singapore is one of the most densely populated countries in the world, with a multicultural population and strong international trade links\. Its history dates back to the 13th century, but modern Singapore was founded in 1819 by Sir Stamford Raffles as a British trading post\.\*\*Question:\*\*What year did Singapore gain independence?\*\*Answerability Rating Scale:\*\*5 \- Fully Answerable:\*\* The question can be clearly and completely answered using only the information present in the provided document\. The answer is explicitly stated or can be directly inferred without external knowledge\.4 \- Mostly Answerable:\*\* The question can be answered using the information in the document, but some minor external knowledge or common sense might be helpful for a complete understanding\.3 \- Partially Answerable:\*\* The document provides some relevant information to the question, but it does not contain a complete answer\. Additional information from outside the document is required\.2 \- Minimally Answerable:\*\* The document contains very little or indirect information related to the question, making it difficult to answer adequately\.1 \- Not Answerable:\*\* The question cannot be answered based on the information provided in the document\. The document lacks relevant information, or the question is completely unrelated to the document\.\*\*Justification:\*\*The document discusses the founding of modern Singapore in 1819 but does not mention when Singapore gained independence\. Therefore, the question cannot be answered based on the provided text\.\*\*Rating:\*\* 1—\*\*Now, it’s your turn to evaluate a new question based on a new document using the same scale:\*\*\*\*Document:\*\*\{DOCUMENT\}\*\*Question:\*\*\{QUESTION\}Table 15:Question Relevance PromptJUDGE QUERY RELATED TO CORRECT DOCUMENTYou are provided with several reference documents and a question\. Each document begins with its unique identifier in the format \[Document ID\]\. Your task is to determine which one document is the most probable source for the answer to the question\. You must select only one document\.\*\*Example Scenario:\*\*\*\*Documents:\*\*\[1\] The Eiffel Tower, located in Paris, France, was completed in 1889 for the World’s Fair\. It is one of the most recognizable structures globally and stands 330 meters tall\. Gustave Eiffel’s company designed and built the tower\.—\[2\] The Statue of Liberty, a gift from France to the United States, stands on Liberty Island in New York Harbor\. It was dedicated on October 28, 1886\. It represents Libertas, the Roman goddess of freedom\.—\[3\] Big Ben is the nickname for the Great Bell of the striking clock at the north end of the Palace of Westminster in London, UK\. The tower housing the clock is officially named the Elizabeth Tower\. It was completed in 1859\.\*\*Question:\*\*Who designed the Eiffel Tower?\*\*Justification:\*\*Document \[1\] is the only document that discusses the Eiffel Tower and explicitly mentions that "Gustave Eiffel’s company designed and built the tower"\. Documents \[2\] and \[3\] describe the Statue of Liberty and Big Ben, respectively, and contain no information relevant to the designer of the Eiffel Tower\.\*\*Document ID:\*\*\[1\]—\*\*Now, it’s your turn to determine which one document is the most probable source:\*\*\*\*Documents:\*\*\{DOCUMENTS\}\*\*Question:\*\*\{QUESTION\}Table 16:Annotation Correctness PromptJUDGE RESPONSE FAITHFULYou are tasked with evaluating the faithfulness of a response generated by a large language model\. Your goal is to determine how accurately the response reflects the information presented in the provided source document\. Please read the document and the response carefully\. Then, rate the response based on its faithfulness to the document using the following criteria:\*\*Faithfulness Rating Scale:\*\*5 \- Fully Faithful:\*\* The entire response is directly supported by the information present in the provided document\. All claims made in the response can be clearly verified against the document\. No information is introduced that is not found in the document\.4 \- Mostly Faithful:\*\* The core claims and the majority of the information in the response are supported by the document\. It might contain minor details or phrasing not explicitly found in the document, but these additions do not contradict the source information and are reasonable inferences or rephrasing\.3 \- Partially Faithful:\*\* The response contains a mix of supported and unsupported information\. Some parts accurately reflect the document, but other significant parts are either not found in the document \(unsupported\) or contradict the information present in the document \(contradictory\)\. 2 \- Mostly Unfaithful:\*\* The majority of the response is not supported by the document or directly contradicts the information provided\. There might be minimal overlap, but the core message misrepresents or significantly deviates from the source document\.1 \- Not Faithful:\*\* The response is completely unsupported by the document, presents information that directly contradicts the document, hallucinates information, or is entirely unrelated to the document’s content\.\*\*Example Scenario:\*\*\*\*Document:\*\*Singapore, officially the Republic of Singapore, is a sovereign island city\-state in maritime Southeast Asia\. It lies about one degree of latitude \(137 kilometres or 85 miles\) north of the equator, off the southern tip of the Malay Peninsula… Modern Singapore was founded in 1819 by Sir Stamford Raffles as a British trading post\. Singapore is known for its strong international trade links and multicultural population\.\*\*Question \(for context\):\*\*Tell me about Singapore’s founding\.\*\*Response:\*\*Singapore was founded in 1819 by Sir Stamford Raffles\. It gained independence at that time and quickly became a major agricultural exporter in Southeast Asia\.\*\*Justification:\*\*The response correctly states that modern Singapore was founded in 1819 by Sir Stamford Raffles, which is supported by the document\. However, it incorrectly claims Singapore gained independence in 1819 \(the document doesn’t mention independence\) and falsely states it became a major agricultural exporter \(the document mentions trade links but not specifically agriculture\)\. These unsupported and contradictory statements make the response mostly unfaithful\.\*\*Rating:\*\* 2—\*\*Now, it’s your turn to evaluate a new response based on the provided document using the same scale:\*\*\*\*Document:\*\*DOCUMENT\*\*Question \(for context\):\*\*\{QUESTION\}\*\*Response:\*\*\{RESPONSE\}Table 17:Response Faithful PromptJUDGE ANSWER QUALITYYou are tasked with evaluating the quality and correctness of a response in relation to a given question\. Your goal is to determine how well the response answers the question, focusing on accuracy, completeness, and relevance\.Please read the document \(for context\), the question, and the response carefully\. Then, rate the response based on its quality in answering the question using the following criteria:\*\*Answer Quality Rating Scale:\*\*5 \- Excellent Answer:\*\* The response is factually accurate, fully addresses all parts of the question completely, and is directly relevant\. It provides a clear and comprehensive solution to the user’s query\.4 \- Good Answer:\*\* The response is factually accurate and addresses the main parts of the question well\. It might miss a minor detail or nuance, or could be slightly clearer, but overall provides a correct and useful answer\.3 \- Partial Answer:\*\* The response addresses some parts of the question correctly but contains significant omissions or inaccuracies regarding other parts\. Or, it might provide generally correct information that only tangentially answers the specific question asked\. It offers some value but is incomplete or partially flawed\.2 \- Poor Answer:\*\* The response contains significant factual inaccuracies, fails to address the core of the question, or is largely irrelevant\. It provides little value in answering the user’s query\.1 \- Inadequate Answer:\*\* The response is completely factually incorrect, makes no attempt to answer the question, or is entirely off\-topic/irrelevant\.\*\*Example Scenario:\*\*\*\*Document:\*\*Singapore, officially the Republic of Singapore, is a sovereign island city\-state in maritime Southeast Asia\. It lies about one degree of latitude \(137 kilometres or 85 miles\) north of the equator… Modern Singapore was founded in 1819 by Sir Stamford Raffles as a British trading post\.\*\*Question:\*\*Tell me about Singapore’s founding and its current major industries\.\*\*Response:\*\*Singapore was founded by Sir Stamford Raffles in 1819\. Its primary industry is currently agriculture and fishing\.\*\*Justification:\*\*The founding information is correct\. However, the claim about major industries \(agriculture and fishing\) is factually incorrect for modern Singapore\. While the response attempts to answer both parts, a significant part of the answer is wrong\.\*\*Rating:\*\* 3 \- Partial Answer \(Correct on founding, incorrect on industries\)—\*\*Now, it’s your turn to evaluate a new response based on the provided document \(for context\) and question using the scale above:\*\*\*\*Document:\*\*\{DOCUMENT\}\*\*Question:\*\*\{QUESTION\}\*\*Response:\*\*\{RESPONSE\}Table 18:Answer Quality Prompt