Caraman at SemEval-2026 Task 8: Three-Stage Multi-Turn Retrieval with Query Rewriting, Hybrid Search, and Cross-Encoder Reranking

# Three-Stage Multi-Turn Retrieval with Query Rewriting, Hybrid Search, and Cross-Encoder Reranking
Source: [https://arxiv.org/html/2605.12028](https://arxiv.org/html/2605.12028)
David\-Maximilian Caraman Babeş\-Bolyai University Cluj\-Napoca, Romania david\.caraman@stud\.ubbcluj\.ro&Gheorghe Cosmin Silaghi Babeş\-Bolyai University Cluj\-Napoca, Romania gheorghe\.silaghi@ubbcluj\.ro

###### Abstract

We describe our system for SemEval\-2026 Task 8 \(MTRAGEval\), participating in Task A \(Retrieval\) across four English\-language domains\. Our approach employs a three\-stage pipeline: \(1\) query rewriting via a LoRA\-fine\-tuned Qwen 2\.5 7B model that transforms context\-dependent follow\-up questions into standalone queries, \(2\) hybrid BM25 and dense retrieval combined through Reciprocal Rank Fusion, and \(3\) cross\-encoder reranking with BGE\-reranker\-v2\-m3\. On the official test set, the system achieves nDCG@5 of 0\.531, ranking 8th out of 38 participating systems and 10\.7% above the organizer baseline\. Development comparisons reveal that domain\-specific temperature tuning for query generation, where technical domains benefit from deterministic decoding and general domains from controlled randomness, provides consistent gains, while more complex strategies such as domain\-aware prompting and multi\-query expansion degrade performance\.

Caraman at SemEval\-2026 Task 8: Three\-Stage Multi\-Turn Retrieval with Query Rewriting, Hybrid Search, and Cross\-Encoder Reranking

David\-Maximilian CaramanBabeş\-Bolyai UniversityCluj\-Napoca, Romaniadavid\.caraman@stud\.ubbcluj\.roGheorghe Cosmin SilaghiBabeş\-Bolyai UniversityCluj\-Napoca, Romaniagheorghe\.silaghi@ubbcluj\.ro

## 1Introduction

Multi\-turn conversational retrieval poses a fundamental challenge for information retrieval systems: users formulate follow\-up questions that depend on prior conversational context through pronouns \(“How much does it cost?”\), ellipsis, and implicit topic continuations\. A retrieval system that processes only the latest user utterance loses critical context leads to severe performance degradation in later conversation turns\(Katsiset al\.,[2025](https://arxiv.org/html/2605.12028#bib.bib1)\)\.

SemEval\-2026 Task 8\(Rosenthalet al\.,[2026b](https://arxiv.org/html/2605.12028#bib.bib3)\)provides a rigorous benchmark for this problem, spanning four English domains with 777 retrieval queries drawn from 110 multi\-turn human conversations\. Task A requires participants to produce passages that are relevant to the user’s final question, while evaluation being conducted only on the subset of answerable questions\.

We address this challenge with a three\-stage pipeline where each component targets a distinct failure mode: \(i\)query rewritingresolves conversational dependencies, \(ii\)hybrid retrievalcaptures both lexical and semantic matches, and \(iii\)cross\-encoder rerankingrefines the final ranking through fine\-grained query\-passage interaction\. The query rewriter is a Qwen 2\.5 7B Instruct model\(Yanget al\.,[2024](https://arxiv.org/html/2605.12028#bib.bib11)\)fine\-tuned with LoRA\(Huet al\.,[2022](https://arxiv.org/html/2605.12028#bib.bib4)\)on the MTRAGEval gold rewrites, trained entirely on Apple Silicon using the MLX framework\.111[https://github\.com/ml\-explore/mlx](https://github.com/ml-explore/mlx)

Development comparison studies presented in this paper surface instructive negative results: domain\-aware prompting and multi\-query expansion both*degraded*performance relative to simple rewriting; fine\-tuning the cross\-encoder reranker on task\-specific data yielded only marginal gains; and larger candidate pools for reranking*decreased*quality, a counterintuitive finding we analyze in detail\. Our code222[https://github\.com/davidcaraman/semeval2026\-mtrag\-retrieval](https://github.com/davidcaraman/semeval2026-mtrag-retrieval)and fine\-tuned model333[https://huggingface\.co/caraman/Qwen2\.5\-7B\-mtrag\-query\-rewriter\-final](https://huggingface.co/caraman/Qwen2.5-7B-mtrag-query-rewriter-final)are publicly available\.

## 2Background

### 2\.1Task Description

MTRAGEval\(Rosenthalet al\.,[2026b](https://arxiv.org/html/2605.12028#bib.bib3)\), built on the MTRAG\-UN benchmark\(Rosenthalet al\.,[2026a](https://arxiv.org/html/2605.12028#bib.bib2)\), evaluates retrieval\-augmented generation in multi\-turn conversational settings\. Task A \(Retrieval\) requires systems to return a ranked list of 10 passages per query from domain\-specific corpora totaling 366,479 passages across 78,170 documents in four English domains:ClapNQ\(Wikipedia, 183K passages\),Cloud\(IBM technical documentation, 72K\),FiQA\(financial forum discussions, 61K\), andGovt\(U\.S\. government policy documents, 50K\)\. Each corpus uses 512\-token passages with 100\-token overlap\. The primary evaluation metric is nDCG@5\(Järvelin and Kekäläinen,[2002](https://arxiv.org/html/2605.12028#bib.bib14)\), with nDCG@10 and Recall@10 as secondary measures\.

Relevance judgments are derived from human annotations\. Following the MTRAG\-UN taxonomy\(Rosenthalet al\.,[2026a](https://arxiv.org/html/2605.12028#bib.bib2)\), the benchmark contains a substantial fraction of unanswerable and underspecified queries; these remain undisclosed in the test set and are excluded from scoring\.

### 2\.2Related Work

Our system builds on three lines of work\.Conversational query rewritingtransforms context\-dependent questions into standalone queries suitable for traditional retrievers\(Vakulenkoet al\.,[2021](https://arxiv.org/html/2605.12028#bib.bib13)\); we extend this with domain\-specific temperature control for the rewriter’s generation\.Hybrid retrievalcombines lexical matching \(BM25;Robertson and Zaragoza,[2009](https://arxiv.org/html/2605.12028#bib.bib5)\) with dense semantic search\(Reimers and Gurevych,[2019](https://arxiv.org/html/2605.12028#bib.bib10)\)through fusion methods such as Reciprocal Rank Fusion\(Cormacket al\.,[2009](https://arxiv.org/html/2605.12028#bib.bib8)\)\.Cross\-encoder reranking\(Nogueira and Cho,[2019](https://arxiv.org/html/2605.12028#bib.bib9)\)jointly encodes query\-passage pairs for fine\-grained relevance scoring, consistently outperforming bi\-encoder approaches at the cost of higher latency\.

## 3System Overview

Our pipeline processes each conversational query through three sequential stages\. We describe each stage’s design, the experimental evidence motivating it, and the key hyperparameters needed for replication\.

### 3\.1Stage 1: Query Rewriting

Multi\-turn queries frequently contain unresolved references that make them unintelligible to a retrieval system in isolation\. For instance, given a conversation about IBM Cloud where the user asks“How much does it cost?”, the pronoun“it”must be resolved to produce the standalone query“What is the pricing for IBM Cloud services?”\.

This stage takes the base Qwen 2\.5 7B Instruct model, fine\-tunes it with LoRA on gold rewrites from the MTRAGEval training set, and applies a fixed prompt template \(Appendix[A](https://arxiv.org/html/2605.12028#A1)\) that provides conversation history and instructs the model to produce a standalone query\. At inference time, each query is rewritten with a domain\-specific temperature identified through a systematic sweep on the holdout set \(Section[5\.1](https://arxiv.org/html/2605.12028#S5.SS1)\), producing one rewritten query per domain corpus\.

#### Model and Training\.

We fine\-tune Qwen 2\.5 7B Instruct\(Yanget al\.,[2024](https://arxiv.org/html/2605.12028#bib.bib11)\)using LoRA\(Huet al\.,[2022](https://arxiv.org/html/2605.12028#bib.bib4)\)with the parameters presented in Appendix[B](https://arxiv.org/html/2605.12028#A2)\. These hyperparameters were selected through systematic experimentation over LoRA dropout \(0\.05–0\.15\), learning rate \(5×10−65\{\\times\}10^\{\-6\}–1×10−51\{\\times\}10^\{\-5\}\), and the number of adapted layers \(16–28\), with the final configuration chosen based on holdout validation loss\. Training uses the MLX framework on an Apple M4 Max \(128 GB unified memory\) with AdamW \(lr=10−5\\text\{lr\}\{=\}10^\{\-5\}, weight decay 0\.01\), effective batch size 16 \(micro\-batch 2, gradient accumulation 8\), and gradient checkpointing for memory efficiency\. The model trains for 500 iterations on 699 query rewriting examples derived from MTRAGEval gold rewrites, with 78 examples held out for validation\.

#### Checkpoint Selection\.

Figure[1](https://arxiv.org/html/2605.12028#S3.F1)shows the training dynamics\. Validation loss decreases rapidly in the first 200 iterations \(3\.084→\\to0\.376\), then plateaus around 0\.37–0\.42\. After iteration 350, training loss drops sharply to∼\{\\sim\}0\.1 while validation loss oscillates, indicating the onset of overfitting\. We select the final checkpoint at iteration 500 \(validation loss 0\.373\), which is within 0\.001 of the best validation loss \(0\.372 at iteration 450\), confirming that the model has converged without significant overfitting despite the low training loss\.

![Refer to caption](https://arxiv.org/html/2605.12028v1/x1.png)Figure 1:LoRA fine\-tuning loss curve for the query rewriter\.The prompt template provides up to 10 turns of conversation history followed by the current question, instructing the model to resolve pronouns, include necessary context, and produce a concise, search\-friendly standalone query \(see Appendix[A](https://arxiv.org/html/2605.12028#A1)for the full prompt\)\.

#### Domain\-Specific Temperature\.

Because optimal rewriting behavior varies across domains, our system generates separate rewritten queries for each domain using domain\-specific temperatures, producing four queries per conversation turn, one per domain corpus\. We conducted a systematic temperature sweep \(0\.0–1\.0\) on the development holdout to identify the optimal temperature per domain; the full results and analysis are presented in Section[5\.1](https://arxiv.org/html/2605.12028#S5.SS1)\.

#### First\-Turn Optimization\.

First\-turn queries have no conversational history and are therefore already standalone\. We validate empirically on 19 first\-turn holdout queries that skipping the rewriter produces identical nDCG@5 \(0\.381\) while reducing inference time by 29% \(32 s vs\. 45 s per domain\)\.

### 3\.2Stage 2: Hybrid Retrieval

No single retrieval paradigm dominates across all domains: lexical matching captures exact technical terms critical for Cloud, while semantic similarity handles paraphrases prevalent in ClapNQ\. We combine both approaches through score\-level fusion\.

#### BM25 \(Lexical\)\.

We use BM25S\(Lù,[2024](https://arxiv.org/html/2605.12028#bib.bib15); Robertson and Zaragoza,[2009](https://arxiv.org/html/2605.12028#bib.bib5)\)with English stopword removal and no stemming\. Omitting stemming preserves exact technical terms \(e\.g\.,“Kubernetes”,“401\(k\)”\) that stemming would corrupt\. Each query retrieves 50 candidates\.

#### Dense \(Semantic\)\.

Queries and passages are encoded with BGE\-base\-en\-v1\.5\(Chenet al\.,[2024](https://arxiv.org/html/2605.12028#bib.bib7)\)\(768 dimensions\) and searched against FAISS IndexFlatIP indices\(Johnsonet al\.,[2021](https://arxiv.org/html/2605.12028#bib.bib12)\)with L2\-normalized embeddings \(cosine similarity\)\. Each query retrieves 50 candidates\.

#### Reciprocal Rank Fusion\.

BM25 and dense result lists are merged using RRF\(Cormacket al\.,[2009](https://arxiv.org/html/2605.12028#bib.bib8)\):

scoreRRF​\(d\)=∑r∈\{B,D\}1k\+rankr​\(d\)\\text\{score\}\_\{\\text\{RRF\}\}\(d\)=\\sum\_\{r\\in\\\{B,D\\\}\}\\frac\{1\}\{k\+\\text\{rank\}\_\{r\}\(d\)\}\(1\)wherek=60k\{=\}60,BBis the BM25 ranking, andDDis the dense ranking\. RRF requires no learned weights and produces a single fused candidate list\. Documents appearing in both lists receive contributions from both rankings, naturally boosting passages with cross\-modal agreement\.

### 3\.3Stage 3: Cross\-Encoder Reranking

The fused candidate list is reranked by BGE\-reranker\-v2\-m3\(Chenet al\.,[2024](https://arxiv.org/html/2605.12028#bib.bib7)\), a 568M\-parameter cross\-encoder that jointly encodes each query\-passage pair\. Table[1](https://arxiv.org/html/2605.12028#S3.T1)compares four reranker models atk=50k\{=\}50candidates\. BGE\-reranker\-v2\-m3 \(nDCG@5 0\.416\) substantially outperforms both ms\-marco\-MiniLM\-L\-12\-v2 \(0\.375\) and BGE\-reranker\-base \(0\.307\)\. We also evaluated IBM Granite Reranker R2\(Granite Team, IBM Research AI,[2025](https://arxiv.org/html/2605.12028#bib.bib6)\), which achieved nDCG@5 of 0\.391; we selected BGE\-v2\-m3 for the final system as it achieved the highest score across all candidate pool sizes \(Table[2](https://arxiv.org/html/2605.12028#S3.T2)\)\.

Table 1:Cross\-encoder reranker comparison atk=50k\{=\}50candidates \(nDCG@5, no query rewriting\)\.#### Candidate Pool Size\.

Reranking candidates \(kk\) involve a trade\-off between recall and noise from borderline passages\. Our sweep overk∈\{30,50,100,250,500\}k\\in\\\{30,50,100,250,500\\\}with four rerankers \(Table[2](https://arxiv.org/html/2605.12028#S3.T2)\) shows that all models degrade from their peak tok=500k\{=\}500, but the optimalkkis model\-dependent: three of four rerankers peak atk=30k\{=\}30, yet BGE\-v2\-m3 achieves its best nDCG@5 atk=50k\{=\}50\(0\.416 vs\. 0\.411 atk=30k\{=\}30\), indicating that stronger cross\-encoders can exploit a moderately larger pool before noise dominates\. We adoptk=50k\{=\}50for the final system\.

Table 2:Effect of candidate pool size on reranking quality \(nDCG@5\)\. Three of four models peak atk=30k\{=\}30, but the selected model \(BGE\-v2\-m3\) peaks atk=50k\{=\}50\.We also evaluated fine\-tuning BGE\-reranker\-v2\-m3 on task\-specific hard negatives \(passages from ranks 20–100\)\. The gain was marginal: nDCG@5 improved from 0\.396 to 0\.402 \(\+1\.5%\), while Recall@10 actually declined from 0\.544 to 0\.526\. The pre\-trained model is already well\-calibrated for this mixed\-domain distribution\.

## 4Experimental Setup

#### Data\.

We split the 110 human conversations from the MTRAGEval training set into 88 training conversations \(613 queries\) and 22 holdout conversations \(164 queries: ClapNQ 27, Cloud 47, Govt 62, FiQA 28\)\. All development results reported in this paper are on the holdout set\. Query rewriting training data comprises 699 training and 78 validation examples extracted from gold rewrites\. For the final submission, the query rewriter was retrained on all 777 examples \(training and holdout combined\) using the same hyperparameters selected during development, since the holdout set was no longer needed for model selection\. Official competition results are on the separate test set released by the organizers\.

#### Metrics\.

We report nDCG@5\(Järvelin and Kekäläinen,[2002](https://arxiv.org/html/2605.12028#bib.bib14)\)as the primary metric \(matching the official evaluation\), with nDCG@10 and Recall@10 as secondary measures, computed usingpytrec\_eval\(Van Gysel and de Rijke,[2018](https://arxiv.org/html/2605.12028#bib.bib16)\)\.

#### Infrastructure\.

All experiments run on an Apple M4 Max with 128 GB unified memory\. LoRA training uses the MLX framework;444mlxv0\.12\+,mlx\-lmv0\.12\+retrieval and reranking use PyTorch with MPS acceleration\.555Key libraries:bm25sv0\.2\+,sentence\-transformersv2\.2\+,faiss\-cpuv1\.7\.4\+,FlagEmbeddingv1\.2\+,transformersv4\.36\+\.

## 5Results and Analysis

### 5\.1Query Rewriting Analysis

To identify the optimal generation temperature for each domain, we performed a systematic end\-to\-end sweep over seven temperature values \(0\.0, 0\.1, 0\.2, 0\.3, 0\.5, 0\.7, 1\.0\) on the 164\-query development holdout set\. For each temperature, we ran the full pipeline \(rewriting→\\tohybrid retrieval→\\toreranking\) and measured nDCG@5, ensuring that the selected temperatures optimize the final retrieval quality rather than an intermediate proxy\. Table[3](https://arxiv.org/html/2605.12028#S5.T3)presents the results\.

Table[4](https://arxiv.org/html/2605.12028#S5.T4)provides the full per\-domain breakdown at each domain’s optimal temperature on the development holdout\.

Table 3:Temperature sweep for query rewriting \(nDCG@5\) on the development holdout\. “None” denotes the no\-rewriting baseline using last\-turn queries\. Bold indicates best per column\.Table 4:Per\-domain results at domain\-optimal temperatures on the development holdout \(164 queries\)\.Query rewriting at the best uniform temperature \(t=0\.2t\{=\}0\.2\) improves nDCG@5 from 0\.371 \(no rewriting\) to 0\.422, a13\.7%relative gain, confirming that resolving conversational dependencies is the single most impactful intervention for multi\-turn retrieval\.

The optimal temperature varies substantially across domains\.Cloud\(technical documentation\) achieves its best result att=0\.0t\{=\}0\.0\(0\.473\): its precise technical vocabulary \(e\.g\.,“Kubernetes”,“VPC peering”\) means any generation randomness risks substituting incorrect technical terms, corrupting the lexical signal that BM25 relies on\.ClapNQ\(Wikipedia\) peaks att=0\.2t\{=\}0\.2\(0\.563\), reflecting its well\-structured language that benefits from minimal reformulation diversity\.FiQA\(financial forums\) andGovt\(government documents\) both prefert=0\.3t\{=\}0\.3\(0\.346 and 0\.373\), as their more ambiguous query patterns, including informal forum language in FiQA and policy\-specific terminology in Govt, benefit from slightly more exploratory rewriting\.

Performance degrades monotonically abovet=0\.3t\{=\}0\.3across all domains, witht=1\.0t\{=\}1\.0reducing overall nDCG@5 to 0\.380, barely above the no\-rewriting baseline of 0\.371\. FiQA is particularly sensitive: att=1\.0t\{=\}1\.0, its nDCG@5 drops to 0\.233,*below*the no\-rewriting baseline \(0\.275\), as high randomness corrupts its already\-ambiguous financial jargon\. This observed correlation between domain formality and optimal temperature, where technical domains favor deterministic generation while informal domains benefit from controlled randomness, confirms that query rewriting demands high precision and motivates the per\-domain temperature configuration adopted in our final system\.

### 5\.2Comparison Studies

#### Query Strategy Comparison\.

We tested two alternatives to simple rewriting\.Domain\-aware promptinginjects domain metadata into the rewriter prompt \(e\.g\.,“This is a technical cloud computing query”\)\. This degraded nDCG@5 to 0\.350,*below*even the no\-rewriting baseline of 0\.371\. The domain context caused the model to over\-specialize queries, narrowing retrieval scope and introducing domain\-specific jargon not present in the original question\.Multi\-query expansiongenerates three query variants and merges their retrieved results\. This also scored 0\.350, as the additional queries diluted the signal from the primary rewrite with noisy alternatives\.

Table[5](https://arxiv.org/html/2605.12028#S5.T5)provides the full comparison\. Both results demonstrate that added complexity in query formulation does not compensate for a well\-tuned simple rewriter\.

Table 5:Query formulation strategy comparison\. Simple rewriting substantially outperforms more complex alternatives\. Both domain\-aware and multi\-query strategies underperform even the no\-rewriting baseline\.

### 5\.3Error Analysis

Manual inspection of failure cases reveals three dominant error patterns\. First,unanswerable queries\(∼\{\\sim\}25% of the dataset\) produce false positives when the system retrieves topically related but non\-relevant passages, a limitation inherent to any retrieval\-only system without answerability prediction\. Second,long conversation histories\(10\+ turns\) may exceed the rewriter’s context window \(2048 tokens\) in production settings; while the current dataset’s conversations remain within this limit, longer real\-world interactions would require truncation of early turns that may contain critical referents\. Third, FiQA is consistently the most challenging domain \(nDCG@5 0\.346 at best\), attributable to its informal forum language, domain\-specific financial jargon \(“401k rollover”,“FIRE movement”\), and abbreviations that create vocabulary mismatch for both lexical and semantic retrievers\.

The substantial gap between development holdout performance \(nDCG@5 0\.422 at best uniform temperature\) and official test set performance \(0\.531\) suggests that domain\-specific temperature tuning and the final pipeline configuration generalize well to unseen conversational patterns, and that our holdout set, which comprises only 22 conversations, may have been a conservative estimate of system capability\.

## 6Conclusion

We presented a three\-stage retrieval pipeline for multi\-turn conversational search that ranks 8th out of 38 systems on the MTRAGEval benchmark of SemEval 2026 Task 8 \- RetrievalRosenthalet al\.\([2026b](https://arxiv.org/html/2605.12028#bib.bib3)\)with nDCG@5 of 0\.531\. Each component \(query rewriting, hybrid retrieval, and cross\-encoder reranking\) is justified through systematic comparison studies\.

Our analysis surfaces two key insights: domain\-specific temperature tuning for query generation yields meaningful gains by adapting to domain vocabulary characteristics, and larger reranking candidate pools counter\-intuitively degrade quality\. Negative results with domain\-aware prompting, multi\-query expansion, and reranker fine\-tuning reinforce that simplicity outperforms complexity when the base components are well\-tuned\.

Current limitations include English\-only support, consumer hardware constraints that limited model sizes, and unweighted rank fusion\. Future work could explore larger rewriter models \(14B\+\), learned fusion weights, ensemble reranking, cross\-lingual extension, and a systematic study of the relationship between domain formality and optimal generation temperature for query rewriting\.

## Acknowledgments

We thank the MTRAGEval organizers for creating a challenging and well\-designed benchmark for multi\-turn retrieval evaluation\.

## References

- M3\-embedding: multi\-linguality, multi\-functionality, multi\-granularity text embeddings through self\-knowledge distillation\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 2318–2335\.External Links:[Link](https://aclanthology.org/2024.findings-acl.137/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.137)Cited by:[§3\.2](https://arxiv.org/html/2605.12028#S3.SS2.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2605.12028#S3.SS3.p1.1)\.
- G\. V\. Cormack, C\. L\. A\. Clarke, and S\. Buettcher \(2009\)Reciprocal rank fusion outperforms Condorcet and individual rank learning methods\.InProceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 758–759\.External Links:[Link](https://doi.org/10.1145/1571941.1572114)Cited by:[§2\.2](https://arxiv.org/html/2605.12028#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2605.12028#S3.SS2.SSS0.Px3.p1.4)\.
- Granite Team, IBM Research AI \(2025\)Granite embedding R2 models\.arXiv preprint arXiv:2508\.21085\.External Links:[Link](https://doi.org/10.48550/arXiv.2508.21085)Cited by:[§3\.3](https://arxiv.org/html/2605.12028#S3.SS3.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§1](https://arxiv.org/html/2605.12028#S1.p3.1),[§3\.1](https://arxiv.org/html/2605.12028#S3.SS1.SSS0.Px1.p1.3)\.
- K\. Järvelin and J\. Kekäläinen \(2002\)Cumulated gain\-based evaluation of IR techniques\.ACM Transactions on Information Systems20\(4\),pp\. 422–446\.External Links:[Link](https://doi.org/10.1145/582415.582418)Cited by:[§2\.1](https://arxiv.org/html/2605.12028#S2.SS1.p1.1),[§4](https://arxiv.org/html/2605.12028#S4.SS0.SSS0.Px2.p1.1)\.
- J\. Johnson, M\. Douze, and H\. Jégou \(2021\)Billion\-scale similarity search with GPUs\.IEEE Transactions on Big Data7\(3\),pp\. 535–547\.External Links:[Link](https://doi.org/10.1109/TBDATA.2019.2921572)Cited by:[§3\.2](https://arxiv.org/html/2605.12028#S3.SS2.SSS0.Px2.p1.1)\.
- Y\. Katsis, S\. Rosenthal, K\. Fadnis, C\. Gunasekara, Y\. Lee, L\. Popa, V\. Shah, H\. Zhu, D\. Contractor, and M\. Danilevsky \(2025\)mtRAG: A Multi\-Turn Conversational Benchmark for Evaluating Retrieval\-Augmented Generation Systems\.Transactions of the Association for Computational Linguistics13,pp\. 784–808\.External Links:ISSN 2307\-387X,[Document](https://dx.doi.org/10.1162/TACL.a.19),[Link](https://doi.org/10.1162/TACL.a.19),https://direct\.mit\.edu/tacl/article\-pdf/doi/10\.1162/TACL\.a\.19/2540217/tacl\.a\.19\.pdfCited by:[§1](https://arxiv.org/html/2605.12028#S1.p1.1)\.
- X\. H\. Lù \(2024\)BM25S: orders of magnitude faster lexical search via eager sparse scoring\.arXiv preprint arXiv:2407\.03618\.External Links:[Link](https://doi.org/10.48550/arXiv.2407.03618)Cited by:[§3\.2](https://arxiv.org/html/2605.12028#S3.SS2.SSS0.Px1.p1.1)\.
- R\. Nogueira and K\. Cho \(2019\)Passage re\-ranking with BERT\.arXiv preprint arXiv:1901\.04085\.External Links:[Link](https://doi.org/10.48550/arXiv.1901.04085)Cited by:[§2\.2](https://arxiv.org/html/2605.12028#S2.SS2.p1.1)\.
- N\. Reimers and I\. Gurevych \(2019\)Sentence\-BERT: sentence embeddings using Siamese BERT\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,pp\. 3982–3992\.External Links:[Link](https://aclanthology.org/D19-1410/)Cited by:[§2\.2](https://arxiv.org/html/2605.12028#S2.SS2.p1.1)\.
- S\. Robertson and H\. Zaragoza \(2009\)The probabilistic relevance framework: BM25 and beyond\.Foundations and Trends in Information Retrieval3\(4\),pp\. 333–389\.Cited by:[§2\.2](https://arxiv.org/html/2605.12028#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2605.12028#S3.SS2.SSS0.Px1.p1.1)\.
- S\. Rosenthal, Y\. Katsis, V\. Shah, L\. He, L\. Popa, and M\. Danilevsky \(2026a\)MTRAG\-UN: A Benchmark for Open Challenges in Multi\-Turn RAG Conversations\.External Links:2602\.23184,[Link](https://arxiv.org/abs/2602.23184)Cited by:[§2\.1](https://arxiv.org/html/2605.12028#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2605.12028#S2.SS1.p2.1)\.
- S\. Rosenthal, V\. Shah, Y\. Katsis, and M\. Danilevsky \(2026b\)SemEval\-2026 Task 8: MTRAGEval: Evaluating Multi\-Turn RAG Conversations\.InProceedings of the 20th International Workshop on Semantic Evaluation \(SemEval\-2026\),San Diego, California\.Cited by:[§1](https://arxiv.org/html/2605.12028#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.12028#S2.SS1.p1.1),[§6](https://arxiv.org/html/2605.12028#S6.p1.1)\.
- S\. Vakulenko, S\. Longpre, Z\. Tu, and R\. Anantha \(2021\)Question rewriting for conversational question answering\.InProceedings of the 14th ACM International Conference on Web Search and Data Mining,pp\. 355–363\.External Links:[Link](https://doi.org/10.1145/3437963.3441748)Cited by:[§2\.2](https://arxiv.org/html/2605.12028#S2.SS2.p1.1)\.
- C\. Van Gysel and M\. de Rijke \(2018\)Pytrec\_eval: an extremely fast python interface to trec\_eval\.InSIGIR ’18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval,pp\. 873–876\.External Links:[Link](https://doi.org/10.1145/3209978.3210065)Cited by:[§4](https://arxiv.org/html/2605.12028#S4.SS0.SSS0.Px2.p1.1)\.
- A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu,et al\.\(2024\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.External Links:[Link](https://doi.org/10.48550/arXiv.2412.15115)Cited by:[§1](https://arxiv.org/html/2605.12028#S1.p3.1),[§3\.1](https://arxiv.org/html/2605.12028#S3.SS1.SSS0.Px1.p1.3)\.

## Appendix AQuery Rewriting Prompt

The following system prompt is used for query rewriting\. The user message concatenates up to 10 turns of conversation history followed by the current question\.

> You are a query rewriting assistant for information retrieval\. Given a conversation history and a current question, rewrite the question to be completely standalone and self\-contained\. Rules: 1\. Resolve all pronouns \(it, they, this, that\) to their explicit referents 2\. Include relevant context from the conversation that’s needed to understand the query 3\. Keep the rewritten query concise and search\-friendly 4\. Do not add information not present in the conversation 5\. If the question is already standalone, return it unchanged

## Appendix BHyperparameters

Table[6](https://arxiv.org/html/2605.12028#A2.T6)lists all hyperparameters needed to reproduce our system\.

Table 6:Complete hyperparameter configuration for the final submission\.