Candidate-Constrained Retrieval-Augmented Generation for LongEval-RAG: System Design and Empirical Analysis

# LongEval-RAG at CLEF 2026
Source: [https://arxiv.org/html/2607.04008](https://arxiv.org/html/2607.04008)
\\copyrightclause

Copyright for this paper by its authors\. Use permitted under Creative Commons License Attribution 4\.0 International \(CC BY 4\.0\)\.

\\conference

CLEF 2026 Working Notes, 21 – 24 September 2026, Jena, Germany

\[orcid=0009\-0007\-5546\-5293, email=yingdongyang0305@outlook\.com, url=https://github\.com/yyd859/, \]\\fnmark\[1\]\\cormark\[1\]

\[orcid=0009\-0007\-0311\-6235, email=willwuhj@gmail\.com, url=https://wijowill\.github\.io/, \]\\fnmark\[1\]

\\cortext

\[1\]Corresponding author\.\\fntext\[1\]These authors contributed equally and share first authorship\.

\(2026\)

###### Abstract

We present a candidate\-constrained retrieval\-augmented generation system for LongEval\-RAG, where each query is associated with an organizer\-provided candidate set and all retrieved evidence and final citations must remain within that set\. The system combines deterministic provenance tracking with passage\-based retrieval, deterministic query expansion, pseudo\-relevance feedback \(PRF\), reciprocal rank fusion \(RRF\), lightweight evidence reranking, citation\-aware evidence aggregation, and optional MiniLM sentence reranking\. We evaluate ten pipeline variants using a primary organizer evaluation and a supplementary self\-generated diagnostic protocol\. The primary evaluation shows that the strongest balanced variant isrule\-minilm: a rule\-based chunking pipeline with query expansion, PRF, RRF, reranking, citation prior, and late MiniLM sentence selection\. This variant obtains the highest BERTScore, retrieval precision, nugget coverage, and average grade among our submissions\. The result suggests that the main gain does not come from more complex semantic or topic\-shift chunking, but from pairing stable rule\-based passages with sentence\-level neural selection before generation\. The supplementary LLM\-judge evaluation remains useful for early diagnosis and additional analysis, but it emphasizes different systems than the primary gold\-answer and nugget\-based evaluation, highlighting the need for multi\-metric RAG evaluation\.

###### keywords:

Retrieval\-augmented generation\\sepLongEval\-RAG\\sepEvidence ranking\\sepCitation\-grounded generation

## 1Introduction

LongEval\-RAG studies answer generation from an organizer\-provided candidate set rather than unrestricted corpus search\[longeval2026task4,lncs\_overview\_longeval\_2026,ceur\_overview\_longeval\_2026\]\. This setting differs from open\-domain RAG, where retrieval is typically performed against a large external corpus such as Wikipedia\[lewis2020rag,gao2023rag\_survey\]\. Instead, each query is paired with an official list of candidate document IDs, and every retrieved passage, selected evidence item, and final citation must remain inside that list\. In our system, the main retrieval\-time selection unit is the passage: the pipeline ranks and selects passages derived from candidate documents, then selects sentence\-level evidence from those passages for answer generation, while the finalreferenceslist is derived from the source document IDs of the selected evidence\.

This constraint makes the task especially suitable for analyzing the internal quality of a RAG system\. Since broad corpus recall is removed from the problem definition, performance depends more directly on how well the system structures documents into passages, ranks evidence, allocates answer space, and preserves provenance from final claims back to source documents\.

This paper makes three main contributions\. First, we describe a complete candidate\-constrained RAG pipeline for LongEval\-RAG, covering data preparation, passage construction, retrieval, evidence selection, answer generation, and deterministic provenance tracking\. Second, we report an ablation study of ten pipeline variants under the primary multi\-metric organizer evaluation, while retaining a supplementary cross\-provider LLM\-judge protocol as a diagnostic view\. Third, we identify the strongest primary\-evaluation configuration:rule\-minilm, which combines deterministic rule\-based chunks with late MiniLM sentence reranking and achieves the best balance of semantic answer similarity, retrieval precision, nugget coverage, and average nugget grade among our submissions\.

The primary evaluation points to a clear empirical message\. Query expansion, PRF, and RRF remain important because they broaden lexical evidence recall inside the fixed candidate pool, following Query2doc\-style expansion\[wang2023query2doc\], relevance\-model ideas\[lavrenko2001relevance\], and reciprocal rank fusion\[cormack2009rrf\]\. The strongest primary\-evaluation configuration,rule\-minilm, builds on this retrieval foundation by keeping stable rule\-based passages and applying MiniLM only at the late sentence\-selection stage\. This suggests that, in the current setup, neural matching is most useful for allocating the final evidence budget rather than for redefining upstream chunk boundaries with semantic or topic\-shift heuristics\. The supplementary LLM\-judge evaluation assigns its highest mean score torrf\_no\_rerank, but we treat that difference as a diagnostic evaluation disagreement rather than as the main empirical conclusion\.

## 2Task Setting and Data Preparation

### 2\.1Query and Document Inputs

The query input is stored as JSONL, with one query per line\. Each record contains a query identifier, the query text, and the official candidate document IDs for that query\. The implementation also supports several field aliases for robustness, for example mapping identifiers from fields such asnarrative\_id,query\_id,qid, orid, and mapping query text from fields such asnarrative,query,text, orquestion\.

The current evaluation slice contains 47 queries\. Each query is paired with exactly 10 official candidate documents, yielding 470 query\-document instances in total\. Thus, the candidate count is fixed across the slice rather than varying by query\.

Document input is loaded from JSONL, JSON, CSV, TSV, or a directory containing those formats\. Each document record should include a document ID, a title, and a usable text field, preferablyfullText, along with an optional publication timestamp\. The loader likewise supports multiple field aliases for document IDs\.

### 2\.2Candidate\-Constrained Join Logic

The entire pipeline is candidate\-constrained by design\. The loading process first reads all query records, extracts the candidatedoc\_idsassociated with each query, collects the union of required document IDs, and loads only those documents from the document collection\. The pipeline then attaches documents back to each query in the order specified by its candidate list\.

This design has an important methodological consequence: the system never retrieves outside the official candidate set\. All later stages, including passage construction, retrieval, reranking, answer generation, and evaluation, operate only over the provided candidate pool\.

### 2\.3Text Selection and Provenance Tracking

For text composition, the loader follows a simple priority rule:

fullText→abstract→title\.\\texttt\{fullText\}\\rightarrow\\texttt\{abstract\}\\rightarrow\\texttt\{title\}\.Whenever full text is available, it is used as the main evidence source\. If full text is missing, the system falls back to the abstract, and then to the title if necessary\. This fallback is an engineering decision for data availability rather than a preferred scientific choice\.

Table[1](https://arxiv.org/html/2607.04008#S2.T1)reports text availability for the current 47\-query slice\. Among the 470 query\-document instances, 222 instances use full text, 248 use abstract\-only fallback, and none require title\-only fallback\. No candidate documents are dropped for missing text in this slice\.

Table 1:Text availability in the current evaluation slice\. Percentages are computed over 470 query\-document instances\.Text source usedCountPercentagefullText22247\.2%Abstract only24852\.8%Title only00\.0%Timestamps are loaded and preserved as document metadata\. In the retrieval configurations reported here, they can also serve as active retrieval\-side signals when the query has temporal intent\. Specifically, the evidence reranker includes a configurable temporal boost, evidence selection can encourage coverage of earlier and later documents for temporally phrased queries, and the extractive generator can construct temporal comparison sentences from timestamp\-derived years\. However, timestamps are not exposed directly in the final batch LLM prompts for theopenai\_batch\_gpt54\_top10evaluation inputs\.

A central implementation feature is deterministic provenance tracking\. After retrieval and evidence selection, the system converts the source document IDs of selected passages into an ordered cited\-document list, stored in the output fieldreferences\. Final answer sentences do not cite raw document IDs directly\. Instead, each sentence stores citation indices into thisreferenceslist\. The reconstruction path is therefore:

citation index→references\[index\]→document ID\.\\text\{citation index\}\\rightarrow\\texttt\{references\[index\]\}\\rightarrow\\text\{document ID\}\.For batch generation, the pipeline adds anevidence\_idlayer so that asynchronous model outputs can still be mapped back to sentence candidates, citation indices, and finally source documents\. This engineering choice makes every cited sentence in the final JSONL output traceable to its supporting document, in the same general spirit as recent citation\-grounded generation setups\[gao2023alce\]\.

### 2\.4Terminology for Text and Evaluation Units

We use five terms consistently throughout the paper\. They are not interchangeable: documents define the candidate set, chunks describe segmentation, passages are retrieval units,referencesrecords cited document IDs, and nuggets are evaluation units\.

Table 2:Core terminology used for retrieval, provenance, and evaluation\.TermDefinitionRole in the paperDocumentOne organizer\-provided candidate document ID for queryqq, denotedd∈𝒟qd\\in\\mathcal\{D\}\_\{q\}Defines the fixed candidate set; every retrieved passage, prompt item, and final citation must trace back to one of these documents\.ChunkA contiguous group of sentences produced by rule\-based, semantic, or topic\-shift segmentationDescribes how a document is split before indexing; chunking strategy is one of the main ablation dimensions\.PassageThe indexed retrieval unit emitted from a chunk, with text, source document ID, title, timestamp metadata, and sentence offsetsUnit scored by BM25, RRF, reranking, and passage\-level selection before prompt construction\.ReferenceOrdered, deduplicated list of source document IDs from selected passagesOutput field used by citation indices; distinct from the bibliography of cited papers\.NuggetMinimal information unit expected in a correct answerEvaluation unit used by organizer metrics such as nugget coverage and average grade\.

## 3System Pipeline

### 3\.1Overview and Notation

For a queryqq, let𝒟q=\{d1,…,d10\}\\mathcal\{D\}\_\{q\}=\\\{d\_\{1\},\\ldots,d\_\{10\}\\\}denote the organizer\-provided candidate document set\. The pipeline first normalizes these documents into passages and then performs retrieval only over

𝒫q=⋃d∈𝒟qchunk⁡\(d\),\\mathcal\{P\}\_\{q\}=\\bigcup\_\{d\\in\\mathcal\{D\}\_\{q\}\}\\operatorname\{chunk\}\(d\),where every passagep∈𝒫qp\\in\\mathcal\{P\}\_\{q\}stores its text, source documentd​\(p\)d\(p\), title, timestamp when available, and sentence offsets\. The candidate constraint is therefore enforced before ranking: no passage outside𝒟q\\mathcal\{D\}\_\{q\}can enter retrieval, reranking, evidence selection, or the final prompt\.

Table[3](https://arxiv.org/html/2607.04008#S3.T3)summarizes the concrete constants used by the reported configurations\. These are the experiment\-level parameters exposed in the current pipeline\. BM25 is used as a lexical ranker, but downstream RRF consumes ranks rather than raw BM25 magnitudes, so BM25 scores are not treated as cross\-component weights in the ablation analysis\.

Table 3:Main implementation parameters for the reported system pipeline\.StageParameterValue / behaviorRule chunkingpassage\_max\_words140 wordspassage\_stride\_sentences1 sentence overlapSemantic chunkingEmbedding modelall\-MiniLM\-L6\-v2semantic\_merge\_threshold0\.72 cosine similarityTopic\-shift chunkingtopic\_shift\_boundary\_threshold0\.18 adjacent\-sentence driftSemantic/topic chunkingMinimum sentences before split1 sentenceMaximum sentence countunset; only word budget appliesLexical retrievalBM25 baselineoriginal\-query passage rankingBM25 parametersk1=1\.5k\_\{1\}=1\.5,b=0\.75b=0\.75; not tunedQuery expansionExtracted query keywordsup to 12 unique non\-stopword tokensquery\_variant\_limit5 variants including the original queryPRFprf\_top\_passagestop 4 original\-query BM25 passagesFeedback termstop 6 raw\-frequency termsFusionRRF constantk=60k=60Rerankingpreselect\_k24 passagesEvidence selectiontop\_k\_passages20 passagesper\_doc\_limitunset; no hard per\-document capCitation priorcitation\_graph\_boost0\.05 additive weightPrompt constructionCandidate sentences per passageup to first 5 sentencesmax\_selected\_answer\_candidates10 sentence\-level evidence itemsSentence rerankingCross\-encoder modelcross\-encoder/ms\-marco\-MiniLM\-L\-6\-v2Answer generationmax\_answer\_sentences5 answer sentences

### 3\.2Passage Construction

The default chunker is a deterministic overlapping sentence\-window procedure\. Let a document be represented as a sentence sequence\(s1,…,sn\)\(s\_\{1\},\\ldots,s\_\{n\}\), and letw​\(C\)w\(C\)be the word count of a candidate chunkCC\. The rule\-based chunker scans left to right, greedily appending the next sentence while

w​\(C∪\{si\}\)≤140\.w\(C\\cup\\\{s\_\{i\}\\\}\)\\leq 140\.When the next sentence would exceed the limit, the current chunk is emitted as a passage\. The next chunk is initialized with the final sentence of the previous chunk, giving a one\-sentence overlap\. The 140\-word value is used as a shared upper bound across the reported rule\-based, semantic, and topic\-shift variants, but it is not tuned in this study\. We therefore treat passage length as a hyperparameter for future work, where it should be evaluated together with the resulting chunk\-length distribution and downstream retrieval quality\.

The semantic chunking variant replaces fixed sentence windows with embedding\-based merging\. Each sentencesis\_\{i\}is embedded withall\-MiniLM\-L6\-v2\. For a current chunkCC, the chunk embedding is the mean of its sentence embeddings:

e¯​\(C\)=1\|C\|​∑si∈Ce​\(si\)\.\\bar\{e\}\(C\)=\\frac\{1\}\{\|C\|\}\\sum\_\{s\_\{i\}\\in C\}e\(s\_\{i\}\)\.A candidate sentencesjs\_\{j\}is merged intoCConly when both constraints hold:

cos⁡\(e​\(sj\),e¯​\(C\)\)≥0\.72andw​\(C∪\{sj\}\)≤140\.\\cos\\bigl\(e\(s\_\{j\}\),\\bar\{e\}\(C\)\\bigr\)\\geq 0\.72\\quad\\text\{and\}\\quad w\(C\\cup\\\{s\_\{j\}\\\}\)\\leq 140\.Otherwise,CCis emitted and a new chunk begins atsjs\_\{j\}\. The minimum number of sentences before a similarity\-based split is one, and no additional maximum sentence\-count cap is active in the reported semantic runs\.

The topic\-shift variant uses the same sentence embeddings but detects local drift between adjacent sentences\. Fori\>1i\>1, define

Δi=1−cos⁡\(e​\(si−1\),e​\(si\)\)\.\\Delta\_\{i\}=1\-\\cos\\bigl\(e\(s\_\{i\-1\}\),e\(s\_\{i\}\)\\bigr\)\.A new chunk starts beforesis\_\{i\}when the current chunk already contains at least one sentence andΔi\>0\.18\\Delta\_\{i\}\>0\.18, or when addingsis\_\{i\}would exceed the 140\-word limit\. This gives a lightweight topic\-boundary heuristic related to TextTiling\-style segmentation\[hearst1997texttiling\], while preserving the same size budget as the rule\-based chunker\.

### 3\.3Lexical Ranking and Query Variants

The single\-query baseline ranks passages with BM25 over the original query text\[robertson2009bm25\]\. For query variantvvand passagepp, the BM25 scoring form is

BM25⁡\(v,p\)=∑t∈vIDF⁡\(t\)​f​\(t,p\)​\(k1\+1\)f​\(t,p\)\+k1​\(1−b\+b​\|p\|avgdl\),\\operatorname\{BM25\}\(v,p\)=\\sum\_\{t\\in v\}\\operatorname\{IDF\}\(t\)\\frac\{f\(t,p\)\(k\_\{1\}\+1\)\}\{f\(t,p\)\+k\_\{1\}\\left\(1\-b\+b\\frac\{\|p\|\}\{\\operatorname\{avgdl\}\}\\right\)\},wheref​\(t,p\)f\(t,p\)is the frequency of termttin passagepp,\|p\|\|p\|is passage length,avgdl\\operatorname\{avgdl\}is the average passage length in𝒫q\\mathcal\{P\}\_\{q\}, and the reported runs usek1=1\.5k\_\{1\}=1\.5andb=0\.75b=0\.75\. These BM25 parameters are implementation constants rather than tuned hyperparameters\. The reported ablations use BM25 to define ranked lists; once lists are produced, fusion depends on rank positions rather than raw BM25 scores\.

Deterministic query expansion is fully local\. It uses no LLM, external dictionary, or retrieval\-derived text\. The system tokenizes the original query narrative, lowercases tokens, removes stopwords and very short tokens, keeps only tokens with length at least three, deduplicates them by first occurrence, and preserves their original query order\. It then keeps at most 12 such tokens as the query\-keyword set\. From this set, the system can construct a keyword\-focused variant, a temporal\-intent variant if the tokens overlap a small hand\-written temporal\-intent list, and a method\-intent variant if the tokens overlap a small hand\-written method\-intent list\. The total number of query variants is capped at five including the original query\.

Pseudo\-relevance feedback is a separate retrieval\-derived expansion step\. LetFqF\_\{q\}be the four highest\-ranked passages under original\-query BM25\. A candidate feedback term must have length at least four, must not be a stopword, and must not already appear in the extracted query\-keyword set\. Its score is raw frequency in the feedback pool:

PRF⁡\(t,q\)=∑p∈Fqf​\(t,p\)\.\\operatorname\{PRF\}\(t,q\)=\\sum\_\{p\\in F\_\{q\}\}f\(t,p\)\.The top six terms under this score are appended only to the query\-keyword set to form one additional PRF query variant\. They are not added back into the original, keyword, temporal, or method variants\. This means PRF contributes one separate BM25 ranked list to RRF rather than changing all query variants\. The separation keeps query\-only expansion and retrieval\-derived expansion distinguishable: original and deterministic variants reflect only the user query, while the PRF variant reflects terms found in the first\-pass feedback passages\.

### 3\.4Rank Fusion

The multi\-query systems retrieve one lexical ranked list for each enabled variant: the original query, keyword query, optional temporal query, optional method query, and optional PRF query\. Dense retrieval is supported by the implementation, but the reported Task 4 runs leave the dense model disabled, so the pre\-reranking fusion is lexical\-only\. The ranked lists are merged with reciprocal rank fusion\[cormack2009rrf\]:

SRRF​\(p∣q\)=∑r∈ℛq160\+rankr⁡\(p\),S\_\{\\operatorname\{RRF\}\}\(p\\mid q\)=\\sum\_\{r\\in\\mathcal\{R\}\_\{q\}\}\\frac\{1\}\{60\+\\operatorname\{rank\}\_\{r\}\(p\)\},whereℛq\\mathcal\{R\}\_\{q\}is the set of enabled ranked lists for queryqq\. If passageppis absent from a list, that list contributes zero\. The only RRF hyperparameter is the denominator offsetk=60k=60\.

### 3\.5Reranking and Citation Prior

Reranked variants take the top 24 passages underSRRFS\_\{\\operatorname\{RRF\}\}as a shortlist\. The reranker is deliberately lightweight: it preserves the RRF score as the base ranking signal and adds a small document\-level citation prior\. For documentdd, letc​\(d\)c\(d\)be the number of edges in the provided citation network in whichddappears either asciting\_doc\_idor ascited\_doc\_id\. The normalized citation prior is

G​\(d\)=c​\(d\)maxd′⁡c​\(d′\),G\(d\)=\\frac\{c\(d\)\}\{\\max\_\{d^\{\\prime\}\}c\(d^\{\\prime\}\)\},soG​\(d\)∈\[0,1\]G\(d\)\\in\[0,1\]\. For a passageppfrom documentd​\(p\)d\(p\), the configured additive citation term is

0\.05​G​\(d​\(p\)\)\.0\.05\\,G\(d\(p\)\)\.Thus the explicitly configured additive score used by the main reranked variants is

Srerank​\(p∣q\)=SRRF​\(p∣q\)\+0\.05​G​\(d​\(p\)\)\+T​\(q,p\),S\_\{\\operatorname\{rerank\}\}\(p\\mid q\)=S\_\{\\operatorname\{RRF\}\}\(p\\mid q\)\+0\.05\\,G\(d\(p\)\)\+T\(q,p\),whereT​\(q,p\)T\(q,p\)denotes the temporal adjustment applied only for temporally phrased queries with usable document timestamps\. The timestamp signal is used on the retrieval side to encourage earlier/later coverage, but timestamps are not printed in the final batch prompts\.

### 3\.6Evidence Selection and Prompt Construction

After reranking, the system selects 20 passages per query\. The main configurations do not impose a hard per\-document quota: a document can contribute multiple passages if they remain highly ranked\. The selector filters near\-duplicate passages and, for temporal queries, may force coverage of the earliest and latest dated documents before filling the remaining passage slots from the ranked list\. The source document IDs of the selected passages are then converted into the orderedreferenceslist, deduplicated by first occurrence\.

The answer\-generation prompt uses a smaller sentence\-level evidence budget\. The generator extracts up to the first five candidate sentences from each selected passage, filters and scores those sentences, and keeps at most 10 sentence\-level evidence items\. MiniLM sentence\-reranking variants insert an additional cross\-encoder scoring step over these sentence candidates; non\-MiniLM variants retain the deterministic order induced by passage selection\. Each sentence\-level evidence item carries anevidence\_idand citation indices into thereferenceslist\.

Therule\-minilmvariant is methodologically important because it separates passage construction from neural sentence selection\. Passage boundaries remain deterministic and overlap\-preserving, while MiniLM is used only after candidate evidence has been narrowed to sentence\-level answer candidates\. This design reduces the risk that embedding\-based segmentation will discard useful local context, while still allowing a neural model to prioritize sentences that are semantically close to the query\.

Unless otherwise noted, all answer generation runs use OpenAIgpt\-5\.4\-mini\. The model receives only the selected candidate\-constrained evidence items and is limited to at most five answer sentences\. After batch generation, returnedevidence\_idvalues are mapped back to sentence candidates, citation indices, and source document IDs\. This preserves the candidate constraint at the prompt level and makes every final citation traceable to an official candidate document\.

## 4Experimental Variants

Table[4](https://arxiv.org/html/2607.04008#S4.T4)summarizes the ten experimental variants currently tracked in the repository\. Together, they serve three purposes: retrieval\-strength ablation, chunking\-strategy comparison, and sentence\-selection comparison\.

Table 4:Method matrix for the current LongEval Task 4 experiments\. Abbreviations: Rule = rule chunking, Sem\. = semantic chunking, Topic = topic\-shift chunking, BM25 = single\-query BM25, QE = query expansion, RR = evidence reranking, Cit\. = citation prior, Sent\. = MiniLM sentence reranking\.No\.ApproachRuleSem\.TopicBM25QEPRFRRFRRCit\.Sent\.1concat\-baseline✓\\checkmark–––––––––2single\-query\-bm25✓\\checkmark––✓\\checkmark––––––3rrf\-no\-rerank✓\\checkmark–––✓\\checkmark✓\\checkmark✓\\checkmark–––4caes\-rag\-rrf∗✓\\checkmark–––✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark–5default∗✓\\checkmark–––✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark–6rule\-minilm✓\\checkmark–––✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark7semantic\-current–✓\\checkmark––✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark–8topic\-shift\-current––✓\\checkmark–✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark–9semantic\-minilm–✓\\checkmark––✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark10topic\-shift\-minilm––✓\\checkmark–✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark∗caes\-rag\-rrfanddefaultuse the same effective setting and serve as a control pair\.

Table[4](https://arxiv.org/html/2607.04008#S4.T4)should be read together with the following role summary\.concat\_baselineis the lower bound,single\_query\_bm25is the lexical baseline, andrrf\_no\_rerankisolates the gain from multi\-query retrieval\.caes\_rag\_rrfanddefaultform a control pair because they use the same effective setting: rule\-based chunks, query expansion, PRF, RRF, evidence reranking, and citation prior, without MiniLM sentence reranking\. The semantic and topic\-shift variants are chunking ablations, while the MiniLM variants test late sentence\-level reranking after passage retrieval\.

The terminology is important because MiniLM appears in two different places\. In thesemanticandtopic\-shiftchunking families,all\-MiniLM\-L6\-v2embeddings define or detect chunk boundaries\. In contrast, theminilmsuffix denotes a later cross\-encoder sentence reranker applied to already selected sentence candidates;currentdenotes the corresponding configuration without that sentence reranker\. Thus,rule\-minilmkeeps rule\-based chunks and adds neural scoring only at sentence allocation time\.

Interpreting the matrix at a high level, the progressionconcat\_baseline→\\rightarrowsingle\_query\_bm25→\\rightarrowrrf\_no\_rerank→\\rightarrowcaes\_rag\_rrfserves as a staged comparison from no retrieval, to single\-query lexical retrieval, to multi\-query lexical fusion, and then to the full reranked variant\. The semantic and topic\-shift families test whether more sophisticated segmentation yields better retrieval passages, while the MiniLM variants test whether a compact Transformer model can serve as an additional reranker after retrieval\[wang2020minilm,nogueira2019passage\_rerank\]\.

## 5Evaluation Protocols

We use two evaluation protocols with different roles\. Theprimary evaluationis the official organizer evaluation and provides the main empirical evidence in the paper\. Thesupplementary evaluationis our self\-generated cross\-provider diagnostic protocol\. It was developed much earlier than the official release, so it allowed us to compare candidate models, estimate rough system quality, and check usability and stability during system development\.

### 5\.1Primary Organizer Evaluation

The primary evaluation is the organizer\-released evaluation\. The organizer report provides a broad evaluation package rather than a single score: it includes 33 available measures and recommends a compact subset that captures complementary aspects of retrieval\-augmented generation\. We report the measures that are most useful for interpreting our systems: ROUGE\-L, BERTScore, retrieval precision, nugget coverage, average grade, and TFC1\.

These metrics evaluate different failure modes\. ROUGE\-L is reported asrougeL\_f1: it computes longest\-common\-subsequence overlap between the generated answer and the organizer\-provided gold answer, then reports the harmonic mean of ROUGE\-L precision and recall\[lin2004rouge\]\. BERTScore is reported asbertscore\_f1: generated\-answer tokens and gold\-answer tokens are matched with contextual embeddings, and the final value is the harmonic mean of BERTScore precision and recall\[zhang2020bertscore\]\. Retrieval precision is the fraction of cited documents that are judged relevant to the query, so a run that cites only relevant documents receives precision 1\.0 and a run that cites only non\-relevant documents receives 0\.0\. Nugget coverage is the recall of minimal information nuggets covered by the response, and average grade is the mean rubric grade assigned to those nuggets; the organizer implementation uses TREC\-AutoJudge\-style nugget evaluation, where nuggets are derived from preference comparisons and then judged for response coverage\[dietz2026nugget\_banks,farzi2026autojudge\]\. TFC1 is an axiomatic RAG measure that gives preference to responses with more occurrences of query terms, following the RAG axiom framework implemented in TREC\-AutoJudge\[merker2025rag\_axioms,farzi2026autojudge\]\.

We use these metrics together because each one isolates a different part of the RAG pipeline\. A system can have good ROUGE\-L by matching surface wording without using the best evidence; it can have high retrieval precision while still omitting important answer nuggets; and it can improve query\-term concentration without improving semantic completeness\. Therefore, the primary evaluation is not treated as a simple leaderboard, but as a multi\-view diagnostic of answer similarity, citation quality, nugget coverage, and query\-focused evidence use\.

This interpretation is especially important for LongEval\-RAG because all runs operate inside the same fixed candidate\-document set\. Since broad corpus recall is not the main variable, differences between systems are more likely to reflect passage construction, evidence allocation, sentence selection, and citation behavior\. For this reason, we treat the primary metrics as the main empirical basis for comparing variants, while using the supplementary protocol to explain disagreements and inspect provenance behavior\.

### 5\.2Supplementary Diagnostic Evaluation

The supplementary evaluation is an internal diagnostic protocol that we built before the primary organizer results were available\. Its first purpose was practical: it gave us an early way to compare variants, select promising configurations, and monitor whether the end\-to\-end system was usable and stable before the organizer\-side gold\-answer and nugget\-based evaluation was released\.

Its second purpose is complementary measurement\. As summarized in Figure[1](https://arxiv.org/html/2607.04008#S5.F1), the supplementary protocol generates a reference\-style answer with document\-ID attribution, generates a candidate\-constrained RAG answer, scores the answer with a separate LLM judge \(Claude Sonnet 4\.5\) and computes document\-ID overlap diagnostics\. These outputs expose dimensions that are useful for development and error analysis, including completeness, correctness, citation\-set overlap, reference\-subset matching, unsupported citations, and provenance consistency\.

Table[6](https://arxiv.org/html/2607.04008#S7.T6)reports six supplementary metrics, each averaged over the 47\-query slice\.Corr\.is the Claude Sonnet 4\.5 judge’s correctness score: it measures whether the RAG answer is factually consistent with the reference answer and available evidence, without unsupported claims\.Comp\.is the same judge’s completeness score: it measures how much of the reference answer’s important information is covered\. Both judge scores use the same bounded rubric, andAvg\.is their arithmetic mean,\(Corr\.\+Comp\.\)/2\(\\mathrm\{Corr\.\}\+\\mathrm\{Comp\.\}\)/2\. This follows the general LLM\-as\-judge setup used in recent generation\-evaluation work, while retaining a cross\-provider separation between answer generator and judge\[zheng2023llm\_judge,liu2023geval\]\. LetGGbe the reference\-side cited document set andRRbe the RAG answer’s cited document set\.Recallis\|G∩R\|/\|G\|\|G\\cap R\|/\|G\|,Precisionis\|G∩R\|/\|R\|\|G\\cap R\|/\|R\|, andJaccardis\|G∩R\|/\|G∪R\|\|G\\cap R\|/\|G\\cup R\|, the standard Jaccard set\-overlap coefficient\[jaccard1901distribution\]\. These citation\-set metrics are diagnostic rather than final answer\-quality measures: they indicate whether the system grounds its answer in documents similar to those used by the reference side\.

Its third purpose is evaluation analysis\. By comparing the supplementary results with the primary results, we can analyze both the systems and the evaluation protocols themselves\. On the system side, the comparison shows which variants look strong under a fluency\- and completeness\-oriented local judge but weaker under gold\-answer, retrieval\-precision, and nugget\-based measures\. On the evaluation side, the disagreement helps reveal what each protocol rewards, where LLM\-as\-judge evaluation may overvalue plausible synthesis, and why RAG systems should be discussed with multiple complementary metrics rather than a single score\.

![Refer to caption](https://arxiv.org/html/2607.04008v1/assets/cross_provider_evaluation_pipeline.png)Figure 1:Overall pipeline used in our supplementary LongEval Task 4 experiments, covering candidate\-constrained retrieval, answer generation, and cross\-provider evaluation\.

## 6Primary Results and Analysis

Table[5](https://arxiv.org/html/2607.04008#S6.T5)reports the primary organizer scores for the submitted runs\. We keep this view distinct from the supplementary evaluation:*primary evaluation*refers to organizer\-released metrics, whereas*supplementary evaluation*refers to our self\-generated cross\-provider judge and overlap diagnostics\.

Table 5:Primary organizer evaluation for Task 4 of LongEval\. We report the F1score of ROUGE\-L and BERTScore as primary measures\. Additionally, we report retrieval precision, nugget coverage, average grade, and TFC1\. A dash indicates that the corresponding value was not present in the local organizer\-results export used for this table\.No\.ApproachROUGE\-LBERTScorePrecisionNugget Cov\.Avg\. GradeTFC11rule\-minilm0\.1570\.1680\.9750\.3672\.9290\.3052topic\-shift\-minilm0\.1570\.1630\.9400\.2882\.6710\.3733semantic\-minilm0\.1600\.1570\.9220\.2912\.6960\.3494topic\-shift\-current0\.1670\.1560\.9290\.2922\.6940\.2155semantic\-current0\.1560\.1500\.9330\.2822\.6510\.2626single\-query\-bm250\.1560\.1520\.9400\.3082\.7440\.1207rrf\-no\-rerank0\.1550\.1510\.9290\.3362\.8160\.0878caes\-rag\-rrf0\.1550\.1440\.9420\.3382\.8300\.0599default0\.1470\.1460\.9310\.3342\.8450\.02510official\-naive\-baseline0\.082\-0\.1180\.2000\.1241\.828\-0\.758

For clarity, each Table[5](https://arxiv.org/html/2607.04008#S6.T5)metric is computed as follows\. ROUGE\-L isrougeL\_f1, the F1form of longest\-common\-subsequence overlap with the gold answer\[lin2004rouge\]\. BERTScore isbertscore\_f1, the F1form of contextual\-embedding token matching with the gold answer\[zhang2020bertscore\]\. Precision is cited\-document precision, nugget coverage is covered\-nugget recall, average grade is the mean nugget rubric grade, and TFC1 is the query\-term\-frequency RAG axiom from the organizer’s TREC\-AutoJudge\-style evaluation\[dietz2026nugget\_banks,farzi2026autojudge,merker2025rag\_axioms\]\.

All submitted retrieval\-based systems substantially outperform the official naive baseline, especially on retrieval precision\. The main analysis therefore focuses less on whether retrieval is useful at all and more on which evidence\-allocation strategy works best inside the fixed candidate pool\.

### 6\.1Primary Evaluation Favors Rule\-Based Chunking With Late MiniLM Reranking

The strongest balanced primary\-evaluation run isrule\-minilm\. It achieves the best BERTScore \(0\.168\), retrieval precision \(0\.975\), nugget coverage \(0\.367\), and average grade \(2\.929\) among our submitted systems\. Its ROUGE\-L score is not the highest, but its advantage across semantic similarity, cited\-evidence precision, and nugget\-oriented measures makes it the most convincing overall configuration under the primary protocol\.

This result suggests that the best intervention in the current pipeline is not upstream embedding\-based segmentation\. Instead, the most effective configuration keeps deterministic rule\-based chunks, uses query expansion, PRF, RRF, lightweight reranking, and a citation prior to form a strong candidate pool, and then applies MiniLM late to allocate the final sentence\-level evidence budget\.

### 6\.2Different Primary Metrics Reward Different Behaviors

The primary metrics do not identify a single winner on every axis\.topic\-shift\-currentobtains the highest ROUGE\-L score \(0\.167\), suggesting that topic\-shift chunking can improve surface lexical overlap with the reference answer\.topic\-shift\-minilmobtains the highest TFC1 score \(0\.373\), suggesting stronger query\-term or factual\-constraint concentration in selected evidence\. However, neither run matchesrule\-minilmon the combination of BERTScore, retrieval precision, nugget coverage, and average grade\.

This metric spread is useful because it shows that end\-to\-end RAG quality should not be reduced to a single lexical\-overlap number\. Lexical similarity, semantic similarity, cited\-evidence precision, nugget coverage, and query\-term faithfulness each expose a different part of the pipeline\.

### 6\.3Current vs\. MiniLM: Late Sentence Reranking Is Helpful But Not Uniform

Within the semantic\-chunking family, adding MiniLM sentence reranking improves ROUGE\-L from 0\.156 to 0\.160, BERTScore from 0\.150 to 0\.157, nugget coverage from 0\.282 to 0\.291, and TFC1 from 0\.262 to 0\.349, while retrieval precision decreases from 0\.933 to 0\.922\. This pattern suggests that MiniLM improves answer\-content matching and query\-term evidence, but can select from a slightly less precise cited\-document set\.

Within the topic\-shift family, MiniLM improves BERTScore from 0\.156 to 0\.163, retrieval precision from 0\.929 to 0\.940, and TFC1 from 0\.215 to 0\.373, but ROUGE\-L decreases from 0\.167 to 0\.157 and nugget coverage is roughly similar, moving from 0\.292 to 0\.288\. This suggests that MiniLM changes the focus and wording of selected evidence rather than uniformly improving every metric\.

The rule\-based family gives the clearest positive case for late sentence reranking\.rule\-minilmexceeds bothrrf\-no\-rerankandcaes\-rag\-rrfon BERTScore, retrieval precision, nugget coverage, average grade, and TFC1, while remaining close on ROUGE\-L\. This points to sentence\-level allocation as a more effective intervention than redefining all chunks with embedding similarity in the current candidate\-constrained setting\.

### 6\.4ROUGE\-L Is Limited in Same\-Retrieval Comparisons

ROUGE\-L remains useful as an end\-to\-end surface\-form similarity metric, but it should not be interpreted as a clean measure of retrieval or evidence\-selection quality\. For example,defaultandcaes\-rag\-rrfuse effectively the same retrieval stack: rule\-based chunking, multi\-query expansion, PRF, RRF fusion, lightweight evidence reranking, and citation prior\. Differences between them in ROUGE\-L are therefore likely driven mainly by final answer wording rather than by genuinely different retrieval behavior\.

This observation supports a broader methodological point: in same\-retrieval comparisons, ROUGE\-L variation can overstate generation\-style differences\. For analyzing core RAG capability, especially grounding and evidence quality, retrieval precision, nugget coverage, and average grade are more informative than ROUGE\-L alone\.

## 7Supplementary Diagnostic Results

We now report the supplementary Claude Sonnet 4\.5 judge evaluation as a diagnostic view rather than as the primary result\. Table[6](https://arxiv.org/html/2607.04008#S7.T6)shows the mean correctness, completeness, average score, and citation\-set statistics for all ten compared methods across the 47 evaluation queries\. Correctness and completeness are Claude Sonnet 4\.5 judge rubric scores for factual consistency and information coverage, respectively; Avg\. is their arithmetic mean\. The three citation\-set metrics compare reference\-side cited documentsGGwith RAG\-side cited documentsRR: Recall=\|G∩R\|/\|G\|=\|G\\cap R\|/\|G\|, Precision=\|G∩R\|/\|R\|=\|G\\cap R\|/\|R\|, and Jaccard=\|G∩R\|/\|G∪R\|=\|G\\cap R\|/\|G\\cup R\|\[zheng2023llm\_judge,liu2023geval,jaccard1901distribution\]\.

Table 6:Supplementary diagnostic evaluation under the Claude Sonnet 4\.5 judge\. Citation\-set metrics are reported as 0–1 ratios\.No\.ApproachCorr\.Comp\.Avg\.RecallPrecisionJaccard1rrf\-no\-rerank3\.0002\.4682\.7340\.7080\.9220\.6952rule\-minilm2\.9152\.4472\.6810\.8160\.9610\.7873single\-query\-bm252\.9792\.3832\.6810\.7610\.9400\.7424caes\-rag\-rrf2\.9362\.3622\.6490\.7260\.9470\.7205default2\.9152\.3622\.6390\.7470\.9360\.7386topic\-shift\-current2\.8302\.3622\.5960\.7760\.9080\.7407semantic\-current2\.7872\.4042\.5950\.7630\.9380\.7358topic\-shift\-minilm2\.8302\.2982\.5640\.7540\.9110\.7159semantic\-minilm2\.7872\.3402\.5630\.7970\.9040\.72410concat\-baseline1\.5741\.3191\.4460\.1320\.2380\.124

### 7\.1Supplementary Evaluation Highlights Lexical Fusion

Under the supplementary Claude Sonnet 4\.5 judge protocol,rrf\_no\_rerankobtains the highest mean score, 2\.734, followed byrule\_minilmandsingle\_query\_bm25, both at 2\.681\. This suggests that the local judge rewards broadly plausible answer quality from a diverse lexical\-fusion evidence pool\. However, because the primary evaluation favorsrule\-minilm, we interpret this local evaluation result as diagnostic rather than definitive\.

The supplementary results still confirm an important lower\-level point: retrieval and evidence selection are necessary\. Theconcat\_baselineperforms substantially worse than retrieval\-based systems, with an average judge score of 1\.446 and mean Jaccard coefficient of 0\.124\.

### 7\.2Primary and Supplementary Evaluations Disagree Informatively

The most important disagreement is that supplementary LLM\-judge evaluation assigns the highest mean score torrf\_no\_rerank, whereas primary metrics favorrule\-minilm\. This suggests that a single local judge can reward fluent or broadly plausible synthesis while underweighting improvements in cited\-evidence precision and nugget coverage\. The primary metrics separately expose lexical overlap, semantic similarity, citation precision, nugget coverage, average grade, and TFC1, so they provide a more detailed picture of where MiniLM sentence reranking helps\.

## 8Discussion

The primary LongEval\-RAG results refine the main lesson of our study\. Candidate\-constrained provenance control is necessary, but not sufficient; the largest gains come from selecting answer\-worthy evidence inside the fixed candidate set\. Among our variants, the strongest primary\-evaluation configuration isrule\-minilm, which combines deterministic rule\-based chunks with late MiniLM sentence reranking\.

A likely explanation is division of labor\. Rule chunks preserve local sentence context with predictable overlap, while query expansion, PRF, and RRF produce a diverse lexical candidate pool\. Lightweight reranking and the citation prior keep evidence selection inside the official candidate set\. MiniLM then operates late, where it can choose answer\-worthy sentences without changing document segmentation\. This may avoid error propagation from semantic or topic\-shift chunking while still benefiting from neural matching\.

The primary results do not imply that semantic or topic\-shift chunking is intrinsically ineffective\. The candidate pool contains only ten documents per query, so upstream chunk boundaries may matter less than final sentence allocation\. Embedding\-based chunking may also split or merge scientific text in locally coherent ways that are not optimized for the gold\-answer nuggets\. Scientific abstracts and full texts contain dense terminology, so lexical overlap and title/body signals may remain strong baselines\. The mixed primary metrics show that semantic and topic\-shift variants can improve some measures while harming others, which means they need tuning against the intended metric target\.

The disagreement between primary and supplementary evaluations is also a useful result\. Local LLM judging appears to reward broadly plausible answer quality, while the primary metrics separately measure gold\-answer overlap, semantic similarity, cited\-document precision, nugget coverage, and query\-term constraints\. This shows why RAG evaluation should combine multiple complementary measures rather than relying on a single judge score or a single lexical\-overlap metric\.

Future work should therefore focus on late\-stage sentence allocation and metric\-aware reranking\. In particular, we should diagnose when MiniLM improves nugget coverage, when it changes only answer phrasing, and when it selects less precise citations\. The fixed\-10\-document setting also calls for more explicit diagnostics separating cases where the right document is present but ranked too low from cases where the document is selected but not used well in the answer\.

## 9Limitations

Several limitations should be considered when interpreting these results\. First, the current experiments use a 47\-query evaluation slice, so small differences between closely ranked systems should be treated as suggestive rather than conclusive without additional queries, confidence intervals, or repeated runs\. Second, primary metrics are more authoritative than our supplementary judge, but they still emphasize particular reference answers, nugget definitions, and metric implementations; alternative gold annotations could change fine\-grained conclusions\. Third, the supplementary reference\-style answers are model\-generated rather than written by human annotators, which means they may omit legitimate evidence, reflect model\-specific synthesis preferences, or encode errors that affect downstream judging\. Fourth, although the supplementary judge uses a different model family from the answer generator, LLM\-as\-judge evaluation can still be sensitive to prompt wording, answer style, verbosity, and subtle factual errors\. Fifth, document\-ID overlap is only a diagnostic proxy: high overlap does not guarantee that an answer uses evidence correctly, and low overlap does not necessarily imply an incorrect answer when several candidate documents contain similar information\. Finally, the fixed ten\-document candidate setting and the mixture of full\-text and abstract\-only inputs limit how directly these scores generalize to other RAG settings\. We therefore interpret both the supplementary and primary numbers as protocol\-dependent comparative evidence about pipeline behavior, not as an absolute estimate of final task performance\.

## 10Conclusion

The primary LongEval\-RAG results refine the main lesson of our study\. Candidate\-constrained provenance control is necessary, but not sufficient; the largest gains come from selecting answer\-worthy evidence inside the fixed candidate set\. Among our variants, the strongest primary\-evaluation configuration isrule\-minilm, which combines deterministic rule\-based chunks with late MiniLM sentence reranking\. This method achieves the best BERTScore, retrieval precision, nugget coverage, and average grade among our submissions, suggesting that neural reranking is most useful when applied after stable lexical retrieval and passage construction\.

At the same time, disagreements between supplementary LLM\-judge evaluation and primary evaluation show that RAG systems should be assessed with multiple complementary measures rather than a single judge score\. These results provide a clear next direction: keep the provenance\-preserving candidate\-constrained framework, retain the strongest rule\-based and MiniLM sentence\-reranking baselines, and focus future work on metric\-aware late\-stage evidence allocation\.

## Code Availability

## Declaration of Generative AI and AI\-assisted technologies

Generative AI tools were used in experiment design and implementation, analysis, and manuscript drafting and editing; all outputs were reviewed and revised by the authors, who take full responsibility for the content\.

## References