Do LLM Attribution Metrics Transfer? Auditing Retrieval-Augmented Generation Evaluation Across Datasets and Constructs

# Do LLM Attribution Metrics Transfer? Auditing Retrieval-Augmented Generation Evaluation Across Datasets and Constructs
Source: [https://arxiv.org/html/2606.23915](https://arxiv.org/html/2606.23915)
Tianyu Ding tianyd@amazon\.com Amazon Web Services &Aditya Nannapaneni anannap@amazon\.com Amazon Web Services &Juan Pablo De la Cruz Weinstein jcruam@amazon\.com Amazon Web Services

###### Abstract

Practice often treats automatic metrics for attribution in LLM retrieval\-augmented generation as interchangeable\. We audit eight automatic scorers — lexical, embedding, and BERTScore baselines alongside entailment/grounding\-trained models \(clean and FEVER NLI, the checker MiniCheck\) — across three evaluation constructs \(provenance/topicality, generated\-answer attribution, and fact\-check entailment\), asking whether any scorer*transfers*: stays within the95%95\\%confidence interval of the best audited scorer on*every*dataset of a multi\-dataset construct\. In the construct with the most multi\-datasethuman\-labeled coverage — generated\-answer attribution \(AttributionBench’s four source datasets,n=1610n\{=\}1610, with independent HAGRID,n=2150n\{=\}2150\) — none does: the per\-dataset metric rankings invert \(Kendallτ=−0\.64\\tau\{=\}\-0\.64,p=0\.031p\{=\}0\.031on AttributedQA vs\. LFQA\), and an off\-the\-shelf NLI scorer that is best on short\-claim AttributedQA \(AUROC0\.900\.90\) collapses to AUROC0\.530\.53\(chance\) on long\-form LFQA, where BERTScore wins \(0\.910\.91\); the flip is not a length or truncation artifact\. This instability has a concrete decision cost: a naive “best\-on\-average” rule for choosing an evaluator*fails*leave\-one\-dataset\-out \(mean held\-out regret0\.1720\.172AUROC, worse than fixing one scorer\), so metric choice must be validated on the target dataset rather than learned from others\. A prompt\-based LLM judge avoids the chance\-level collapses the automatic scorers suffer \(no LFQA collapse\) but is not uniformly best,∼100×\{\\sim\}100\\timescostlier, and non\-deterministic — relocating, not removing, the validation burden\.

Do LLM Attribution Metrics Transfer? Auditing Retrieval\-Augmented Generation Evaluation Across Datasets and Constructs

Tianyu Dingtianyd@amazon\.comAmazon Web ServicesAditya Nannapanenianannap@amazon\.comAmazon Web ServicesJuan Pablo De la Cruz Weinsteinjcruam@amazon\.comAmazon Web Services

## 1Introduction

Retrieval\-augmented language models are increasingly evaluated for*grounding*: whether each claim in an answer is supported by the provided evidence\. A growing toolbox of automatic scores — lexical overlap, sentence\-embedding similarity, NLI/entailment — stands in for human grounding judgments, and practitioners often report them interchangeably, as if “the attribution metric went up” meant the same thing regardless of metric or benchmark\. By “LLM attribution metrics” we mean metrics evaluating attribution in LLM/RAG outputs \(lexical, embedding, NLI/checker, and prompted LLM\-judge variants\); our core audit is the cheaper*automatic*scorers, with the prompted LLM judge as a boundary case\. We show this assumption is unsafe at two levels\. Coarsely, the evaluation*construct*shapes which metric*family*is appropriate\. More sharply,*within*a single construct the best metric is not stable across its datasets, so no audited automatic scorer in our core audit transfers across the datasets of a construct\.

Concretely, we audit eight metrics — off\-the\-shelf scores with no attribution\-specific fine\-tuning \(lexical Jaccard, MiniLM and MPNet cosine, their fixed blend, BERTScore\) and trained entailment/grounding models \(clean\-MNLI NLI, FEVER NLI, and the purpose\-built checker MiniCheck\(Tanget al\.,[2024](https://arxiv.org/html/2606.23915#bib.bib35)\); a lexical/semantic/combined/NLI subset on the provenance ranking task\) — across three evaluation constructs the literature routinely conflates:\(i\) provenance/topicality— does a score rank a relevant passage top \(ASQA\(Stelmakhet al\.,[2022](https://arxiv.org/html/2606.23915#bib.bib4)\), MS MARCO\(Nguyenet al\.,[2016](https://arxiv.org/html/2606.23915#bib.bib16)\), HotpotQA\(Yanget al\.,[2018](https://arxiv.org/html/2606.23915#bib.bib30)\); passage\-ranking labels, not sentence/claim\-level support\);\(ii\) generated\-answer attribution— does a score predict*human*judgments of whether a generated answer sentence is supported by its cited evidence \(AttributionBench\(Liet al\.,[2024](https://arxiv.org/html/2606.23915#bib.bib31)\)\);\(iii\) fact\-check entailment— does a score predict*human*support labels on short edited claims \(VitaminC\(Schusteret al\.,[2021](https://arxiv.org/html/2606.23915#bib.bib32)\)\)\. At the coarse, cross\-construct level the construct shapes the appropriate metric family, on the audited benchmarks \(Table[2](https://arxiv.org/html/2606.23915#S4.T2)\): the best off\-the\-shelf relevance score per provenance gate reaches0\.730\.73–0\.970\.97top\-1 accuracy \(within overlapping bootstrap CIs of NLI on MS MARCO/HotpotQA, though NLI is unreliable on ASQA at0\.5850\.585\), whereas entailment\-trained models score above all no\-fine\-tuning baselines on fact\-check claims \(VitaminC,0\.810\.81–0\.890\.89vs\. surface≈0\.60\{\\approx\}0\.60AUROC\)\. The sharper finding is*within*generated\-answer attribution: across AttributionBench’s four constituent datasets the best metric is not stable \(the per\-dataset rankings invert, Kendallτ=−0\.64\\tau\{=\}\-0\.64,p=0\.031p\{=\}0\.031on AttributedQA vs\. LFQA\)\. A naive NLI scorer is best on AttributedQA \(AUROC0\.900\.90\) yet drops to0\.530\.53\(chance=0\.50=0\.50\) on LFQA, where BERTScore reaches0\.910\.91— though on a*different*long\-form set \(HAGRID\) the same NLI scorer is best \(0\.800\.80\), so the failure is dataset\-specific and not explained by long\-form*answers*alone \(long*evidence*remains a candidate; §[4](https://arxiv.org/html/2606.23915#S4)\)\.

This matters because practitioners silently read success on one construct, or on one benchmark, as evidence of grounding in general\. Prior benchmarking work establishes that automatic attribution metrics disagree with humans and that no single metric wins everywhere\(Honovichet al\.,[2022](https://arxiv.org/html/2606.23915#bib.bib34); Dziriet al\.,[2022](https://arxiv.org/html/2606.23915#bib.bib33); Yueet al\.,[2023](https://arxiv.org/html/2606.23915#bib.bib22); Liet al\.,[2024](https://arxiv.org/html/2606.23915#bib.bib31)\); beyond that qualitative “they disagree,” our contribution is decision\-relevant: an operational*portability*criterion, a leave\-one\-dataset\-out*regret*that prices committing to one default, and a naive cross\-dataset selector that*fails*\(0\.1720\.172regret\) — a validation discipline, on a controlled audit with human labels on the in\-domain attribution and fact\-check sides and a clean separation from proxy\-labeled provenance \(mechanism discovery is out of scope\)\. This is an evaluation\-resource/cautionary paper, not a new\-metric paper\. Our central contribution is the cross\-dataset evaluator\-selection audit; the rest are supporting diagnostics and boundary probes:

- •Across\-dataset audit\(Section[4](https://arxiv.org/html/2606.23915#S4), Table[2](https://arxiv.org/html/2606.23915#S4.T2)\) showing that the construct coarsely shapes the appropriate metric family, but that*within*in\-domain answer attribution no single*automatic*scorer is consistently near\-best across datasets \(rankings discordant, KendallW=0\.07W\{=\}0\.07,p=0\.029p\{=\}0\.029\): a raw MNLI scorer is best on AttributedQA \(0\.900\.90\) yet AUROC0\.530\.53\(chance=0\.50=0\.50\) on the LFQA dataset — though best again on the independent long\-form HAGRID set \(0\.800\.80\) — while a purpose\-built checker \(MiniCheck\) has the lowest mean leave\-one\-dataset\-out regret among the eight automatic scorers*tested*, yet still leaves a measurable per\-dataset regret \(mean0\.0440\.044AUROC, CI\[0\.014,0\.072\]\[0\.014,0\.072\]\)\.
- •As supporting diagnostics, a training\-freesentence\-unit provenance/topicality diagnosticand aconfusable\-evidence stress\-test protocol\(Sections[3](https://arxiv.org/html/2606.23915#S3),[3\.1](https://arxiv.org/html/2606.23915#S3.SS1.SSS0.Px1)\): under confusable evidence \(high\-overlap distractors from other examples\), the semantic component separates co\-provenant passages from distractors where lexical overlap is fooled\. We benchmark an off\-the\-shelf NLI entailment head against it and find NLI benchmark\-dependent, not uniformly best — a construct\-\(i\) diagnostic, not a general\-purpose grounding metric\.
- •Onvalidation discipline\(Section[4](https://arxiv.org/html/2606.23915#S4)\): a naive “best\-on\-average” selection rule fails leave\-one\-dataset\-out \(mean regret0\.1720\.172, worse than the lowest\-mean\-regret fixed metric among those tested\), so metric choice must be validated on the target dataset; and across\-family LLM\-annotationprobe \(Opus 4\.8 \+ GPT\-5\.4,n=160n\{=\}160\) showing high inter\-model but only moderate, dataset\-dependent human agreement — usable as a corroborative proxy only in the easiest \(short\-factoid\) dataset, bounding off\-the\-shelf LLM label substitution\.
- •Finally, aboundary metric\-sensitivity probe\(Section[5](https://arxiv.org/html/2606.23915#S5)\): a structured GRPO reward rewrites most of the output yet moves none of the deterministic answer\-level metrics \(QA\-EM, ROUGE\-L, citation count\) at 3B or 7B\. We use this not to propose the reward, nor to claim it improved grounding, but to show that QA\-EM, ROUGE\-L, and citation count did not detect this large ASQA rewrite intervention\.

In short, “attribution metric” names a family of signals capturing different constructs \(provenance, topical relatedness, human\-judged support\); the appropriate family is construct\-dependent, and*within*answer attribution no metric transfers across datasets without target\-dataset validation\. Evaluations should report the dataset and construct they target and prefer a checker validated on that dataset, not treat any single metric as a universal proxy for grounding\.

## 2Related Work

#### Correctness vs\. faithfulness\.

An answer can be correct while its citations do not support its claims\(Rashkinet al\.,[2023](https://arxiv.org/html/2606.23915#bib.bib20); Liuet al\.,[2023](https://arxiv.org/html/2606.23915#bib.bib21)\)— a faithfulness gap distinct from factual correctness\(Jiet al\.,[2023](https://arxiv.org/html/2606.23915#bib.bib17)\), acute for long\-form answers exact match cannot score\(Fanet al\.,[2019](https://arxiv.org/html/2606.23915#bib.bib19)\)and worsened by uneven use of mid\-context evidence\(Liuet al\.,[2024](https://arxiv.org/html/2606.23915#bib.bib18)\)\.

#### Attribution and citation quality\.

AIS\(Rashkinet al\.,[2023](https://arxiv.org/html/2606.23915#bib.bib20)\)formalizes source support, and audits find fluent answers often carry unsupported citations\(Liuet al\.,[2023](https://arxiv.org/html/2606.23915#bib.bib21)\)\. ALCE\(Gaoet al\.,[2023](https://arxiv.org/html/2606.23915#bib.bib5)\)introduced NLI\-based citation recall/precision \(with QAMPARI\(Amouyalet al\.,[2023](https://arxiv.org/html/2606.23915#bib.bib24)\)\); FActScore\(Minet al\.,[2023](https://arxiv.org/html/2606.23915#bib.bib6)\)decomposes into atomic facts; Attribute\-First\(Slobodkinet al\.,[2024](https://arxiv.org/html/2606.23915#bib.bib13)\)and LongCite\(Zhanget al\.,[2024](https://arxiv.org/html/2606.23915#bib.bib14)\)produce attributable text; AttrScore\(Yueet al\.,[2023](https://arxiv.org/html/2606.23915#bib.bib22)\)and RAGAS\(Eset al\.,[2024](https://arxiv.org/html/2606.23915#bib.bib23)\)judge support with LLM/entailment models\. AttributionBench\(Liet al\.,[2024](https://arxiv.org/html/2606.23915#bib.bib31)\)benchmarks such evaluators against human labels — but, crucially, only*trained/LLM*evaluators, not the off\-the\-shelf lexical/embedding baselines we find within overlapping CIs in\-domain\. These methods need entailment/LLM judges or modify generation; our scorers include a deterministic sentence\-unit one\(Reimers and Gurevych,[2019](https://arxiv.org/html/2606.23915#bib.bib15)\), and our contribution is the cross\-dataset*audit*\.

#### RL for grounding\.

Prior work trains models to cite evidence via rewards, from verified\-quote RLHF\(Menicket al\.,[2022](https://arxiv.org/html/2606.23915#bib.bib28)\)to citation rewards\(Huanget al\.,[2024](https://arxiv.org/html/2606.23915#bib.bib27)\), gated sufficiency/abstention rewards\(Zhaoet al\.,[2026](https://arxiv.org/html/2606.23915#bib.bib9)\), and verifiable\-reward training\(Simet al\.,[2025](https://arxiv.org/html/2606.23915#bib.bib10)\)\. These*fix the model*; we*diagnose the evaluation*, and our null result \(Section[5](https://arxiv.org/html/2606.23915#S5)\) is consistent with their report that answer\-level metrics are hard to move\.

#### Distractors and robustness\.

Contextual distractors cause large drops\(Leeet al\.,[2026](https://arxiv.org/html/2606.23915#bib.bib11)\), especially*highly semantically related*ones\(Wuet al\.,[2024](https://arxiv.org/html/2606.23915#bib.bib29)\), and grounding can fail even in instruction\-tuned models under adversarial conditions\(Koneruet al\.,[2026](https://arxiv.org/html/2606.23915#bib.bib12)\)\. Our confusable\-evidence test \(Section[3\.1](https://arxiv.org/html/2606.23915#S3.SS1.SSS0.Px1)\) is a controlled, passage\-ranking instance\.

## 3A Sentence\-Unit Provenance\-Ranking Score

We score how well each sentence of an answer is*matched*to a set of candidate passages, for the provenance\-ranking diagnostic of construct \(i\) \(distinct from the human\-support task of constructs ii, iii\)\. Let an answer be split into sentence unitsu1,…,umu\_\{1\},\\dots,u\_\{m\}and let the evidence be passagesp1,…,pkp\_\{1\},\\dots,p\_\{k\}\. For a unituuand passageppwe compute a combined relevance score

s​\(u,p\)=0\.40​lex​\(u,p\)\+0\.45​sem​\(u,p\)\+0\.10​h​\(u\)\+0\.05​c​\(u,p\),\\begin\{split\}s\(u,p\)=\{\}&0\.40\\,\\mathrm\{lex\}\(u,p\)\+0\.45\\,\\mathrm\{sem\}\(u,p\)\\\\ &\{\}\+0\.10\\,h\(u\)\+0\.05\\,c\(u,p\),\\end\{split\}\(1\)wherelex\\mathrm\{lex\}is token Jaccard,sem\\mathrm\{sem\}is sentence\-transformer cosine\(Reimers and Gurevych,[2019](https://arxiv.org/html/2606.23915#bib.bib15)\)\(all\-MiniLM\-L6\-v2\), the*answer\-hint*h​\(u\)h\(u\)is the fraction of gold short answers in the unit, and the*citation bonus*c​\(u,p\)=1c\(u,p\)\{=\}1whenuucitespp\. A unit’s score ismaxp⁡s​\(u,p\)\\max\_\{p\}s\(u,p\)\. It istraining\-free\(only a small frozen encoder\)\.

The score is a fixed*provenance*diagnostic for construct \(i\),nota proposed general\-purpose grounding metric \(far from best on the human\-support tasks of Section[4](https://arxiv.org/html/2606.23915#S4)\) andnotthe paper’s central contribution — the human\-labeled cross\-dataset audit of Section[4](https://arxiv.org/html/2606.23915#S4)\. We report the single canonical weighting, not a tuned one\. The answer\-hinth​\(u\)h\(u\)is partly label\-aware, but it does not affect the confusable ranking:h​\(u\)h\(u\)is constant within a unit \(so it cannot reorder that unit’s candidate passages\), and the citation termc​\(u,p\)=0c\(u,p\)\{=\}0throughout because ASQA gold answers carry no inline citations\. The confusable test thus ranks on lexical/semantic only and the central result needs no gold\-label access \(ablated below\)\.

### 3\.1Provenance/topicality \(construct i\)

The first construct asks whether a score routes an answer sentence to the passage it came from\. On ASQA we form a gold\-reference vs\. unadapted\-model contrast: the gold long\-form answer vs\. an unadaptedQwen2\.5\-3B\-Instructgeneration over the example’s≥2\\geq 2passages \(623623dev examples; the confusable test uses a stricter≥3\\geq 3\-passage filter,339339\)\.111Two small ASQA checks \(App\.[A](https://arxiv.org/html/2606.23915#A1)\) are*internal consistency*only \(circular labels\), so we do not rely on them\.Holding short\-answer correctness roughly equal \(9898examples within0\.150\.15recall\), the combined provenance gap is\+0\.160\+0\.160, CI\[\+0\.133,\+0\.189\]\[\+0\.133,\+0\.189\]\(full\-dev\+0\.164\+0\.164\), and is not a hint artifact \(zeroing the hint weight moves the matched gap only to\+0\.153\+0\.153, CI excluding0; App\.[B](https://arxiv.org/html/2606.23915#A2)\)\. So provenance\-sensitive scores move even when correctness does not — but this is provenance sensitivity, not human\-judged support\.

#### Confusable\-evidence stress test\.

To test the “just lexical overlap” objection, for each gold answer sentence we build a pool from the unit’s own ASQA passages plusk=5k\{=\}5*confusable distractors*\(highest lexical\-overlap passages from*other*examples\)\. The label is aprovenance proxy\(did the top passage come from the unit’s own example\) — not a human groundedness judgment\. We report top\-1 provenance accuracy and the*fooled rate*\(a confusable distractor outranks every co\-provenant passage\); ranking uses lexical/semantic only\.

Table 1:Confusable\-evidence stress test on ASQA \(top\-1 averaged across four seeds; best in bold; semantic encoderall\-mpnet\-base\-v2,k=5k\{=\}5distractors\)\. Lexical / semantic / combined are on the full usable dev pool \(339 ex / 1309 units\)\. The combined−\-lexical gap is\+0\.144\+0\.144, 95% CI\[\+0\.122,\+0\.165\]\[\+0\.122,\+0\.165\]: all four independent\-seed gaps \(\+0\.144,\+0\.144,\+0\.144,\+0\.143\+0\.144,\+0\.144,\+0\.144,\+0\.143\) have CIs above0\.†The NLI row is computed on the 200\-example\-per\-seed subsample \(3 seeds,≈\\approx760 units each\) for cost, so its absolute value is not directly pooled with the others; the lexical and semantic values on that same subsample are 0\.786 and 0\.938, so NLI’s 0\.585 is well below both on matched data\. Ground truth is a passage\-provenance proxy; distractors are highest\-lexical\-overlap passages from other examples\. NLI = an off\-the\-shelf DeBERTa\-v3 MNLI/FEVER/ANLI entailment head \(passage→\\rightarrowsentence\) on this ranking task, not full ALCE citation evaluation\.
#### The semantic component survives; lexical is fooled; NLI does not help\.

On the full dev pool \(339339examples,13091309units; semantic encoderall\-mpnet\-base\-v2for this confusable test\), top\-1 provenance accuracy is0\.7750\.775lexical\-only,0\.9340\.934semantic\-only,0\.9190\.919combined \(combined−\-lexical\+0\.144\+0\.144, CI\[\+0\.122,\+0\.165\]\[\+0\.122,\+0\.165\], all four fresh\-example seed gaps\>0\>0:0\.144,0\.144,0\.144,0\.1430\.144,0\.144,0\.144,0\.143\); the fooled rate drops from22\.5%22\.5\\%\(lexical\) to8\.1%8\.1\\%\(combined\) and6\.6%6\.6\\%\(semantic\)\. An off\-the\-shelf DeBERTa MNLI/FEVER/ANLI scorer reaches only0\.5850\.585here — below lexical, far below semantic — a construct mismatch \(ASQA sentences*synthesize*across passages; cf\.Labanet al\.,[2022](https://arxiv.org/html/2606.23915#bib.bib25)\), not a verdict on entailment in general\. Sweeping the lexical weightα\\alpha\(Fig\.[1](https://arxiv.org/html/2606.23915#A2.F1), App\.[B](https://arxiv.org/html/2606.23915#A2)\) confirms the canonical blend \(α≈0\.47\\alpha\\approx 0\.47\) is past the knee, so we donotclaim it optimal; this is specific to construct \(i\), where semantic carries the signal — on human\-judged support \(ii, iii\) it is*not*dominant\.

## 4The Cross\-Dataset Audit

We make our central point in two steps: coarsely the*construct*shapes which metric*family*is appropriate; more sharply,*within*a construct the best metric stays unstable across datasets\. For this audit we use*portable*\(its ranking*transfers*\) as one near\-best screen, operational*for metric selection*: a metric stays within the95%95\\%CI of the best*audited*metric on*every*dataset of a construct\. We do not offer this as a universal definition of transfer, and our consequence is at the level of choosing an evaluator, not ranking generators \(system\-level transfer is future work; see Limitations\)\. In our data the same non\-portability reading is corroborated by discordant per\-dataset metric rankings \(concordanceW=0\.07W\{=\}0\.07,p=0\.029p\{=\}0\.029; the AttributedQA\-vs\-LFQA ranking inverts, Kendallτ=−0\.64\\tau\{=\}\-0\.64,p=0\.031p\{=\}0\.031\), a non\-zero leave\-one\-dataset\-out regret \(mean0\.0440\.044AUROC for the best fixed choice\), paired\-bootstrap sign flips excluding zero \(NLI−\-BERTScore=−0\.378=\-0\.378on LFQA vs\.\+0\.213\+0\.213on AttributedQA\), theϵ\\epsilon\-regret reading \(CI\[0\.014,0\.072\]\[0\.014,0\.072\]\), and the sensitivity analyses in App\.[F](https://arxiv.org/html/2606.23915#A6); the flip persists when the overlapping datasets \(ExpertQA and LFQA\) are excluded \(n=7336n\{=\}7336, App\.[F](https://arxiv.org/html/2606.23915#A6)\)\. We audit eight metrics — off\-the\-shelf scores with no attribution\-specific fine\-tuning \(lexical Jaccard, MiniLM and MPNet cosine, their blend, BERTScore\(Zhanget al\.,[2020](https://arxiv.org/html/2606.23915#bib.bib26)\)\) and trained models \(clean non\-FEVER MNLI NLI, FEVER NLI, and the purpose\-built checker MiniCheck\(Tanget al\.,[2024](https://arxiv.org/html/2606.23915#bib.bib35)\)\) — on the two support constructs, and a lexical/semantic/combined/NLI subset on the provenance ranking task; we never pool the three constructs:\(i\) provenance/topicality\(ASQA, MS MARCO, HotpotQA: rank candidate passages and check the top one; ASQA uses a provenance proxy, MS MARCO and HotpotQA use human passage\-level labels, none sentence/claim\-level support\);\(ii\) generated\-answer attributionwith*human*support labels \(AttributionBench\(Liet al\.,[2024](https://arxiv.org/html/2606.23915#bib.bib31)\): is a generated answer sentence attributable to its cited evidence?\);\(iii\) fact\-check entailmentwith*human*support labels \(VitaminC\(Schusteret al\.,[2021](https://arxiv.org/html/2606.23915#bib.bib32)\): is a short claim supported by its evidence?\)\. For \(ii, iii\) we score each \(claim/sentence, evidence\) pair and report AUROC \(clustered bootstrap CIs\); the clean NLI baseline uses a non\-FEVER MNLI model to avoid VitaminC leakage, while the provenance ranker \(c\) uses an off\-the\-shelf DeBERTa MNLI/FEVER/ANLI head \(so the NLI*column*differs by construct\)\. Table[2](https://arxiv.org/html/2606.23915#S4.T2)is the headline\.

Table 2:Benchmark\-by\-benchmark audit; constructs are reported separately and never pooled\.The main multi\-dataset non\-portability result is the four\-source AttributionBench audit in \(a\); HAGRID, VitaminC, and the provenance benchmarks are independent boundary/contrast checks, not evidence for a universal metric ranking\. AUROC for the support tasks \(a, b; binary support against*human*sentence/claim labels\) and top\-1*passage\-ranking*accuracy \(not AUROC\) for provenance \(c; passage\-level labels — a provenance proxy for ASQA, human passage\-relevance for MS MARCO/HotpotQA — which are*not*sentence/claim\-level support judgments\); best per row in bold\. For the human\-support tasks \(a, b\) NLIclis the*clean, non\-FEVER*MNLI model \(roberta\-large\-mnli\) and NLIfeis FEVER\-trained; MiniChk is MiniCheck\(Tanget al\.,[2024](https://arxiv.org/html/2606.23915#bib.bib35)\)\. For the human\-support tasks \(a, b\), Comb is the two\-term blend0\.47⋅lex\+0\.53⋅sem0\.47\\cdot\\text\{lex\}\+0\.53\\cdot\\text\{sem\}over each \(claim/sentence, evidence\) pair \(the answer\-hint and citation terms of Eq\. \([1](https://arxiv.org/html/2606.23915#S3.E1)\) do not apply — support pairs carry no gold answer or inline citation — and are omitted\); MiniLM is the semantic encoder there\. The provenance NLI column \(c\) is the off\-the\-shelf DeBERTa\-v3 MNLI/FEVER/ANLI head used as a ranker \(Section[3\.1](https://arxiv.org/html/2606.23915#S3.SS1.SSS0.Px1)\)\.Coarsely, the construct shapes the appropriate family: entailment\-trained models score above all no\-fine\-tuning baselines on fact\-check \(b\), while off\-the\-shelf relevance is within overlapping CIs of NLI on provenance \(c\)\.But within one construct\(a\), the best metric is not stable across the four AttributionBench source datasets \(the per\-dataset metric rankings are discordant: KendallW=0\.07W\{=\}0\.07,p=0\.029p\{=\}0\.029; the AttributedQA\-vs\-LFQA ranking inverts,τ=−0\.64\\tau\{=\}\-0\.64; HAGRID is an independent long\-form set\): the clean MNLI scorer is best on short\-claim AttributedQA \(0\.9040\.904\) yet AUROC0\.5310\.531\(chance=0\.50=0\.50\) on long\-form LFQA, where BERTScore wins \(0\.9090\.909\)\. The flip is not a truncation artifact \(512512\-token context unchanged\), and length alone does not explain the instability \(it persists within sources and at matched length across sources\)\. MiniCheck has the lowest mean leave\-one\-dataset\-out regret*among those tested*\(AttributionBench mean AUROC0\.7740\.774; regret0\.0440\.044\) but still not uniformly best\. ExpertQA is an underspecified dataset \(oracle best\-of\-eight only0\.6180\.618\)\. The semantic provenance ranker \(c\) isall\-mpnet\-base\-v2for the ASQA confusable test \(Section[3\.1](https://arxiv.org/html/2606.23915#S3.SS1.SSS0.Px1); hence its0\.9340\.934sits in the MPNet column\) andall\-MiniLM\-L6\-v2for MS MARCO/HotpotQA; BERTScore, NLIfe, and MiniCheck are AUROC support scorers and are not run as provenance*rankers*\(c\)\. Human\-labeled*support*\(a, b\) and proxy\-labeled*provenance*\(c\) are distinct constructs and are not pooled\.#### Coarsely, the construct shapes the appropriate metric family \(bounded to the audited benchmarks\)\.

For fact\-check entailment we have only one audited boundary dataset, VitaminC \(n=4000n\{=\}4000human labels\), where the entailment\-trained models score above all no\-fine\-tuning baselines: FEVER NLI0\.8940\.894, clean NLI0\.8110\.811, MiniCheck0\.8100\.810, versus the surface metrics clustered near0\.600\.60–0\.630\.63\(BERTScore0\.5960\.596, lexical0\.6190\.619, combined0\.6280\.628\)\. We treat this as a one\-dataset contrast on short edited claims,*not*a general law about fact\-checking\. For provenance/topicality we report only passage\-ranking results \(ASQA provenance proxy; human passage\-level labels for MS MARCO/HotpotQA\), reaching0\.730\.73–0\.970\.97top\-1 accuracy \(within overlapping bootstrap CIs of NLI, which is however unreliable on ASQA at0\.5850\.585\); these labels are distinct from sentence/claim\-level support and are not pooled with the support tasks\. The coarse separation thus holds only loosely and only on these audited benchmarks\.

#### But the “attribution” label is not a reliability boundary\.

AttributionBench \(n=1610n\{=\}1610human labels\) aggregates four source datasets the literature files under one task — is a generated answer sentence attributable to its cited evidence? — and a practitioner who reads “attribution metric” expects a single applicable choice\. We make no claim that the four are cognitively identical \(they differ sharply in evidence length and structure, which we use below to characterize*where*metrics fail\); our point is the practitioner\-facing one: this shared label does*not*predict which metric to trust\. The best metric is not stable across the four, and the evidence that isolates this*crossover*is rank\-based: the per\-dataset metric rankings are discordant \(Kendall’sW=0\.07W\{=\}0\.07; rank\-permutationp=0\.029p\{=\}0\.029\), the AttributedQA\-vs\-LFQA ranking literally inverts \(Kendallτ=−0\.64\\tau\{=\}\-0\.64,p=0\.031p\{=\}0\.031\), and the best\-vs\-runner\-up sign flips \(NLI−\-BERTScore=−0\.378=\-0\.378on LFQA vs\.\+0\.213\+0\.213on AttributedQA, CIs exclude0\) — a pure dataset\-difficulty \(main\) effect cannot produce a rank reversal\. A two\-way decomposition of the AUROC matrix places35%35\\%of the cross\-cell variance in the interaction residual \(a residual permutation that holds each dataset’s difficulty fixed is borderline,p=0\.053p\{=\}0\.053\), while most of the raw cross\-cell spread \(61%61\\%\) is dataset\-difficulty main effect: some datasets are simply harder for every metric\. \(An exchangeability permutation that shuffles the dataset assignment of per\-example pairs rejects atp<0\.002p<0\.002, but that test conflates the main effect with the interaction, so we rely on the rank\-based statistics above\.\) The clean MNLI scorer is the best metric on short\-claim AttributedQA \(0\.9040\.904;\+0\.270\+0\.270over lexical Jaccard, paired95%95\\%CI\[0\.198,0\.347\]\[0\.198,0\.347\]\) yet falls to AUROC0\.5310\.531\(chance=0\.50=0\.50\) on long\-form LFQA, where BERTScore reaches0\.9090\.909\(MNLI−\-BERTScore=−0\.378=\-0\.378, CI\[−0\.500,−0\.238\]\[\-0\.500,\-0\.238\]\)\. This collapse pins to one*checkpoint*, not the entailment family: on the same LFQA, FEVER\-trained NLI scores0\.7200\.720and the checker MiniCheck0\.8520\.852— so “no single metric transfers” is about individual scorers, not a verdict on entailment models as a class\. This sign\-flip is not a context\-window artifact \(512512\- vs256256\-token: LFQA0\.531→0\.5270\.531\\\!\\to\\\!0\.527\)\. Length contributes but does not account for it: at matched evidence length \(8989–180180words\) MNLI still differs by source \(0\.9340\.934AttributedQA vs\.0\.5970\.597ExpertQA\), and the within\-source short\-vs\-long drop \(\+0\.06\+0\.06LFQA,\+0\.09\+0\.09ExpertQA\) is far below the0\.380\.38cross\-dataset flip\. A purpose\-built checker \(MiniCheck,20482048\-token window\) repairs the long\-form failure \(LFQA0\.8520\.852\) and has the lowest mean leave\-one\-dataset\-out regret*among the metrics we tested*\(AttributionBench mean AUROC0\.7740\.774\) — yet is not uniformly best, losing to BERTScore on Stanford\-GenSearch \(−0\.055\-0\.055, CI\[−0\.104,−0\.010\]\[\-0\.104,\-0\.010\]\)\. Committing to one metric everywhere still incurs modest per\-dataset*regret*vs\. oracle selection \(mean0\.0440\.044, CI\[0\.014,0\.072\]\[0\.014,0\.072\]\) — modest in that it is well below the per\-dataset sign\-flips \(0\.3780\.378\) that drive the headline, but its CI excludes zero, so it is a real if small cost\. ExpertQA is an underspecified dataset where every metric is weak \(oracle best\-of\-eight only0\.6180\.618, barely above chance\); three supplementary metrics agree \(cross\-encoder0\.580\.58, MiniCheck\-FT50\.600\.60, AlignScore0\.600\.60; App\.[D](https://arxiv.org/html/2606.23915#A4)\), so the weakness is not isolated to one scorer, and we lean no claim on it\.

#### An external boundary: long\-form alone does not predict the NLI failure\.

To test whether “long\-form” explains the LFQA collapse, we score the full zoo onHAGRID\(Kamallooet al\.,[2023](https://arxiv.org/html/2606.23915#bib.bib42)\), an*independent*long\-form attribution set \(not an AttributionBench source\) with human sentence/claim\-levelattributablelabels \(n=2150n\{=\}2150\)\. Here the clean MNLI scorer is instead the*best*metric \(0\.8000\.800; BERTScore0\.7930\.793, MiniCheck0\.7910\.791, lexical0\.7520\.752\)\. HAGRID answers have LFQA\-comparable claim length but much shorter, explicitly\[n\]\-cited evidence \(short cited quotes rather than LFQA’s∼\\sim320\-word passages\)\. This refutes the simplest reading — that NLI fails whenever the*answer*is long\-form\. It does*not*, however, rule out long*evidence*\(which HAGRID lacks\) as the mediator; if anything, HAGRID’s short cited evidence is consistent with our evidence\-length account below\. Rankings are thus dataset\-specific even across superficially similar long\-form settings; we cannot isolate the causal latent characteristic from dataset identity here, and leave that to future work\.

We also include BEGIN\(Dziriet al\.,[2022](https://arxiv.org/html/2606.23915#bib.bib33)\)only as an appendix boundary check in a related but different setting \(knowledge\-grounded*dialogue*with response\-level labels; App\.[E](https://arxiv.org/html/2606.23915#A5)\)\. Its pattern is consistent with dataset dependence, but it is not part of the main multi\-dataset audit, not used in the transfer criterion, and not co\-equal with the AttributionBench/HAGRID evidence\. We caution that the provenance results \(construct i\) use passage\-level provenance/relevance labels, not human sentence/claim\-level support judgments, and we do not pool them with \(ii, iii\)\.

#### Naive cross\-dataset metric selection does not generalize\.

Can one*learn*a good default from observed datasets? Picking the metric with the best mean AUROC on seen datasets fails leave\-one\-dataset\-out over the four sources: mean realized regret is0\.1720\.172AUROC, and the learned rule is on average0\.0600\.060AUROC*worse*than fixing the raw MNLI metric\. The loss is*not*on the LFQA fold \(there the rule picks MNLI and ties it\); it comes from the other folds — most sharply AttributedQA\-held\-out, where the rule picks BERTScore \(0\.690\.69\) over MNLI \(0\.900\.90\), a−0\.21\-0\.21swing — so averaging over seen datasets actively mis\-selects on the held\-out one\. The actionable consequence is a*validation discipline*, not a free\-lunch selector: do not commit to a single global metric \(least of all raw MNLI\) without checking the target dataset; validate on*target*\-dataset labels; and report the dataset\. These conclusions are bounded to the four datasets and eight metrics audited here\.

#### When can LLM adjudication stand in for human labels?

Since target\-dataset human labels are often missing, we probe cross\-family LLM adjudication: two frontier models from*different*families \(Claude Opus 4\.8, GPT\-5\.4\) independently annotate160160AttributionBench items \(4040/source\), reported as*LLM\-adjudicated*, never human/gold\. The families agree with*each other*\(κ=0\.832\\kappa\{=\}0\.832\) but only moderately with*human*labels \(κ≈0\.47\\kappa\{\\approx\}0\.47\), and that human\-agreement is itself dataset\-dependent \(accuracy AttributedQA0\.93/0\.880\.93/0\.88down to ExpertQA0\.58/0\.630\.58/0\.63\); on Stanford they agree strongly with each other \(κ=0\.90\\kappa\{=\}0\.90\) yet diverge from humans, so high inter\-model agreement is*not*evidence of validity\. LLM adjudication is thus a corroborative proxy only in the easiest \(short\-factoid\) dataset\.

#### Is a prompt\-based LLM judge more stable across datasets?

Our non\-transfer finding is about*automatic*scorers \(including trained ones: clean/FEVER NLI, MiniCheck\)\. The natural rejoinder — practitioners increasingly use a prompt\-based*LLM judge*— we test directly \(Opus 4\.8 as a scoring metric, Table[14](https://arxiv.org/html/2606.23915#A6.T14)\): on the same four AttributionBench sources it never drops to the chance\-level collapses the strong automatic scorers suffer \(AUROC0\.730\.73–0\.920\.92, vs\. clean MNLI0\.530\.53–0\.900\.90and BERTScore0\.570\.57–0\.910\.91on those four\) and does*not*collapse on LFQA\. So the cheap automatic proxies people substitute for human judgment are the ones most prone to non\-transfer\. The LLM judge*avoids the collapses*here, but this is a boundary observation, not an escape hatch: it is still not uniformly best, costs∼100×\{\\sim\}100\\timesmore, and is non\-deterministic; its AUROC is computed over the408408of480480items the judge scored \(a parsed judgment;15%15\\%refused or truncated, dropped not imputed\), so the judge relocates rather than removes the validation burden\.

We now detail the provenance benchmarks behind Table[2](https://arxiv.org/html/2606.23915#S4.T2)\(c\), whose labels are passage\-level \(a provenance proxy for ASQA,*human*passage\-relevance for MS MARCO and HotpotQA\),*not*sentence/claim\-level support\. OnMS MARCOv2\.1\(Nguyenet al\.,[2016](https://arxiv.org/html/2606.23915#bib.bib16)\)\(11,83611\{,\}836dev queries with≥3\\geq 3passages, a well\-formed answer, and a human\-selected passage; “combined” is the gold\-free0\.47​lex\+0\.53​sem0\.47\\,\\mathrm\{lex\}\+0\.53\\,\\mathrm\{sem\}blend\), we rank each query’s own passages and ask whether the top\-1 is*human*\-selected \(is\_selected\)\. Over20002000queries/seed×\\times3, all scorers far exceed the random baseline \(0\.1040\.104\): lexical0\.6830\.683, semantic0\.6460\.646, NLI0\.7080\.708, combined0\.7260\.726\(combined and NLI within bootstrap noise, both beating the components\) — so the score tracks*human*passage relevance, a construct\-\(i\) sanity check, not grounded support\. Under the confusable protocol \(n=300×4n\{=\}300\\times 4\) lexical\-only top\-1 is0\.8850\.885, semantic0\.9950\.995, combined0\.9960\.996\(combined−\-lexical\+0\.111\+0\.111, every per\-seed CI excludes zero\), and the added NLI baseline reaches0\.9520\.952— above lexical, below semantic\. The same lexical\-fooled, semantic\-survives ordering as ASQA, near\-ceiling because MS MARCO is more extractive\.

OnHotpotQA\(Yanget al\.,[2018](https://arxiv.org/html/2606.23915#bib.bib30)\)\(distractor setting\), with*human*supporting\-fact annotations over ten candidate paragraphs \(15001500/seed×\\times3\), all scorers far exceed the random baseline0\.2010\.201\(Table[9](https://arxiv.org/html/2606.23915#A4.T9)c\); here, on short multi\-hop answers reusing the question’s entities,*lexical and NLI overtake semantic*— the mirror image of ASQA\.

#### Within provenance, no single component is universal; the blend is a convenient baseline\.

The three*provenance*benchmarks complicate any simple “semantic wins” summary: each single component has a dataset where it is weakest: semantic similarity on short, entity\-heavy multi\-hop answers \(HotpotQA\), and lexical overlap and off\-the\-shelf NLI on synthesized long\-form answers \(ASQA, where NLI falls to0\.5850\.585\)\. The*combined*score stays at or near the top across these*provenance*tasks \(Tables[2](https://arxiv.org/html/2606.23915#S4.T2)c,[9](https://arxiv.org/html/2606.23915#A4.T9); on the humanis\_selectedgate it leads at0\.7260\.726, and on the MS MARCO confusable top\-1 test it reaches0\.9960\.996\): within construct \(i\) only, the fixed blend is a convenient provenance baseline that stays at or near the top across these retrieval datasets\. This blend is*not*a general\-purpose*grounding*metric: on the human\-support tasks \(constructs ii, iii\) it is far from best \(Table[2](https://arxiv.org/html/2606.23915#S4.T2)\), consistent with our headline that no single audited automatic scorer transfers across the datasets of a construct\. The provenance behavior is stable across encoders and distractor counts:all\-MiniLM\-L6\-v2gives ASQA semantic accuracy0\.9190\.919–0\.9390\.939acrossk∈\{3,8,10\}k\{\\in\}\\\{3,8,10\\\}\(vs\.all\-mpnet\-base\-v2’s0\.9340\.934atk=5k\{=\}5\), with the combined−\-lexical gap positive \(\+0\.13\+0\.13to\+0\.15\+0\.15\) throughout\.

## 5ERCR as a Boundary Probe: Did QA\-EM, ROUGE\-L, and Citation Count Detect This Rewrite?

A negative result forced the measurement question above\. We trained a structured, claim\-level grounding reward, ERCR \(Evidence\-Relative Commitment Reward; a factorized deterministic reward over type\-aware claim support, counterfactual margin, evidence coverage, and epistemic commitment\) with GRPO\(Shao and others,[2024](https://arxiv.org/html/2606.23915#bib.bib3)\), expecting improved grounding; it did not move the answer\-level metrics\. We use that null as a probe: if QA\-EM and related metrics tracked grounding, optimizing a reward built to change grounding ought to perturb them; here, QA\-EM, ROUGE\-L, and citation count did not detect this large ASQA rewrite intervention\. We are*not*proposing ERCR as a method — we compare an ERCR\-LoRA adapter \(QLoRA, GRPO\-trained fromQwen2\.5\-\{3B,7B\}\-Instruct\) against the*unadapted*base \(a base\-vs\-base\+\+LoRA comparison, not SFT\-vs\-RL\), regenerating all948948ASQA dev answers greedily \(full setup in App\.[C](https://arxiv.org/html/2606.23915#A3)\)\.

#### ERCR moves no deterministic metric meaningfully\.

At 3B, QA\-EM \(corrected ASQAstr\_em, any\-alias match\) is0\.2990\.299for the base and0\.2970\.297for ERCR \(delta−0\.002\-0\.002,95%95\\%paired\-bootstrap CI\[−0\.011,\+0\.008\]\[\-0\.011,\+0\.008\]\): no statistically meaningful change\. ROUGE\-L moves\+0\.006\+0\.006and citation count is flat\. At 7B the pattern repeats \(QA\-EM0\.377→0\.3810\.377\\\!\\to\\\!0\.381, CI\[−0\.004,\+0\.012\]\[\-0\.004,\+0\.012\]\)\. Under a legacy substring metric the 7B values are0\.240→0\.2410\.240\\\!\\to\\\!0\.241, the same flat pattern\. We measure citation*count*, which is flat; we did not regenerate NLI\-based ALCE citation recall/precision\(Gaoet al\.,[2023](https://arxiv.org/html/2606.23915#bib.bib5)\)\(it requires a large entailment model and a different decoding setup\), and we make no claim about it here\.

#### The reward did change the text — the metrics just did not see it\.

The probe is not vacuous: at 3B, ERCR alters89\.9%89\.9\\%of the948948generations \(mean token Jaccard0\.590\.59;81\.2%81\.2\\%differ substantially, Jaccard<0\.9<0\.9\), yet QA\-EM, ROUGE\-L, and citation count register essentially nothing\. We do*not*claim ERCR improved grounding \(a single untuned run could be weak\), only that these answer\-level metrics did not detect this large ASQA rewrite — motivating the dataset\-level audit on evidence independent of the probe\.

## 6Conclusion

Across the audited datasets and under our operational near\-best screen, no single*automatic*attribution scorer is a reliable fixed default — including in generated\-answer attribution, the construct with the most multi\-dataset human\-labeled coverage \(the four AttributionBench sources,n=1610n\{=\}1610, with independent HAGRID,n=2150n\{=\}2150\)\. The construct shapes the appropriate family, but within answer attribution the best metric is unstable across datasets \(per\-dataset rankings discordant, KendallW=0\.07W\{=\}0\.07,p=0\.029p\{=\}0\.029; the NLI scorer flips from best on AttributedQA to chance on LFQA\), and even the lowest\-regret metric leaves measurable per\-dataset regret\. A prompt\-based LLM judge avoids the chance\-level collapses \(AUROC0\.730\.73–0\.920\.92, no LFQA collapse\) but is costlier and non\-deterministic, relocating rather than removing the problem\. Evaluations should report their dataset and construct and validate the metric on them\.

## 7Limitations

Construct and domain shift are entangled\.Our largest\-gap “NLI wins” contrast \(0\.810\.81–0\.890\.89vs\.≈0\.60\{\\approx\}0\.60AUROC\) is also our only out\-of\-domain benchmark \(VitaminC\), so we cannot fully separate construct from domain shift; the in\-domain AttributionBench result \(where the gap closes\) licenses the construct\-dependence claim, not a claim that entailment is intrinsically superior\.Three sentence/claim\-level human\-support benchmarks, all English\.Only AttributionBench, HAGRID, and VitaminC carry human*sentence/claim\-level*support labels \(BEGIN, App\.[E](https://arxiv.org/html/2606.23915#A5), has human labels but at the*response*level — an external boundary, not a main construct\); all are English\. Broader in\-construct human labels \(other domains/languages\) are the natural next step\.Provenance uses passage\-ranking labels, not the sentence\-attribution task of \(ii, iii\)\.We pose construct \(i\) as passage ranking: ASQA uses a provenance proxy, while MS MARCO \(is\_selected\) and HotpotQA \(supporting\-fact annotations\) provide human relevance labels\. We use all three only to ask whether a score ranks a relevant passage top,*not*to predict sentence/claim\-level support judgments\. We keep these strictly separate from the human\-support constructs and never pool them\.Decoding and correctness\.The ERCR regeneration uses greedy decoding \(the flatness reproduces, but absolute QA\-EM differs from prior temperature\-sampled prose\), and the full\-dev provenance gap is partly correctness\-confounded \(mitigated, not removed, by the matched subset\)\.Scope of the probe\.The ERCR probe shows insensitivity to a large output change; it does*not*establish those metrics would miss a*genuine*grounding improvement\. It is motivation, not proof; the diagnostic does not depend on it\.Item\-level, not system\-level\.Our consequence is at the level of choosing an evaluator under dataset uncertainty \(per\-item AUROC and the failure of a naive cross\-dataset selector\); we do*not*show that this metric instability reverses*system*rankings across generators\. Whether item\-level non\-transfer induces model\-selection reversals is important future work\.

## References

- QAMPARI: a benchmark for open\-domain questions with many answers\.InProceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics \(GEM\),Singapore,pp\. 97–110\.Cited by:[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px2.p1.1)\.
- N\. Dziri, H\. Rashkin, T\. Linzen, and D\. Reitter \(2022\)Evaluating attribution in dialogue systems: the BEGIN benchmark\.Transactions of the Association for Computational Linguistics10,pp\. 1066–1083\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00506)Cited by:[Table 10](https://arxiv.org/html/2606.23915#A5.T10),[Appendix E](https://arxiv.org/html/2606.23915#A5.p1.1),[§1](https://arxiv.org/html/2606.23915#S1.p3.1),[§4](https://arxiv.org/html/2606.23915#S4.SS0.SSS0.Px3.p2.1)\.
- S\. Es, J\. James, L\. Espinosa Anke, and S\. Schockaert \(2024\)RAGAs: automated evaluation of retrieval augmented generation\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations,St\. Julians, Malta,pp\. 150–158\.Cited by:[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Fan, Y\. Jernite, E\. Perez, D\. Grangier, J\. Weston, and M\. Auli \(2019\)ELI5: long form question answering\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,Florence, Italy,pp\. 3558–3567\.External Links:[Document](https://dx.doi.org/10.18653/v1/P19-1346)Cited by:[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px1.p1.1)\.
- T\. Gao, H\. Yen, J\. Yu, and D\. Chen \(2023\)Enabling large language models to generate text with citations\.InProceedings of EMNLP,Cited by:[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2606.23915#S5.SS0.SSS0.Px1.p1.9)\.
- O\. Honovich, R\. Aharoni, J\. Herzig, H\. Taitelbaum, D\. Kukliansy, V\. Cohen, T\. Scialom, I\. Szpektor, A\. Hassidim, and Y\. Matias \(2022\)TRUE: re\-evaluating factual consistency evaluation\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 3905–3920\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.naacl-main.287)Cited by:[§1](https://arxiv.org/html/2606.23915#S1.p3.1)\.
- C\. Huang, Z\. Wu, Y\. Hu, and W\. Wang \(2024\)Training language models to generate text with citations via fine\-grained rewards\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 2926–2949\.Cited by:[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Ji, N\. Lee, R\. Frieske, T\. Yu, D\. Su, Y\. Xu, E\. Ishii, Y\. Bang, A\. Madotto, and P\. Fung \(2023\)Survey of hallucination in natural language generation\.ACM Computing Surveys55\(12\),pp\. 1–38\.External Links:[Document](https://dx.doi.org/10.1145/3571730)Cited by:[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px1.p1.1)\.
- E\. Kamalloo, A\. Jafari, X\. Zhang, N\. Thakur, and J\. Lin \(2023\)HAGRID: a human\-LLM collaborative dataset for generative information\-seeking with attribution\.External Links:2307\.16883Cited by:[§4](https://arxiv.org/html/2606.23915#S4.SS0.SSS0.Px3.p1.6),[Table 2](https://arxiv.org/html/2606.23915#S4.T2.2.2.9.7.2)\.
- S\. Koneru, E\. Joe, C\. Kirchhoff, J\. Wu, and S\. Rajtmajer \(2026\)Evaluating evidence grounding under user pressure in instruction\-tuned language models\.External Links:2603\.20162Cited by:[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px4.p1.1)\.
- P\. Laban, T\. Schnabel, P\. N\. Bennett, and M\. A\. Hearst \(2022\)SummaC: re\-visiting NLI\-based models for inconsistency detection in summarization\.Transactions of the Association for Computational Linguistics10,pp\. 163–177\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00453)Cited by:[Table 13](https://arxiv.org/html/2606.23915#A6.T13),[Appendix F](https://arxiv.org/html/2606.23915#A6.p3.1),[§3\.1](https://arxiv.org/html/2606.23915#S3.SS1.SSS0.Px2.p1.16)\.
- S\. Lee, Y\. Jo, M\. Seo, M\. Lee, and M\. Seo \(2026\)Lost in the noise: how reasoning models fail with contextual distractors\.External Links:2601\.07226Cited by:[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px4.p1.1)\.
- Y\. Li, X\. Yue, Z\. Liao, and H\. Sun \(2024\)AttributionBench: how hard is automatic attribution evaluation?\.InFindings of the Association for Computational Linguistics: ACL 2024,Bangkok, Thailand,pp\. 14919–14935\.Cited by:[§1](https://arxiv.org/html/2606.23915#S1.p2.13),[§1](https://arxiv.org/html/2606.23915#S1.p3.1),[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px2.p1.1),[Table 2](https://arxiv.org/html/2606.23915#S4.T2.2.2.5.3.2),[§4](https://arxiv.org/html/2606.23915#S4.p1.12)\.
- N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang \(2024\)Lost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00638)Cited by:[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px1.p1.1)\.
- N\. F\. Liu, T\. Zhang, and P\. Liang \(2023\)Evaluating verifiability in generative search engines\.InFindings of the Association for Computational Linguistics: EMNLP 2023,Singapore,pp\. 7001–7025\.Cited by:[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Menick, M\. Trebacz, V\. Mikulik, J\. Aslanides, F\. Song, M\. Chadwick, M\. Glaese, S\. Young, L\. Campbell\-Gillingham, G\. Irving, and N\. McAleese \(2022\)Teaching language models to support answers with verified quotes\.External Links:2203\.11147Cited by:[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Min, K\. Krishna, X\. Lyu,et al\.\(2023\)FActScore: fine\-grained atomic evaluation of factual precision in long form text generation\.InProceedings of EMNLP,Cited by:[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px2.p1.1)\.
- T\. Nguyen, M\. Rosenberg, X\. Song, J\. Gao, S\. Tiwary, R\. Majumder, and L\. Deng \(2016\)MS MARCO: a human generated machine reading comprehension dataset\.InProceedings of the Workshop on Cognitive Computation \(CoCo\) at NIPS,External Links:1611\.09268Cited by:[§1](https://arxiv.org/html/2606.23915#S1.p2.13),[§4](https://arxiv.org/html/2606.23915#S4.SS0.SSS0.Px6.p2.17),[Table 2](https://arxiv.org/html/2606.23915#S4.T2.2.2.14.12.2)\.
- H\. Rashkin, V\. Nikolaev, M\. Lamm, L\. Aroyo, M\. Collins, D\. Das, S\. Petrov, G\. S\. Tomar, I\. Turc, and D\. Reitter \(2023\)Measuring attribution in natural language generation models\.Computational Linguistics49\(4\),pp\. 777–840\.External Links:[Document](https://dx.doi.org/10.1162/coli%5Fa%5F00486)Cited by:[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px2.p1.1)\.
- N\. Reimers and I\. Gurevych \(2019\)Sentence\-BERT: sentence embeddings using siamese BERT\-networks\.InProceedings of EMNLP\-IJCNLP,Cited by:[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2606.23915#S3.p1.11)\.
- T\. Schuster, A\. Fisch, and R\. Barzilay \(2021\)Get your vitamin C\! robust fact verification with contrastive evidence\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 624–643\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.52)Cited by:[§1](https://arxiv.org/html/2606.23915#S1.p2.13),[Table 2](https://arxiv.org/html/2606.23915#S4.T2.2.2.11.9.2),[§4](https://arxiv.org/html/2606.23915#S4.p1.12)\.
- Z\. Shaoet al\.\(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300Cited by:[§5](https://arxiv.org/html/2606.23915#S5.p1.2)\.
- S\. H\. Sim, T\. D\. Pala, V\. Toh, H\. L\. Chieu, A\. Zadeh, C\. Li, N\. Majumder, and S\. Poria \(2025\)Lessons from training grounded LLMs with verifiable rewards\.External Links:2506\.15522Cited by:[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Slobodkin, E\. Hirsch, A\. Cattan, O\. Ernst, and I\. Dagan \(2024\)Attribute first, then generate: locally\-attributable grounded text generation\.InProceedings of ACL,Cited by:[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px2.p1.1)\.
- I\. Stelmakh, Y\. Luan, B\. Dhingra, and M\. Chang \(2022\)ASQA: factoid questions meet long\-form answers\.External Links:2204\.06092Cited by:[§1](https://arxiv.org/html/2606.23915#S1.p2.13),[Table 2](https://arxiv.org/html/2606.23915#S4.T2.2.2.13.11.2)\.
- L\. Tang, P\. Laban, and G\. Durrett \(2024\)MiniCheck: efficient fact\-checking of LLMs on grounding documents\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[Table 11](https://arxiv.org/html/2606.23915#A6.T11),[§1](https://arxiv.org/html/2606.23915#S1.p2.13),[Table 2](https://arxiv.org/html/2606.23915#S4.T2),[§4](https://arxiv.org/html/2606.23915#S4.p1.12)\.
- S\. Wu, J\. Xie, J\. Chen, T\. Zhu, K\. Zhang, and Y\. Xiao \(2024\)How easily do irrelevant inputs skew the responses of large language models?\.InConference on Language Modeling \(COLM\),External Links:2404\.03302Cited by:[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px4.p1.1)\.
- Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning \(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Brussels, Belgium,pp\. 2369–2380\.Cited by:[§1](https://arxiv.org/html/2606.23915#S1.p2.13),[§4](https://arxiv.org/html/2606.23915#S4.SS0.SSS0.Px6.p3.3),[Table 2](https://arxiv.org/html/2606.23915#S4.T2.2.2.15.13.2)\.
- X\. Yue, B\. Wang, Z\. Chen, K\. Zhang, Y\. Su, and H\. Sun \(2023\)Automatic evaluation of attribution by large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2023,Singapore,pp\. 4615–4635\.Cited by:[§1](https://arxiv.org/html/2606.23915#S1.p3.1),[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Zha, Y\. Yang, R\. Li, and Z\. Hu \(2023\)AlignScore: evaluating factual consistency with a unified alignment function\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[Table 8](https://arxiv.org/html/2606.23915#A4.T8),[Appendix D](https://arxiv.org/html/2606.23915#A4.p1.3)\.
- J\. Zhang, Y\. Bai, X\. Lv,et al\.\(2024\)LongCite: enabling LLMs to generate fine\-grained citations in long\-context QA\.External Links:2409\.02897Cited by:[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px2.p1.1)\.
- T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. Artzi \(2020\)BERTScore: evaluating text generation with BERT\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§4](https://arxiv.org/html/2606.23915#S4.p1.12)\.
- Y\. Zhao, J\. Zhu, Z\. Ding, and X\. Li \(2026\)GRACE: reinforcement learning for grounded response and abstention under contextual evidence\.External Links:2601\.04525Cited by:[§2](https://arxiv.org/html/2606.23915#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix AASQA Internal Consistency Checks

For completeness we report two ASQA checks that we treat as internal consistency checks,*not*validation, because their labels are threshold heuristics derived from the same lexical/semantic features under test \(hence circular\)\.\(1\) Gold\-slice retrieval:on a 20\-example slice with 2–3 distractor passages, top\-1 retrieval is0\.9660\.966combined /0\.9150\.915lexical /0\.9660\.966semantic against heuristic “gold” labels\.\(2\) Matched\-pair agreement:on1010strong/decent pairs the score’s preference agrees10/1010/10with a lexical\-overlap groundedness proxy \(gap\+0\.185\+0\.185\)\. Both are corroborative only; the human\-labeled constructs in Table[2](https://arxiv.org/html/2606.23915#S4.T2)carry the paper’s claims\. The full gold\-reference vs\. unadapted\-model gates and the answer\-hint ablation are in Tables[3](https://arxiv.org/html/2606.23915#A1.T3)–[4](https://arxiv.org/html/2606.23915#A1.T4)\.

Table 3:Three\-gate internal\-consistency check on ASQA dev\. Labels are heuristic/proxy, not human\. The correctness\-matched subset \(Gate 3, row 4\) shows the largest gap\.Table 4:Answer\-hint ablation on Gate 3 \(ASQA full dev,n=623n\{=\}623; matched subsetn=98n\{=\}98\)\. Removing the label\-aware answer\-hint term \(wh=0w\_\{h\}\{=\}0, other weights fixed\) leaves the grounding gap essentially intact, with the matched\-subset CI still excluding zero\. The gap is not an artifact of the label\-aware term\.
## Appendix BPer\-Component Ablation

Table[5](https://arxiv.org/html/2606.23915#A2.T5)reports a leave\-one\-out ablation of the four score components on the Gate\-3 strong−\-decent gap\. It complements the lexical\-weight sweep \(Section[3\.1](https://arxiv.org/html/2606.23915#S3.SS1.SSS0.Px1)\) and the answer\-hint ablation \(Section[3\.1](https://arxiv.org/html/2606.23915#S3.SS1)\): the lexical and semantic terms each carry a substantial share of the gap, while the answer\-hint and citation terms are minor — consistent with the main\-text finding that the score’s signal does not depend on the label\-aware hint\.

Table 5:Per\-component leave\-one\-out on the Gate\-3 strong−\-decent attribution gap \(ASQA full dev; matched subsetn=98n\{=\}98, full devn=623n\{=\}623; all matched CIs exclude0\)\. Dropping lexical or semantic shrinks the gap most \(both contribute\); dropping the label\-aware hint barely changes it \(\+0\.160→\+0\.153\+0\.160\\\!\\to\\\!\+0\.153, confirming the gap is not a hint artifact\); dropping citation is harmless\. The lexical\-only and semantic\-only rows show each of those two components carries a gap on its own\.![Refer to caption](https://arxiv.org/html/2606.23915v1/x1.png)Figure 1:Lexical\-weight ablation under confusable evidence \(construct i\)\. Top\-1 provenance accuracy is flat at lowα\\alphaand then declines as the lexical weightα\\alpharises; the canonical blend \(α≈0\.47\\alpha\\approx 0\.47\) is already past the knee\. Semantic\-only \(α=0\\alpha\{=\}0\) is best\.
## Appendix CERCR Training Details

For reproducibility, Table[6](https://arxiv.org/html/2606.23915#A3.T6)lists the LoRA and GRPO configuration used to train the ERCR adapters, recovered from the saved adapter configs and the training setup\. We reiterate that ERCR functions in this paper as a metric\-sensitivity probe \(Section[5](https://arxiv.org/html/2606.23915#S5)\), not a proposed method; we report these settings for methodological transparency, not as a tuned recipe\. Training\-time loss and reward trajectories were not retained alongside the saved adapters, so we do not report them\.

Table 6:ERCR training and evaluation configuration, recovered from the saved LoRA adapter configs and the GRPO setup\. We report these for methodological transparency; ERCR is a probe \(Section[5](https://arxiv.org/html/2606.23915#S5)\), not a proposed method\. Training\-time loss/reward trajectories were not retained with the saved adapters and are not reported\.ModelQA\-EMROUGE\-LCites*3B*base \(unadapted\)0\.2990\.2191\.05\+ ERCR\-LoRA0\.2970\.2251\.04Δ\\DeltaQA\-EM \-0\.002, CI\[\-0\.011,\+0\.008\]*7B*base \(unadapted\)0\.3770\.1911\.83\+ ERCR\-LoRA0\.3810\.1931\.83Δ\\DeltaQA\-EM \+0\.004, CI\[\-0\.004,\+0\.012\]Table 7:ERCR negative result on full ASQA dev \(948 ex, greedy decoding\)\. QA\-EM is corrected ASQAstr\_em\. ERCR produces no statistically meaningful change at either scale\.
## Appendix DSupplementary Metric Families

Beyond the eight metrics of Table[2](https://arxiv.org/html/2606.23915#S4.T2), we ran three additional supplementary metrics on the AttributionBench sources to check that the per\-source pattern is not specific to our metric set: a cross\-encoder reranker \(ms\-marco\-MiniLM\-L6\), MiniCheck\-FT5 \(flan\-t5\-large\), and AlignScore\-large\(Zhaet al\.,[2023](https://arxiv.org/html/2606.23915#bib.bib36)\), a metric purpose\-built for factual consistency\. Table[8](https://arxiv.org/html/2606.23915#A4.T8)shows all three reproduce the pattern, and in particular all are near chance on ExpertQA \(AUROC0\.580\.58–0\.600\.60, chance=0\.50=0\.50\) — so the weak performance there is not isolated to the originally audited scorers\.

Table 8:Three supplementary metrics reproduce the per\-source pattern\(AUROC on AttributionBench human attribution labels\): a cross\-encoder reranker \(ms\-marco\-MiniLM\-L6\), MiniCheck\-FT5 \(flan\-t5\-large\), and AlignScore\-large\(Zhaet al\.,[2023](https://arxiv.org/html/2606.23915#bib.bib36)\)\. All three reach AUROC0\.8670\.867–0\.9010\.901on AttributedQA, weaker but usable on LFQA/Stanford, and weak on ExpertQA \(AUROC0\.580\.58–0\.600\.60\) — matching the eight metrics of Table[2](https://arxiv.org/html/2606.23915#S4.T2)\. ExpertQA is weak for every scorer we tried \(AUROC0\.580\.58–0\.620\.62, only just above chance\), so its low ceiling is not isolated to any single metric\.*\(a\) Human passage\-relevance gate*\(top\-1is\_selected\)ScorerTop\-1 acc\.vs\. randomRandom passage0\.104—Lexical\-only0\.683\+0\.579\+0\.579Semantic\-only0\.646\+0\.542\+0\.542NLI entailment0\.708\+0\.604\+0\.604Combined \(canon\.\)0\.726\+0\.622\+0\.622*\(b\) MS MARCO confusable test*\(top\-1 provenance\)ScorerTop\-1 acc\.FooledLexical\-only0\.8850\.115Semantic\-only0\.9950\.005Combined \(canon\.\)0\.9960\.004NLI entailment†0\.9520\.048*\(c\) HotpotQA human supporting\-fact gate*ScorerTop\-1 acc\.vs\. randomRandom paragraph0\.201—Lexical\-only0\.959\+0\.758\+0\.758Semantic\-only0\.938\+0\.737\+0\.737NLI entailment0\.979\+0\.778\+0\.778Combined \(canon\.\)0\.972\+0\.771\+0\.771Table 9:Additional benchmarks \(best in bold\)\.\(a\)MS MARCO human passage\-relevance gate: rank a query’s retrieved passages per answer sentence; ground truth is the*human*is\_selectedlabel \(passage\-level, not sentence/claim\-level support\),20002000queries×\\times3 seeds\.\(b\)MS MARCO confusable\-evidence test \(Section[3\.1](https://arxiv.org/html/2606.23915#S3.SS1.SSS0.Px1)design\),n=300×4n\{=\}300\\times 4seeds: the combined/semantic advantage over lexical holds \(gap\+0\.111\+0\.111; four\-seed gaps\+0\.097\+0\.097to\+0\.130\+0\.130, all CIs\>0\{\>\}0\)\.\(c\)HotpotQA distractor\-setting human supporting\-fact gate,1500×31500\\times 3seeds: rank a human\-supporting paragraph top \(2\-hop, so≈2\\approx 2supporting among 10; random0\.2010\.201\)\. On these extractive/multi\-hop answers lexical and NLI overtake semantic — the mirror image of ASQA — yet the combined score stays near\-best on every benchmark\.†NLI is a separaten=200×3n\{=\}200\\times 3subsample\.
## Appendix EBEGIN: Appendix Boundary Check \(Dialogue, Response\-Level Labels\)

As an external, different\-domain check \(knowledge\-grounded*dialogue*rather than QA\), we score the metric zoo on BEGIN\(Dziriet al\.,[2022](https://arxiv.org/html/2606.23915#bib.bib33)\),n=11,466n\{=\}11\{,\}466system responses with human “fully attributable” vs “not” labels \(we drop the “generic” class\), using knowledge as evidence and the response as the claim\. Table[10](https://arxiv.org/html/2606.23915#A5.T10)shows the grounding\-trained family \(clean MNLI, MiniCheck\) has top AUROC on every dataset here — the opposite of LFQA, where clean MNLI collapsed\. This is a boundary condition that sharpens, rather than contradicts, non\-portability: an otherwise top\-scoring family still has a dataset \(LFQA\) where it fails, so no single metric is the lowest\-regret fixed choice across datasets\. We treat BEGIN’s response\-level binary labels as related to, but distinct from, the sentence/claim\-level attribution of the main constructs, and do not pool them\.

Table 10:BEGIN dialogue\-attribution\(Dziriet al\.,[2022](https://arxiv.org/html/2606.23915#bib.bib33)\)AUROC \(knowledge\-grounded dialogue;n=11,466n\{=\}11\{,\}466,2,8782\{,\}878attributable; best per column in bold\)\. In this*new domain*, the grounding\-trained family \(NLIcl, MiniCheck\) is uniformly strongest, in contrast to the QA generated\-answer attribution of Table[2](https://arxiv.org/html/2606.23915#S4.T2)\(a\) where NLIcl’s rank was unstable\. BEGIN is a boundary condition — a dataset where these metrics rank at the top across all three sub\-domains — which is consistent with \(not a contradiction of\) non\-portability: the same NLIclcollapses on LFQA\.
## Appendix FIndependent Replication and Robustness

Independent multi\-dataset replication\.To test whether non\-transfer holds beyond AttributionBench, we score the same zoo on eight*disjoint*LLM\-AggreFact source datasets \(excluding the two, ExpertQA and LFQA, that overlap our audit;n=7336n\{=\}7336pairs\)\. No metric transfers there either \(Table[11](https://arxiv.org/html/2606.23915#A6.T11)\): four different metrics win across the eight datasets and the same clean MNLI scorer swings AUROC0\.494→0\.9460\.494\\\!\\to\\\!0\.946— the winning metric changes by dataset, the signature of a genuine crossover rather than a uniform difficulty shift\.

Table 11:Independent replication: no audited metric transfers across a second, disjoint multi\-dataset benchmark\.AUROC against*human*support labels on eight LLM\-AggreFact\(Tanget al\.,[2024](https://arxiv.org/html/2606.23915#bib.bib35)\)source datasets \(n=7336n\{=\}7336pairs\), excluding the two \(ExpertQA, LFQA\) that overlap our AttributionBench audit so the panel is independent; best per row in bold\. As in Table[2](https://arxiv.org/html/2606.23915#S4.T2), no single audited scorer is best across datasets: four different metrics win the eight rows — BERTScore \(2\), MiniCheck \(4\), clean MNLI \(Reveal\), and the lexical/semantic blend \(Comb, TofuEval\-MeetB\) — the same clean MNLI scorer swings from AUROC0\.4940\.494\(Wice,≤\\leqchance\) to0\.9460\.946\(Reveal\), so the winning metric changes by dataset, and committing to the lowest\-regret single metric \(MiniCheck\) still leaves mean leave\-one\-dataset\-out regret0\.0150\.015\(max0\.0990\.099\)\. This is a faithfulness/grounding benchmark \(claim\-vs\-document support\), adjacent to but distinct from sentence\-level citation attribution; we report it as an independent*boundary replication*and do*not*pool it with the AttributionBench audit\.Portability\-criterion sensitivity\.The non\-transfer conclusion does not hinge on the CI\-relative screen: anϵ\\epsilon\-regret sweep, Kendall\-τ\\taurank\-inversion, a cluster\-respecting permutation, paired best\-vs\-rest tests, and threshold\-free AUPRC/balanced\-accuracy all agree \(Table[12](https://arxiv.org/html/2606.23915#A6.T12)\)\.

Table 12:The non\-transfer conclusion does not hinge on the CI\-relative criterion\.Four independent readings agree on AttributionBench’s four sources: \(a\) under anϵ\\epsilon\-regret screen no audited metric is withinϵ\\epsilonof the per\-dataset oracle on*every*dataset for anyϵ≤0\.05\\epsilon\\leq 0\.05; \(b\) metric rankings are uncorrelated\-to\-negative across datasets \(the AttributedQA/LFQA ranking literally inverts\); \(c\) the crossover survives controlling for dataset difficulty — within\-dataset rank concordance is near zero \(KendallW=0\.07W\{=\}0\.07,p=\.029p\{=\}\.029\) and a residual permutation that holds each dataset’s difficulty fixed puts35%35\\%of the cross\-cell variance in the interaction residual \(borderline,p=\.053p\{=\}\.053; the looser exchangeability shuffle that conflates difficulty with interaction givesp<\.002p\{<\}\.002and is shown for reference only\); \(d\) the headline LFQA reversal is decisive \(BERTScore beats clean MNLI by0\.3780\.378\)\.Honestly, “no single metric transfers” is robust, but the best metric is not always*uniquely*best: on LFQA, BERTScore vs\. MiniCheck is not significant \(p=0\.127p=0\.127\)\. All quantities are recomputed and match the Table[2](https://arxiv.org/html/2606.23915#S4.T2)AUROCs\.A descriptive correlate of the collapse\.Evidence length / sentence count tracks where clean MNLI fails \(Table[13](https://arxiv.org/html/2606.23915#A6.T13)\), consistent with single\-premise entailment degrading under cross\-sentence aggregation\(Labanet al\.,[2022](https://arxiv.org/html/2606.23915#bib.bib25)\); we report this as descriptive, not causal\.

Table 13:A descriptive correlate of the clean\-MNLI collapse: evidence length / sentence count\.Mean evidence length \(words\) and evidence sentence count per AttributionBench source, against per\-dataset AUROC\. The clean MNLI scorer is strongest where evidence is short and few\-sentence \(AttributedQA: 103 words, 4\.2 sentences→\\to0\.9040\.904\) and collapses where evidence is long and many\-sentence \(LFQA: 321 words, 13\.9 sentences→\\to0\.5310\.531\), while token\-level BERTScore moves the opposite way \(0\.692→0\.9090\.692\\to 0\.909\)\. This is consistent with a single\-premise entailment head degrading when support must be aggregated across many sentences\(Labanet al\.,[2022](https://arxiv.org/html/2606.23915#bib.bib25)\)\. We report this as adescriptivecorrelate, not a causal mechanism: with four datasets the dataset\-level association is underpowered, the per\-example abstractiveness signal does not cleanly separate NLI from BERTScore, and ExpertQA is a near\-chance exception\. Identifying the causal driver is left to future work\.Is a prompt\-based LLM judge more stable across datasets?We also run a prompt\-based LLM judge \(Opus 4\.8\) as a scoring metric across the four AttributionBench sources \(Table[14](https://arxiv.org/html/2606.23915#A6.T14)\)\. On those same four sources it never drops to the chance\-level collapses the strong automatic scorers suffer \(range0\.7310\.731–0\.9180\.918vs\. clean MNLI’s0\.5310\.531–0\.9040\.904on the four\) and does not collapse on LFQA, but is not uniformly best, costs∼100×\{\\sim\}100\\timesmore, and is non\-deterministic\. Its AUROC is computed over the408408of480480sampled items it actually scored \(15%15\\%refused or truncated, dropped rather than imputed\), so any upward bias from dropping hard cases would only*narrow*its apparent stability advantage\. The automatic\-metric non\-transfer is real; the LLM judge avoids the collapses but is not a free substitute for target\-dataset validation\.

Table 14:A prompt\-based LLM judge avoids the chance\-level collapses the strong automatic scorers suffer, but is neither free nor uniformly best\.An LLM judge \(Claude Opus 4\.8, scored as a graded support probability, AUROC against the*human*AttributionBench labels;n=408n\{=\}408parsed of480480, balanced; see App\.[F](https://arxiv.org/html/2606.23915#A6)\) ranges AUROC0\.7310\.731–0\.9180\.918\(spread0\.1870\.187\) across the four sources, never dropping to the chance\-level collapses the strong automatic scorers suffer \(on the same four sources clean MNLI spans0\.5310\.531–0\.9040\.904, BERTScore0\.5690\.569–0\.9090\.909\); a narrower\-range metric like MPNet \(spread0\.1610\.161\) is more uniform but never strong\. The judge does*not*collapse on long\-form LFQA \(0\.8780\.878vs\. clean MNLI0\.5310\.531\) and gains the most over the best automatic scorer on the underspecified ExpertQA \(0\.7310\.731vs\.∼0\.57\{\\sim\}0\.57–0\.620\.62\), though that is also its lowest absolute AUROC\. But it still loses to BERTScore on LFQA/Stanford, costs∼100×\{\\sim\}100\\timesmore per judgment, and is non\-deterministic in practice \(15%15\\%refusal/truncation, dropped not coerced\)\. The automatic\-metric non\-transfer is therefore real on these datasets; the LLM judge avoids the collapses here, but is not a free substitute for reporting and validating on the target dataset/construct\. Disclosed as an LLM judge \(single model family, Opus 4\.8\), not human/gold\.LLM\-judge prompt templates\.For reproducibility we give the exact prompts verbatim \(Figure[2](https://arxiv.org/html/2606.23915#A6.F2)\)\. The judge runs as a*scoring metric*:claude\-opus\-4\-8, greedy decoding,maxTokens=20=20, evidence/claim/question truncated to15001500/600600/300300characters; the support probability is parsed with a regex and clamped to\[0,1\]\[0,1\], with refusals/parse\-failures dropped \(not imputed\)\. The separate cross\-family*annotation*probe \(Section[4](https://arxiv.org/html/2606.23915#S4); Opus 4\.8 and GPT\-5\.4,n=160n\{=\}160\) instead elicits a discrete label, scored as inter\-annotator agreement, never as gold\.

\(a\) Graded scoring prompt\(LLM judge as a metric\):

```
You are an automated ATTRIBUTION-scoring
rubric for a QA system. Given a CLAIM (a
sentence from a generated answer) and the
EVIDENCE it cites, rate how fully the claim
is supported by (attributable to) the
evidence.

Output EXACTLY one line and nothing else,
even if uncertain:
SUPPORT: <number 0.00-1.00>

1.00 = every factual element of the claim is
directly supported by the evidence; 0.00 =
the evidence is unrelated to or contradicts
the claim; intermediate values for partial
support. Use the full range and be
calibrated. This is an automated rubric:
always return a number (use 0.50 if truly
unsure). Do not refuse and do not explain.

QUESTION: {question}
CLAIM:    {claim}
EVIDENCE: {evidence}
```

\(b\) Discrete annotation prompt\(cross\-familyκ\\kappaprobe\):

```
You are judging ATTRIBUTION for a question-
answering system. Given a CLAIM (a sentence
from a generated answer) and the EVIDENCE it
cites, decide: is the claim fully supported
by (attributable to) the evidence?

Answer STRICTLY in this format on one line:
LABEL: <attributable|not_attributable>
CONF:  <0-1>

A claim is "attributable" only if all of its
factual content is supported by the evidence.
If the evidence is unrelated, contradicts, or
only partially supports the claim, answer
not_attributable.

QUESTION: {question}
CLAIM:    {claim}
EVIDENCE: {evidence}
```

Figure 2:Verbatim LLM prompt templates\.\(a\) The graded support\-probability prompt used to run Claude Opus 4\.8 as a scoring metric \(Table[14](https://arxiv.org/html/2606.23915#A6.T14)\); \(b\) the discrete\-label prompt used by then=160n\{=\}160cross\-family adjudication probe \(Opus 4\.8 and GPT\-5\.4\)\.\{question\},\{claim\}, and\{evidence\}are filled per item\. Both are disclosed as LLM outputs, never human/gold\.Cross\-benchmark landscape\.Pooling per\-dataset AUROCs across all17human\-labeled datasets we audit \(Table[15](https://arxiv.org/html/2606.23915#A6.T15)\), six different metrics are best on at least one dataset, the most frequent winner \(MiniCheck\) is best on only6/176/17\(35%35\\%\), and every metric’s AUROC swings by0\.340\.34–0\.450\.45across datasets\. \(Descriptive landscape; constructs are tagged and not pooled into a single test\.\)

Table 15:Cross\-benchmark landscape: every audited metric swings widely across datasets, and no metric dominates\.Per\-metric AUROC range across the17 human\-labeled datasetswe audit, spanning four evaluation settings \(5 generated\-answer attribution incl\. HAGRID; 1 fact\-check; 3 dialogue\-attribution sub\-domains; 8 LLM\-AggreFact faithfulness datasets\)\. Six different metrics are best on at least one dataset; the most frequent winner \(MiniCheck\) is best on only6/176/17\(35%35\\%\)\. The same clean MNLI scorer spans AUROC0\.4940\.494\(below chance\) to0\.9460\.946— a0\.450\.45swing\. The best fixed default \(MiniCheck, mean AUROC0\.7910\.791\) still leaves mean leave\-one\-dataset\-out regret0\.0260\.026\(max0\.0990\.099\)\. This is a*descriptive*landscape pooling the per\-dataset AUROCs reported in the paper; datasets span distinct constructs and are*not*pooled into a single significance test\.