Evaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis
Summary
This paper evaluates citation faithfulness in agentic scientific synthesis systems, showing that current verifiers are unreliable with unsupported-citation rates varying from 3% to 18% depending on strictness. It proposes a gold-anchored evaluation protocol and a deployable guard that uses split-conformal prediction to provide a distribution-free bound on truly unsupported citations.
arXiv:2607.20527v1 Announce Type: new
Abstract: Agentic LLM systems such as OpenScholar and PaperQA2 read the scientific literature and return cited answers, and both they and their benchmarks already check whether those citations hold, with a fixed attribution model or human graders. Neither audits the reliability of that check itself. We show it is not reliable, and that this matters. On identical agent outputs the measured unsupported-citation rate ranges from about 3% to about 18% depending only on the verifier's strictness, and although verifiers agree on which citations are supported, they disagree on which to flag (negative-specific agreement 0.27 to 0.30), so no single flag set is trustworthy and cross-paper comparison is invalid without a named verifier and protocol.
We present a gold-anchored evaluation protocol and a deployable guard that make this behavior measurable and bounded. The protocol validates the verifier, measures re-attribution, and calibrates a guarantee against human gold rather than another model's verdict; the verifier is a swappable instrument chosen on cost (recall 0.94 on the supported class, held out), and re-attribution is a commodity step where a deterministic BM25 matches the best open generator. The guard adds a split-conformal layer placing a distribution-free, finite-sample bound on truly unsupported citations that slip past a chosen flagging rule, a guarantee on catch rate rather than conclusion correctness. The bound holds on held-out gold, and we identify and quantify the condition governing its transfer to deployment, calibration-negative difficulty, with a concrete recalibration recipe, left untested by prior conformal-factuality work.
Validated across four open 27-35B models and three agentic pipelines on public benchmarks (SciFact, QASA, PubMedQA), with confidence intervals on every headline number, the protocol and guard ship as an open single-GPU kit.
View Cached Full Text
Cached at: 07/24/26, 05:05 AM
# Evaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis Source: [https://arxiv.org/html/2607.20527](https://arxiv.org/html/2607.20527) ,Junsik KimSeoul National UniversitySeoulRepublic of Korea,Kyulhee HanBioNexusSuwonRepublic of Korea,GwonYul JoBioNexusSuwonRepublic of Korea,Jong\-Soo KimBioNexusSuwonRepublic of KoreaandTae\-Hyung Kim[0000\-0001\-7280\-3978](https://orcid.org/0000-0001-7280-3978)BioNexusSuwonRepublic of Korea[thkim@bionexus\.kr](https://arxiv.org/html/2607.20527v1/mailto:[email protected]) ###### Abstract\. Agentic LLM systems such as OpenScholar and PaperQA2 read the scientific literature and return cited answers, and both they and their benchmarks already check whether those citations hold, with a fixed attribution model or human graders\. Neither audits the reliability of that check itself\. We show it is not reliable, and that this matters\. On identical agent outputs the measured unsupported\-citation rate ranges from about 3% to about 18% depending only on the verifier’s strictness, and although verifiers agree on which citations are*supported*, they disagree on which to*flag*\(negative\-specific agreement 0\.27 to 0\.30\), so no single flag set is trustworthy and cross\-paper comparison is invalid without a named verifier and protocol\. We present a gold\-anchored evaluation protocol and a deployable guard that make this behavior measurable and bounded\. The protocol validates the verifier, measures re\-attribution, and calibrates a guarantee against human gold rather than another model’s verdict; the verifier is a swappable instrument chosen on cost \(recall 0\.94 on the supported class, held out\), and re\-attribution is a commodity step where a deterministic BM25 matches the best open generator\. The guard adds a split\-conformal layer placing a distribution\-free, finite\-sample bound on truly unsupported citations that slip past a chosen flagging rule, a guarantee on catch rate rather than conclusion correctness\. The bound holds on held\-out gold, and we identify and quantify the condition governing its transfer to deployment, calibration\-negative difficulty, with a concrete recalibration recipe, left untested by prior conformal\-factuality work\. Validated across four open 27–35B models and three agentic pipelines on public benchmarks \(SciFact, QASA, PubMedQA\), with confidence intervals on every headline number, the protocol and guard ship as an open single\-GPU kit\. citation faithfulness, attribution evaluation, conformal prediction, LLM\-as\-judge reliability, trustworthy AI ††copyright:none††journal:TIST## 1\.Introduction Agentic literature\-synthesis systems such as OpenScholar\(Asaiet al\.,[2026](https://arxiv.org/html/2607.20527#bib.bib1)\)and PaperQA2\(Skarlinskiet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib2)\)now operate in a closed loop of retrieval, reasoning, and writing, returning answers with citations to the scientific record\. Such systems are part of a broader move toward agentic AI for science\(Wanget al\.,[2023](https://arxiv.org/html/2607.20527#bib.bib10); Gaoet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib9); Gottweiset al\.,[2026](https://arxiv.org/html/2607.20527#bib.bib4); Ghareebet al\.,[2026](https://arxiv.org/html/2607.20527#bib.bib8)\)\. For these agents the citation is the trust contract, telling the reader that a specific claim is backed by a specific source\. A growing body of work checks that contract automatically and reports that AI citations are frequently unsupported\(Gaoet al\.,[2023](https://arxiv.org/html/2607.20527#bib.bib36); Wuet al\.,[2025](https://arxiv.org/html/2607.20527#bib.bib45)\), and that even a factually correct citation may not faithfully reflect the passage the model actually used\(Wallatet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib42)\)\. The agentic systems themselves do this too\. OpenScholar scores citation support with a fixed attribution model and PaperQA2 with human graders\. Yet these evaluations take the checker for granted, reporting a citation\-support number without asking whether that judgment is itself reliable\. Evaluating this behavior at deployment scale is itself an intelligent\-systems problem, because the judgment must then be made by a model rather than by the human graders a one\-off benchmark can afford\. We begin from an uncomfortable observation:*the reported unsupported\-citation rate depends on the verifier, and the verifiers do not agree*\. Deciding whether a passage “supports” a claim is a judgment, and automated verifiers make it inconsistently and in disagreement with one another\. This is one instance of a fragility already documented for model graders in general\(Liet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib41); Haldar and Hockenmaier,[2025](https://arxiv.org/html/2607.20527#bib.bib31)\): whenever an evaluation hands a judgment to a model, the reported metric inherits that judge’s idiosyncrasies\. Three gaps follow for benchmarking agentic citation faithfulness\.\(G1\)The reported unsupported\-citation rate is verifier\-dependent, so a single number does not compare across studies\.\(G2\)Verifiers agree on what is*supported*but disagree on what to*flag*, so no single deployment flag set can be trusted\.\(G3\)No method places a distribution\-free guarantee on how many truly unsupported citations slip through a chosen flagging rule\. Our contribution is not the observation behind G1 and G2 but its consequence for benchmarking, and a remedy: anchoring the metric on human gold to make G1 and G2 measurable, and wrapping an imperfect verifier in a conformal guarantee that closes G3\. If deployment judgments cannot be trusted, the responsible move is to anchor evaluation on human gold labels, where “supported” has a checkable meaning\. We do this at three points: we*validate*the verifier against gold rather than choosing it by reputation; we*measure*re\-attribution accuracy against gold\-labeled supporting passages rather than against another verifier; and we*calibrate*a distribution\-free guarantee on gold\. The result is both an evaluation protocol and a deployable guard that wraps an imperfect verifier in a finite\-sample bound\. Everything ships as an open single\-GPU kit\. Figure[1](https://arxiv.org/html/2607.20527#S1.F1)summarizes the design end to end\. The result is a new evaluation protocol, benchmark methodology, and reliability guard for a critical agentic behavior, faithful citation, in scientific\-synthesis agents, treated as an intelligent\-systems evaluation problem in which the deployment metric is itself produced by a model\. The work spans formal analysis \(a distribution\-free finite\-sample guarantee with a proof\), evaluation methodology \(the gold\-anchored protocol\), and a deployable system \(a single\-GPU lab\-audit kit\)\. #### Contributions\. - •An agentic\-evaluation finding \(G1, G2\)\.On agentic scientific synthesis we quantify how unreliable deployment citation\-faithfulness judgments are: the same agent outputs score about 3% to about 18% unsupported across five gold\-validated verifiers, and although the verifiers agree on which citations are supported, they disagree on which to flag \(negative\-specific agreement 0\.27 to 0\.30 among the three continuous\-scored verifiers\)\. That model graders are imperfect is known; we show it leaves no single deployment flag set trustworthy and makes cross\-paper comparison invalid without a named verifier and protocol\. - •A gold\-anchored evaluation protocol\.We validate the verifier, measure re\-attribution, and calibrate the guarantee against human gold labels rather than against another verifier’s verdict, turning an unreliable judgment into a checkable measurement\. - •A distribution\-free guard \(G3\)\.Split\-conformal calibration on gold converts an imperfect, moderate\-agreement verifier into a finite\-sample bound on unflagged\-unsupported citations at a chosen tolerance; the bound holds empirically on held\-out gold, and we show its transfer is governed by calibration\-negative difficulty, so a deployment recalibrates on target\-domain negatives rather than inheriting our threshold\. - •An open, reproducible kit\.The protocol and guard run on one 80GB GPU or on CPU and are released under the MIT license at[https://github\.com/GooTec/citation\-guard](https://github.com/GooTec/citation-guard), so a laboratory can audit an agentic synthesis system on hardware it already has\. Figure 1\.Study design\. The agentic systems under audit \(top\) produce cited answers; four evaluation stages then anchor a verifier\-dependent metric on human gold and guarantee the catch rate\.S1 Rate:the unsupported\-citation rate has no single value across verifiers\.S2 Verifier:on SciFact gold the verifiers separate comparably at a matched catch rate, so the cheap local AttrScore\-3B is adopted on cost\.S3 Calibration:split\-conformal calibration on QASA gold turns the verifier score into a distribution\-free catch\-rate guarantee \(the contribution\)\.S4 Re\-attribution:recovering the true source is a swappable commodity slot \(lexical BM25 on CPU, or a generator on GPU\)\. The adopted guard \(verify→\\rightarrowre\-attribute→\\rightarrowflag\) runs on one GPU\. ## 2\.Related Work ### 2\.1\.Agentic AI for scientific synthesis LLM agents increasingly conduct literature synthesis and discovery, from OpenScholar and PaperQA2\(Asaiet al\.,[2026](https://arxiv.org/html/2607.20527#bib.bib1); Skarlinskiet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib2)\)to autonomous chemistry and biology agents\(Boikoet al\.,[2023](https://arxiv.org/html/2607.20527#bib.bib5); M\. Branet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib6); Swansonet al\.,[2025](https://arxiv.org/html/2607.20527#bib.bib7); Ghareebet al\.,[2026](https://arxiv.org/html/2607.20527#bib.bib8)\)and co\-scientist systems\(Gottweiset al\.,[2026](https://arxiv.org/html/2607.20527#bib.bib4)\)\. These pipelines retrieve, reason, and write in a loop, often with self\-reflection\(Shinnet al\.,[2023](https://arxiv.org/html/2607.20527#bib.bib14); Asaiet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib12)\), meta\-knowledge\-guided self\-correction\(Zhanget al\.,[2026](https://arxiv.org/html/2607.20527#bib.bib67)\), and corrective retrieval\(Yanet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib13)\)\. Their scientific value rests on faithful citation, yet studies warn that agentic reasoning can also produce confident, well\-formed, but unsupported claims\(Messeri and Crockett,[2024](https://arxiv.org/html/2607.20527#bib.bib11); Wallatet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib42)\)\. ### 2\.2\.Evaluating and benchmarking agentic systems A large benchmarking literature evaluates agents on task success: AgentBench\(Liuet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib19)\), GAIA\(Mialonet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib20)\),τ\\tau\-bench\(Yao and others,[2024](https://arxiv.org/html/2607.20527#bib.bib21)\), AgentBoard\(Maet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib22)\), tool use\(Patil and others,[2025](https://arxiv.org/html/2607.20527#bib.bib23); Guo and others,[2024](https://arxiv.org/html/2607.20527#bib.bib25)\), and software engineering\(Jimenezet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib24)\), and surveys chart LLM and LLM\-agent evaluation broadly\(Changet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib65); Guanet al\.,[2026](https://arxiv.org/html/2607.20527#bib.bib66)\)\. A recent certification scheme issues assurances for LLM\-based applications\(Benaet al\.,[2026](https://arxiv.org/html/2607.20527#bib.bib64)\)\. These measure whether an agent reaches a goal or certify high\-level application properties\. Our concern is orthogonal and finer\-grained: not whether the agent succeeds, but whether the evidence it cites actually supports what it writes, and crucially whether*that*measurement is itself reliable\. Where a certification scheme attests arbitrary application\-level properties, we place a distribution\-free, finite\-sample bound on one concrete, passage\-level quantity, the unsupported\-citation slip\-through rate, and treat the reliability of the metric as a first\-class object\. ### 2\.3\.LLM\-as\-judge reliability Model\-graded evaluation is now pervasive\(Zhenget al\.,[2023](https://arxiv.org/html/2607.20527#bib.bib28); Kimet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib30)\), and so are its known failure modes: self\-preference\(Panicksseryet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib29)\), sensitivity to spurious features such as length\(Duboiset al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib32)\), self\-inconsistency\(Haldar and Hockenmaier,[2025](https://arxiv.org/html/2607.20527#bib.bib31)\), and the general difficulty of automatic attribution evaluation\(Liet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib41)\)\. Proposed mitigations include specialized judges\(Kimet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib30)\), reward\-model benchmarks\(Lambertet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib33)\), judge benchmarks\(Tan and others,[2025](https://arxiv.org/html/2607.20527#bib.bib34)\), and juries of diverse models\(Vergaet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib35)\)\. We add a quantitative account of judge disagreement specifically for citation support on agentic outputs, and, rather than seek a better judge, we anchor the metric on gold and bound the residual error\. A jury of the disagreeing verifiers does not resolve this: averaging idiosyncratic thresholds still has no ground\-truth anchor, and a majority\-vote flag set inherits the same untrustworthy operating point we document; gold anchoring supplies the reference that consensus cannot\. ### 2\.4\.Citation and attribution evaluation Attribution evaluation spans the AIS framework\(Rashkinet al\.,[2023](https://arxiv.org/html/2607.20527#bib.bib39)\), ALCE\(Gaoet al\.,[2023](https://arxiv.org/html/2607.20527#bib.bib36)\), TRUE\(Honovichet al\.,[2022](https://arxiv.org/html/2607.20527#bib.bib38)\), AttrScore\(Yueet al\.,[2023](https://arxiv.org/html/2607.20527#bib.bib37)\), RAGAS\(Eset al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib40)\), CiteEval\(Xuet al\.,[2025](https://arxiv.org/html/2607.20527#bib.bib43)\), and medical citation auditing\(Wuet al\.,[2025](https://arxiv.org/html/2607.20527#bib.bib45)\)\. A separate line verifies whether a cited reference*exists*and whether its metadata is correct, catching fabricated references\(Shiet al\.,[2026](https://arxiv.org/html/2607.20527#bib.bib44); Lee,[2026](https://arxiv.org/html/2607.20527#bib.bib46); Chelliet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib47)\); our concern is the orthogonal one, whether a genuinely existing passage*supports*the claim attached to it\. AttributionBench shows this support judgment is hard even for strong models\(Liet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib41)\), which motivates treating any single verifier’s verdict as uncertain\. ### 2\.5\.Conformal and selective prediction Conformal prediction provides distribution\-free, finite\-sample guarantees\(Angelopoulos and Bates,[2021](https://arxiv.org/html/2607.20527#bib.bib52)\)and has been applied to language\-model factuality\(Mohri and Hashimoto,[2024](https://arxiv.org/html/2607.20527#bib.bib53)\), conformal language modeling\(Quachet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib54)\), and hallucination abstention\(Abbasi\-Yadkoriet al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib55)\), in the tradition of selective prediction\(El\-Yaniv and Wiener,[2010](https://arxiv.org/html/2607.20527#bib.bib56); Geifman and El\-Yaniv,[2017](https://arxiv.org/html/2607.20527#bib.bib57)\)\. The closest work bounds answer\-level claim correctness; we instead place a distribution\-free bound on the passage\-level unsupported\-*citation*slip\-through rate, a quantity specific to citation faithfulness\. ## 3\.Preliminaries and Problem Formulation ### 3\.1\.Notation and definitions An agentic synthesis system answers a query by retrieving a set of passagesP=\{p1,…,pm\}P=\\\{p\_\{1\},\\dots,p\_\{m\}\\\}and returning an answer composed of cited sentences, which we decompose into claim–citation pairs\(c,pi\)\(c,p\_\{i\}\)where claimccis a sentence andpi∈Pp\_\{i\}\\in Pis the passage it cites\. Table[1](https://arxiv.org/html/2607.20527#S3.T1)collects the notation\. ###### Definition 3\.1 \(Verifier and support\)\. A*verifier*is a functionv\(c,p\)∈\{0,1\}v\(c,p\)\\in\\\{0,1\\\}that judges whether passageppsupports claimcc, optionally exposing a continuous scores\(c,p\)∈\[0,1\]s\(c,p\)\\in\[0,1\]that estimates the probability of support\. A citation\(c,pi\)\(c,p\_\{i\}\)is*unsupported undervv*ifv\(c,pi\)=0v\(c,p\_\{i\}\)=0\. A*gold*labely\(c,p\)∈\{0,1\}y\(c,p\)\\in\\\{0,1\\\}is a human annotation of support, available only on benchmark data\. ###### Definition 3\.2 \(Unsupported\-citation rate\)\. The*unsupported\-citation rate*of a system under verifiervvis the fraction of its claim–citation pairs that are unsupported undervv\. As a function of both the outputs andvv, it is meaningful only together with the verifier and protocol that produced it\. Table 1\.Notation used throughout\. ### 3\.2\.Problem statement Let a deployed agent produce claim–citation pairs whose true support status is unknown\. A*gold*labely\(c,p\)∈\{0,1\}y\(c,p\)\\in\\\{0,1\\\}is a human annotation of support, available only on benchmark data\. Two problems follow\. \(1\)*Measurement\.*The reported unsupported rate is a function of both the outputs and the verifier; we ask how much it varies with the choice ofvvand how much verifiers agree on the per\-citation verdict\. \(2\)*Guarantee\.*Given an imperfect verifier, we seek a flagging rule that, for a chosen toleranceα\\alpha, catches at least a1−α1\-\\alphafraction of the truly unsupported citations, with a finite\-sample, distribution\-free guarantee calibrated on gold\. The first problem establishes that no single verifier verdict can be trusted; the second turns that imperfect verifier into a bounded triage\. ## 4\.Method ### 4\.1\.The verifier as a swappable measurement instrument Any verifier is an automated judge that emits a continuous scores\(c,p\)s\(c,p\)separating supported from unsupported claims, and different verifiers make different errors, so the verifier must be validated against gold before we trust anything it reports\. Because the conformal layer below is what controls the catch rate on unsupported citations, the verifier need not be the strictest judge, only a good separator\. We therefore compare gold\-validated verifiers at a*matched catch rate*and, finding their separation comparable, adopt the attribution\-tuned 3B model AttrScore\(Yueet al\.,[2023](https://arxiv.org/html/2607.20527#bib.bib37)\)on cost because it runs locally\. Its separation on gold is the signal the conformal layer turns into a guarantee \(Figure[2](https://arxiv.org/html/2607.20527#S5.F2)\)\. ### 4\.2\.Quantifying verifier disagreement To measure how much the metric depends on the instrument, we score identical agent outputs with a panel of gold\-validated verifiers spanning the frontier, from high\-recall AttrScore to a high\-specificity natural\-language\-inference \(NLI\) judge, with RAGAS\(Eset al\.,[2024](https://arxiv.org/html/2607.20527#bib.bib40)\)and a frontier judge in between\. We report \(i\) the unsupported rate under each verifier and \(ii\) class\-specific pairwise agreement on a fixed sample of deployment citations\. Because the supported class is highly prevalent, we read*negative\-specific*agreement \(agreement on what to flag\) rather than Cohen’sκ\\kappaalone, which the base rate depresses \(the kappa paradox; Supplementary S2\)\. ### 4\.3\.The conformal guard The guard processes each cited sentence in three steps \(Figure[1](https://arxiv.org/html/2607.20527#S1.F1)\)\. It*verifies*the claim against its cited passage with the 3B verifier; if unsupported, it attempts to*re\-attribute*the claim to another provided passage that supports it, keeping the claim and fixing only the pointer; what cannot be repaired it*flags*rather than deletes\. The verifier is imperfect, so a hard threshold would have unknown error on new data\. Split conformal removes that uncertainty\. On a held\-out gold calibration set we choose a toleranceα\\alphaand set the flag thresholdτ\\tauto the appropriate quantile of the unsupported\-class scores with a finite\-sample correction: \(1\)τ=s\(k\),k=⌈\(ncal\+1\)\(1−α\)⌉,\\tau=s\_\{\(k\)\},\\qquad k=\\big\\lceil\(n\_\{\\text\{cal\}\}\+1\)\(1\-\\alpha\)\\big\\rceil,wheres\(1\)≤⋯≤s\(ncal\)s\_\{\(1\)\}\\leq\\dots\\leq s\_\{\(n\_\{\\text\{cal\}\}\)\}are the sorted scores of the calibration unsupported citations\. Flagging every citation withs\(c,p\)≤τs\(c,p\)\\leq\\tauthen catches at least1−α1\-\\alphaof the truly unsupported citations, in finite samples and without any assumption on the score distribution; the only assumption is exchangeability between calibration and deployment\. ###### Proposition 4\.1 \(Finite\-sample catch rate\)\. Let the calibration unsupported\-class scoress1,…,sncals\_\{1\},\\dots,s\_\{n\_\{\\text\{cal\}\}\}and a fresh unsupported\-class scoresnews\_\{\\mathrm\{new\}\}be exchangeable, and setτ=s\(k\)\\tau=s\_\{\(k\)\}withk=⌈\(ncal\+1\)\(1−α\)⌉k=\\lceil\(n\_\{\\text\{cal\}\}\+1\)\(1\-\\alpha\)\\rceil\. Then the rule “flag ifs≤τs\\leq\\tau” catches the new unsupported citation with probability at least1−α1\-\\alpha:Pr\[snew≤τ\]≥1−α\\Pr\[\\,s\_\{\\mathrm\{new\}\}\\leq\\tau\\,\]\\geq 1\-\\alpha\. ###### Proof\. By exchangeability the rank ofsnews\_\{\\mathrm\{new\}\}among thencal\+1n\_\{\\text\{cal\}\}\+1scores is uniform on\{1,…,ncal\+1\}\\\{1,\\dots,n\_\{\\text\{cal\}\}\+1\\\}, soPr\[snew≤s\(k\)\]≥k/\(ncal\+1\)≥1−α\\Pr\[\\,s\_\{\\mathrm\{new\}\}\\leq s\_\{\(k\)\}\\,\]\\geq k/\(n\_\{\\text\{cal\}\}\+1\)\\geq 1\-\\alphaby the choice ofkk\. This is the standard split\-conformal argument\(Angelopoulos and Bates,[2021](https://arxiv.org/html/2607.20527#bib.bib52)\)applied to the unsupported\-class score distribution; it gives coverage≥1−α\\geq 1\-\\alphaunder exchangeability and is exact for continuous, tie\-free scores\. The discrete verifier score can tie; we break ties conservatively \(treatingsnew≤τs\_\{\\mathrm\{new\}\}\\leq\\tauat equality\), which preserves the≥1−α\\geq 1\-\\alphabound\. This is what earns the word “guarantee” even when gold is scarce\. ∎ The bound is on the catch rate, the safety axis; we verify below that it holds empirically on held\-out gold \(Figure[4](https://arxiv.org/html/2607.20527#S5.F4)A\)\. ### 4\.4\.Re\-attribution When a citation is flagged, the guard ranks the other provided passages, proposes the top\-ranked one as a replacement pointer, and re\-verifies that candidate before the pointer is moved\. The ranker is a swappable slot: we adopt a deterministic lexical BM25 by default and compare it against the verifier’s own attribution scores\(c,pj\)s\(c,p\_\{j\}\)and against the generators asked to re\-attribute their own citations\. We evaluate re\-attribution against gold: on data where a human\-labeled supporting passage exists, we measure how often the gold passage is ranked first or within the topkk, scored against truth rather than against the verifier that did the flagging\. Because the ranker only proposes and the verifier disposes, a lexical default is safe even though it cannot itself distinguish support from contradiction\. ## 5\.Experiments ### 5\.1\.Setup #### Agents and conditions\. We audit four open instruction\-tuned models in the 27–35B range, two dense and two mixture\-of\-experts from the Gemma\-4 and Qwen3\.6 families\(Google DeepMind,[2026](https://arxiv.org/html/2607.20527#bib.bib59); Alibaba Qwen Team,[2026](https://arxiv.org/html/2607.20527#bib.bib60)\), run locally on a single GPU, across two published agentic pipelines, OpenScholar and PaperQA2, against a bare no\-retrieval floor\. We hold the retrieved context fixed and reproduce each system’s published prompting, so the rates reflect generation\-time citation behavior rather than each system’s own retrieval\. Running every generator locally keeps the audit fully reproducible on commodity hardware\. The deployment citations we audit for the rate and disagreement analyses are these systems’ answers on the ScholarQABench multi\-paper tasks\(Asaiet al\.,[2026](https://arxiv.org/html/2607.20527#bib.bib1)\)\. #### Verifiers\. Five gold\-validated verifiers span the recall–specificity frontier: AttrScore\-3B \(high recall\), an OpenScholar post\-hoc judge, GPT\-4o, RAGAS, and a DeBERTa\-NLI judge \(high specificity\)\. #### Gold data\. Verifier validation uses SciFact \(n=208n=208human\-labeled claim–passage test pairs, with a disjointn=802n=802split used only for prompt selection\)\. Re\-attribution and conformal calibration use the full QASA test set \(n=1375n=1375\), a question\-answering benchmark over research papers\(Leeet al\.,[2023](https://arxiv.org/html/2607.20527#bib.bib68)\)with human\-labeled supporting passages\. A scope control uses PubMedQA \(n=843n=843\)\. #### Metrics\. We report per\-class recall and specificity \(not a single accuracy, which moves with class balance\), Cohen’sκ\\kappawith bootstrap 95% confidence intervals \(CIs\), unsupported\-citation rates, re\-attribution recall@kkwith binomial 95% CIs, and conformal catch rate against the declared target\. ### 5\.2\.The verifier, a swappable instrument chosen on cost Figure 2\.The verifier as a swappable instrument, validated on gold\. \(A\) Matched catch\-rate frontier on SciFact: at any common catch rate on unsupported citations the three verifiers separate comparably \(AUC and review cost overlap\), so a strict frontier judge buys no advantage and the cheap local AttrScore\-3B is adopted on cost\. \(B\) The continuous score separates supported from unsupported claims on gold \(means 0\.54 versus 0\.20\), the signal the conformal layer turns into a guarantee\.Because the conformal layer controls the catch rate, the verifier need only separate the two classes well, so we compare verifiers at a*matched catch rate*rather than at their native thresholds\. On SciFact gold the continuous\-scored verifiers separate comparably \(Figure[2](https://arxiv.org/html/2607.20527#S5.F2)A\): their ROC\-AUC values overlap \(AttrScore\-3B 0\.88 \[0\.84, 0\.93\], DeBERTa\-NLI 0\.90 \[0\.85, 0\.94\], GPT\-4o 0\.87 \[0\.82, 0\.92\]; bootstrap 95% CIs, SciFactn=208n=208\), and at any common catch rate their review cost is comparable, so a strict frontier judge such as GPT\-4o buys no separation advantage over the local 3B model\. Given comparable separation, the verifier is a swappable instrument and the deciding factor is cost: we adopt AttrScore\-3B because it runs locally on one GPU\. Its native operating point is high recall on the supported class \(0\.90 on SciFact, Table[2](https://arxiv.org/html/2607.20527#S5.T2); 0\.94 on a disjoint held\-out split,n=802n=802\), which suits a guard that must not discard genuine citations, while the per\-class trade\-offs of the stricter verifiers \(DeBERTa recall 0\.25/specificity 0\.97; GPT\-4o recall 0\.46\) explain the inflated rates they report in Figure[3](https://arxiv.org/html/2607.20527#S5.F3)\. Supplementary S1 gives the full per\-verifier metrics, the matched\-catch comparison, the prompt\-selection procedure, and the held\-out validation\. Table 2\.The verifier frontier on SciFact gold \(n=208n=208\): recall on the supported class \(the safety axis\) versus specificity\. AttrScore\-3B sits at the high\-recall end\. The adopted verifier is re\-validated on a disjoint held\-out split \(recall 0\.94,n=802n=802\); full metrics, Cohen’sκ\\kappawith CIs, and the prompt\-selection comparison are in Supplementary S1\. ### 5\.3\.The unsupported rate is verifier\-dependent Figure 3\.The unsupported\-citation rate has no single value: on the same agent outputs it ranges threefold to sixfold across five gold\-validated verifiers ordered by recall\. Most of the climb is the strict verifiers falsely rejecting genuinely supported citations\.On the same agent outputs the measured unsupported rate rises monotonically as the verifier gets stricter, from about 3% at the bare floor under the high\-recall verifier to roughly 14–18% under a strict NLI judge, with the agentic pipelines a few points above the floor at each setting \(Figure[3](https://arxiv.org/html/2607.20527#S5.F3)\)\. Much of that climb is not extra unfaithfulness but extra false rejection: the strict verifiers have low recall on genuinely supported claims \(0\.25 on gold\), so their high rates partly count supported citations\. The rates are pooled over the four open models and three pipelines and stay single\-digit at the lenient end while climbing with verifier strictness, so the effect is a property of the verifier, not of any one generator\. We therefore report no single rate without naming its operating point; two unsupported rates from different papers are not comparable unless they name the same verifier and protocol\. ### 5\.4\.Verifiers disagree on what is unsupported Disagreement runs deeper than the headline rate, and it is specifically about what to*flag*\. On 300 cited sentences from the deployed pipelines, three gold\-validated verifiers spanning the recall–specificity range \(AttrScore\-3B, DeBERTa\-NLI, and the GPT\-4o judge; the two binary\-only judges, RAGAS and OpenScholar post\-hoc, are omitted from this pairwise analysis\) agree strongly on which citations are supported \(positive\-specific agreement 0\.90–0\.96\) but agree on which to flag as unsupported only weakly \(negative\-specific agreement 0\.27–0\.30\); concretely, 60 to 80% of the citations one verifier flags, another calls supported\. \(Full agreement statistics, including observed agreement, Gwet’s AC1, and Cohen’sκ\\kappawith the base\-rate caveat and CIs, are in Supplementary S2\.\) The verifiers thus encode genuinely different flag thresholds: there is no single deployment flag set to trust\. Everything load\-bearing below, the verifier’s validation, the re\-attribution accuracy, and the conformal threshold, is therefore anchored on human gold rather than on a model’s idiosyncratic cutoff\. ### 5\.5\.The conformal guarantee holds in\-distribution Figure 4\.The conformal guarantee, calibrated on QASA gold \(n=1375n=1375\)\. \(A\) The guarantee holds: the achieved catch rate tracks the declared target1−α1\-\\alphaon repeated held\-out splits across the operating range \(target 0\.90/0\.94/0\.98→\\rightarrowachieved 0\.90/0\.94/0\.98\)\. \(B\) The recall\-cost dial, per open model: declaring a guaranteed catch rate fixes the review budget in advance\. The verify→\\rightarrowre\-attribute→\\rightarrowflag pipeline itself is shown in Figure[1](https://arxiv.org/html/2607.20527#S1.F1)\.Calibrated on QASA gold \(the full test set, 1375 supported and 1375 unsupported pairs\), the conformal flag rule’s achieved catch rate tracks the declared target across the operating range on repeated held\-out splits \(Figure[4](https://arxiv.org/html/2607.20527#S5.F4)A\), so the distribution\-free bound is empirical, not only asymptotic\. This coverage is*marginal*over the calibration draw: an individual split can fall below target, which is why we calibrate on as much gold as is available and recalibrate when the target setting differs\. The finite\-sample correction is what earns the word “guarantee” when gold is scarce: at a calibration size of 25 a naive empirical\-quantile threshold under\-covers \(0\.85 at a 0\.90 target\) while the conformal rule holds \(0\.92\); the gap closes by a calibration size of about 200 \(Supplementary S3\)\. At large calibration sizes the two coincide, so the value of conformal here is distribution\-free validity, not a numerical gain\. #### Calibration\-negative difficulty governs transfer\. The bound is distribution\-free only under exchangeability between calibration and deployment negatives, and that assumption is consequential\. Our calibration negatives are QASA in\-paper distractors, which are easy: the verifier scores them low \(meanP\(attributable\)=0\.20P\(\\text\{attributable\}\)=0\.20\)\. Genuinely mis\-cited claims are harder, because the cited passage is on topic but does not support the specific claim; on SciFact’s human\-labeled unsupported claims the verifier scores them far higher \(mean0\.500\.50\)\. A thresholdτ\\taucalibrated on the easy distractors to catch90%90\\%of them therefore catches only37%37\\%of the harder human\-labeled negatives \(44%44\\%and61%61\\%at the94%94\\%and98%98\\%targets; Supplementary S6\)\. The conformal machinery is not at fault: recalibratingτ\\tauon negatives drawn from the deployment distribution restores the guarantee, at a higher review budget\. The operational rule is therefore that the calibration set must resemble the deployment negatives; the QASA coverage above is an in\-distribution demonstration, and a deployment should recalibrate on target\-domain negatives rather than inherit the QASA threshold\. Identifying and quantifying this calibration\-negative\-difficulty condition is itself part of the contribution: prior conformal\-factuality work names distribution shift as a limitation but does not characterize which shift matters, whereas we isolate one such condition empirically and show how to restore the guarantee\. The guarantee pays for safety in review effort, the genuine citations it also flags \(Figure[4](https://arxiv.org/html/2607.20527#S5.F4)B\)\. At a 90% guarantee, pooled across the four open models, the flag rate at the high\-recall operating point is about 3% for OpenScholar and 6% for PaperQA2, against about 2\.5% for the bare model: a few cited sentences per hundred sent for review\. Table[3](https://arxiv.org/html/2607.20527#S5.T3)breaks the budget down by model: it stays single\-digit at the bare floor for every open model and rises with pipeline agency, with no single model driving the pooled figure\. Table 3\.Per\-model review budget: flag rate \(%\) at a 90% guaranteed catch rate, by generator and pipeline \(QASA\-calibratedτ\\tau\)\. The budget stays single\-digit at the bare floor for every open model and rises with pipeline agency; no single model drives the pooled rate\. Each cell isn=250n=250deployment citations; Wilson 95% CIs span a few points \(e\.g\. bare Gemma\-4\-26B0\.8%0\.8\\%\[0\.2, 2\.9\], PaperQA2 Qwen3\.6\-35B9\.5%9\.5\\%\[6\.5, 13\.8\]\)\.Even a single\-digit unsupported rate matters at synthesis scale, where one survey can carry hundreds of cited sentences; the value is not a large defect rate but a bounded, declared review budget over many citations\. ### 5\.6\.Re\-attribution is a swappable, commodity slot Figure 5\.Re\-attribution validated against gold \(QASA,n=1375n=1375, recall@1, 100% coverage\)\. Recovering the supporting passage is a swappable commodity slot: a deterministic lexical BM25 on CPU \(0\.69\) matches the verifier score \(0\.58\) and is reliable, while the open generators run locally on GPU vary from 0\.51 \(Qwen\) to 0\.76 \(Gemma\)\. All crush the 3% random baseline\. Wilson 95% CIs\.When a citation is flagged, recovering its true supporting passage among the provided candidates is an easy retrieval task, and it is not where the contribution lies\. On QASA gold \(n=1375n=1375; Table[4](https://arxiv.org/html/2607.20527#S5.T4), Figure[5](https://arxiv.org/html/2607.20527#S5.F5)\) every reasonable ranker crushes the 3% random baseline and they form a clear order: a deterministic lexical BM25 reaches recall@1 0\.69 on CPU, the verifier’s own attribution score reaches 0\.58, and the open generators asked to re\-attribute their own citations range from 0\.51 \(Qwen3\.6\) to 0\.74–0\.76 \(Gemma\-4\) on GPU\. The most accurate ranker is therefore a generator, not BM25 and not the verifier; but generator quality is model\-dependent, and the advantage over BM25 is concentrated where it is least needed\. A lexical\-overlap stratification \(Supplementary S4\) makes this concrete: on the half of items whose answer shares vocabulary with the gold passage, BM25 and Gemma are tied \(0\.88 each\); the generator’s edge appears only on the low\-overlap tail \(0\.31 versus 0\.53 at the lowest\-overlap quartile\), exactly the cases that need reading for entailment rather than matching words\. Two properties make BM25 the sensible default for this slot despite not being the most accurate\. It is deterministic and free, with no model to host, 100% coverage, and reproducible output; and re\-attribution is a retrieval step, not a support judgment, so a lexical ranker is appropriate\. Crucially, a lexical ranker cannot tell support from contradiction: on SciFact an answer and a refuting passage share almost identical vocabulary \(Jaccard 0\.064 versus 0\.055\), so BM25 would surface a contradicting passage as readily as a supporting one, whereas the verifier separates them \(meanP\(attributable\)P\(\\text\{attributable\}\)0\.83 versus 0\.50; Supplementary S4\)\. The division of labor is therefore deliberate: the re\-attributor proposes a candidate by lexical match, and the verifier then re\-checks that the proposed passage actually supports the claim before the pointer is moved\. Because the slot is swappable, an operator already running a strong generator can plug it in for the low\-overlap tail; either way re\-attribution is a commodity and the guard’s load\-bearing value is flagging with a guarantee, not repair\. On real deployed outputs the picture is soberer still: most genuinely flagged claims are unsupported by*any*provided passage, so there is no better source to move them to\. Table 4\.Re\-attribution recall@kkon QASA gold \(n=1375n=1375, 100% coverage\)\. The open generators \(run locally on GPU\) are most accurate but vary by model; a deterministic lexical BM25 on CPU is close and reliable; all crush chance\. Wilson CIs, the lexical\-overlap stratification, and the paired McNemar test \(BM25 vs\. verifier score\) are in Supplementary S4\. ### 5\.7\.Worked examples Two cases from the deployed outputs \(Gemma\-4 under PaperQA2 and OpenScholar\) illustrate the guard’s two non\-trivial actions\.*Re\-attribution\.*For the claim “these attacks are used to identify sensitive information leakage,” the cited passage only states that word embeddings are trained on potentially sensitive data, so the verifier finds it unsupported; BM25 then proposes another provided passage \(“we quantitatively investigate how machine learning models leak information…\\ldotsthe membership inference attack”\), the verifier confirms it supports the claim, and the guard moves the pointer to it while keeping the sentence unchanged\.*Flagging\.*For the claim “unsupervised fine\-tuning may require further supervised or RL\-based alignment for optimal performance,” the cited passage only compares fine\-tuning with retrieval\-augmented generation and never makes the alignment claim; no other provided passage supports it either, so the guard marks the citation\[N UNVERIFIED\]rather than deleting it, leaving the sentence for human review\. Both decisions are made by the external verifier on gold\-validated scores, not by the generator that produced the citation\. ### 5\.8\.Scope control A scope control sharpens what the verifier does\. Applied to PubMedQA, where the label is whether a study’s conclusion is yes or no, the verifier shows near\-zero agreement \(κ=0\.02\\kappa=0\.02,n=843n=843\)\. This is the intended result: the verifier checks whether a passage*supports*a claim, not whether a conclusion is*correct*\. The guard is a citation\-faithfulness auditor for agentic outputs, not a conclusion\-correctness checker\. ## 6\.Discussion The broader lesson is twofold: trust in machine\-generated citations should be measured and bounded, not assumed, and the measurement is itself a choice\. The same agent outputs look several times worse under a strict verifier than a lenient one, so a faithfulness number reported without its operating point is not a fact a reader can use\. For agentic intelligent systems this reframes evaluation: a benchmark leaderboard of citation faithfulness is meaningful only relative to a named, gold\-validated verifier and protocol, and a deployable agent needs not a better detector but a stated operating point with a guarantee\. We conjecture the same caution applies wherever a model or rubric stands in as judge of an agent’s output, common across agentic evaluation, though our evidence here is limited to citation support\. For a practitioner the guard changes the operating posture: rather than trust a verifier’s verdicts at face value, an operator declares the residual risk they will tolerate and receives a written promise, conditional on the calibration negatives resembling deployment and optimistic otherwise, on how many unsupported citations can survive review, with the review budget that promise costs\. The promise isolates a single checkable assumption, that calibration data resemble deployed outputs; when that is in doubt, recalibrating on a small labeled sample from the target setting restores it\. ## 7\.Limitations The unsupported\-citation rate is relative to a verifier’s operating point, so our numbers and any others in this literature are interpretable only together with the verifier and protocol that produced them\. We verify each claim against the full cited passage with a sliding\-window protocol so that support located anywhere counts; truncating the passage to a fixed prefix, a tempting shortcut, inflates the apparent unsupported rate severalfold and must be avoided\. The conformal bound is conditional on exchangeability between calibration and deployed outputs, and this condition is consequential, not cosmetic: our calibration negatives are easy in\-paper distractors \(verifier meanP\(attributable\)=0\.20P\(\\text\{attributable\}\)=0\.20\), whereas genuinely mis\-cited claims are harder \(mean0\.500\.50on SciFact’s human\-labeled unsupported claims\), so a threshold calibrated to catch90%90\\%of the distractors catches only37%37\\%of the harder human\-labeled negatives \(Supplementary S6\)\. The guaranteed catch rate must therefore be read as optimistic for real mis\-citations, and a deployment must recalibrateτ\\tauon target\-domain negatives, which restores the guarantee at a higher review budget; the QASA coverage we report is an in\-distribution demonstration of the method, not a transferable operating point\. Our audit isolates the generation step, holding retrieval fixed, so it does not capture errors each agent’s own retrieval would introduce\. Re\-attribution helps only where an alternative supporting passage is present and is not guaranteed out of distribution, so the guard must not be a sole gate in safety\-critical settings, where expert review of flagged sentences remains required\. ## 8\.Conclusion We treated the reliability of citation\-faithfulness evaluation for agentic scientific synthesis as a first\-class problem and showed that deployment verifier judgments disagree enough that no single reported number is trustworthy\. Anchoring measurement on human gold and wrapping an imperfect verifier in a distribution\-free conformal guarantee yields an evaluation protocol and a deployable guard that hold despite that disagreement, released as an open single\-GPU kit\. Two directions follow\. The first is to raise the guarantee a level, from whether a passage supports a claim to whether the full body of cited evidence adequately justifies it, accounting for contradiction and selective citation\. The second is to harden the calibration against adversarial and distribution\-shifted inputs, so the bound still holds when an agent is optimized against the verifier\(Manheim and Garrabrant,[2018](https://arxiv.org/html/2607.20527#bib.bib63)\)\. A local, auditable guard of this kind is, in our view, a prerequisite for trusting agentic systems that read and cite the scientific record\. ###### Acknowledgements\. The authors thank the maintainers of the open benchmarks and models used in this study\. ## References - Y\. Abbasi\-Yadkori, I\. Kuzborskij, D\. Stutz,et al\.\(2024\)Mitigating LLM hallucinations via conformal abstention\.External Links:2405\.01563Cited by:[§2\.5](https://arxiv.org/html/2607.20527#S2.SS5.p1.1)\. - Alibaba Qwen Team \(2026\)Qwen3\.6 model family\.Note:qwen\.ai/blogCited by:[§5\.1](https://arxiv.org/html/2607.20527#S5.SS1.SSS0.Px1.p1.1)\. - A\. N\. Angelopoulos and S\. Bates \(2021\)A gentle introduction to conformal prediction and distribution\-free uncertainty quantification\.Note:Found\. Trends Mach\. Learn\., 2023External Links:2107\.07511Cited by:[§2\.5](https://arxiv.org/html/2607.20527#S2.SS5.p1.1),[§4\.3](https://arxiv.org/html/2607.20527#S4.SS3.1.p1.8)\. - A\. Asai, J\. He, R\. Shao,et al\.\(2026\)Synthesizing scientific literature with retrieval\-augmented language models\.Nature\.Note:OpenScholar; preprint arXiv:2411\.14199External Links:[Document](https://dx.doi.org/10.1038/s41586-025-10072-4)Cited by:[§1](https://arxiv.org/html/2607.20527#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20527#S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.20527#S5.SS1.SSS0.Px1.p1.1)\. - A\. Asai, Z\. Wu, Y\. Wang, A\. Sil, and H\. Hajishirzi \(2024\)Self\-RAG: learning to retrieve, generate, and critique through self\-reflection\.InProceedings of ICLR,Note:arXiv:2310\.11511Cited by:[§2\.1](https://arxiv.org/html/2607.20527#S2.SS1.p1.1)\. - N\. Bena, M\. Anisetti, E\. Damiani, A\. Della Bruna, C\. Y\. Yeun, and C\. A\. Ardagna \(2026\)A certification scheme for large language models\-based applications\.ACM Transactions on Intelligent Systems and Technology\.External Links:[Document](https://dx.doi.org/10.1145/3819077)Cited by:[§2\.2](https://arxiv.org/html/2607.20527#S2.SS2.p1.1)\. - D\. A\. Boiko, R\. MacKnight, B\. Kline, and G\. Gomes \(2023\)Autonomous chemical research with large language models\.Nature624\.External Links:[Document](https://dx.doi.org/10.1038/s41586-023-06792-0)Cited by:[§2\.1](https://arxiv.org/html/2607.20527#S2.SS1.p1.1)\. - Y\. Chang, X\. Wang, J\. Wang, Y\. Wu, L\. Yang, K\. Zhu, H\. Chen, X\. Yi, C\. Wang, Y\. Wang, W\. Ye, Y\. Zhang, Y\. Chang, P\. S\. Yu, Q\. Yang, and X\. Xie \(2024\)A survey on evaluation of large language models\.ACM Transactions on Intelligent Systems and Technology15\(3\)\.External Links:[Document](https://dx.doi.org/10.1145/3641289)Cited by:[§2\.2](https://arxiv.org/html/2607.20527#S2.SS2.p1.1)\. - M\. Chelli, J\. Descamps, V\. Lavoué,et al\.\(2024\)Hallucination rates and reference accuracy of ChatGPT and Bard for systematic reviews\.Journal of Medical Internet Research26\.External Links:[Document](https://dx.doi.org/10.2196/53164)Cited by:[§2\.4](https://arxiv.org/html/2607.20527#S2.SS4.p1.1)\. - Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. Hashimoto \(2024\)Length\-controlled AlpacaEval: a simple way to debias automatic evaluators\.Note:COLM 2024External Links:2404\.04475Cited by:[§2\.3](https://arxiv.org/html/2607.20527#S2.SS3.p1.1)\. - R\. El\-Yaniv and Y\. Wiener \(2010\)On the foundations of noise\-free selective classification\.Journal of Machine Learning Research11\.Cited by:[§2\.5](https://arxiv.org/html/2607.20527#S2.SS5.p1.1)\. - S\. Es, J\. James, L\. Espinosa\-Anke, and S\. Schockaert \(2024\)RAGAs: automated evaluation of retrieval augmented generation\.InProceedings of EACL: System Demonstrations,External Links:[Document](https://dx.doi.org/10.18653/v1/2024.eacl-demo.16)Cited by:[§2\.4](https://arxiv.org/html/2607.20527#S2.SS4.p1.1),[§4\.2](https://arxiv.org/html/2607.20527#S4.SS2.p1.1)\. - S\. Gao, A\. Fang, Y\. Huang,et al\.\(2024\)Empowering biomedical discovery with AI agents\.Cell187\.External Links:[Document](https://dx.doi.org/10.1016/j.cell.2024.09.022)Cited by:[§1](https://arxiv.org/html/2607.20527#S1.p1.1)\. - T\. Gao, H\. Yen, J\. Yu, and D\. Chen \(2023\)Enabling large language models to generate text with citations\.InProceedings of EMNLP,External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.398)Cited by:[§1](https://arxiv.org/html/2607.20527#S1.p1.1),[§2\.4](https://arxiv.org/html/2607.20527#S2.SS4.p1.1)\. - Y\. Geifman and R\. El\-Yaniv \(2017\)Selective classification for deep neural networks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:1705\.08500Cited by:[§2\.5](https://arxiv.org/html/2607.20527#S2.SS5.p1.1)\. - A\. E\. Ghareeb, B\. Chang, L\. Mitchener,et al\.\(2026\)A multi\-agent system for automating scientific discovery\.Nature\.Note:Robin; preprint arXiv:2505\.13400 \(2025\)External Links:[Document](https://dx.doi.org/10.1038/s41586-026-10652-y)Cited by:[§1](https://arxiv.org/html/2607.20527#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20527#S2.SS1.p1.1)\. - Google DeepMind \(2026\)Gemma 4: frontier multimodal intelligence on\-device\.Note:deepmind\.google/models/gemma/gemma\-4Cited by:[§5\.1](https://arxiv.org/html/2607.20527#S5.SS1.SSS0.Px1.p1.1)\. - J\. Gottweis, W\. Weng, A\. Daryin,et al\.\(2026\)Accelerating scientific discovery with Co\-Scientist\.Nature\.Note:Preprint arXiv:2502\.18864 \(2025\)External Links:[Document](https://dx.doi.org/10.1038/s41586-026-10644-y)Cited by:[§1](https://arxiv.org/html/2607.20527#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20527#S2.SS1.p1.1)\. - S\. Guan, J\. Wang, J\. Bian, B\. Zhu, J\. Lou, and H\. Xiong \(2026\)Evaluating llm\-based agents for multi\-turn conversations: a survey\.ACM Transactions on Intelligent Systems and Technology17\(4\)\.External Links:[Document](https://dx.doi.org/10.1145/3793671)Cited by:[§2\.2](https://arxiv.org/html/2607.20527#S2.SS2.p1.1)\. - Z\. Guoet al\.\(2024\)StableToolBench: towards stable large\-scale benchmarking on tool learning\.ACL Findings\.Note:arXiv:2403\.07714Cited by:[§2\.2](https://arxiv.org/html/2607.20527#S2.SS2.p1.1)\. - R\. Haldar and J\. Hockenmaier \(2025\)Rating roulette: self\-inconsistency in LLM\-as\-a\-judge frameworks\.InFindings of the Association for Computational Linguistics: EMNLP 2025,External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1361)Cited by:[§1](https://arxiv.org/html/2607.20527#S1.p2.1),[§2\.3](https://arxiv.org/html/2607.20527#S2.SS3.p1.1)\. - O\. Honovich, R\. Aharoni, J\. Herzig,et al\.\(2022\)TRUE: re\-evaluating factual consistency evaluation\.InProceedings of NAACL,External Links:[Document](https://dx.doi.org/10.18653/v1/2022.naacl-main.287)Cited by:[§2\.4](https://arxiv.org/html/2607.20527#S2.SS4.p1.1)\. - C\. E\. Jimenez, J\. Yang, A\. Wettig,et al\.\(2024\)SWE\-Bench: can language models resolve real\-world GitHub issues?\.InProceedings of ICLR,Cited by:[§2\.2](https://arxiv.org/html/2607.20527#S2.SS2.p1.1)\. - S\. Kim, J\. Suk, S\. Longpre,et al\.\(2024\)Prometheus 2: an open source language model specialized in evaluating other language models\.InProceedings of EMNLP,External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.248)Cited by:[§2\.3](https://arxiv.org/html/2607.20527#S2.SS3.p1.1)\. - N\. Lambert, V\. Pyatkin, J\. Morrison,et al\.\(2024\)RewardBench: evaluating reward models for language modeling\.arXiv:2403\.13787\.Cited by:[§2\.3](https://arxiv.org/html/2607.20527#S2.SS3.p1.1)\. - J\. Lee \(2026\)Citecheck: an MCP server for automated bibliographic verification and repair in scholarly writing\.External Links:2603\.17339Cited by:[§2\.4](https://arxiv.org/html/2607.20527#S2.SS4.p1.1)\. - Y\. Lee, K\. Lee, S\. Park, D\. Hwang, J\. Kim, H\. Lee, and M\. Lee \(2023\)QASA: advanced question answering on scientific articles\.InInternational Conference on Machine Learning \(ICML\),PMLR, Vol\.202\.Cited by:[§5\.1](https://arxiv.org/html/2607.20527#S5.SS1.SSS0.Px3.p1.4)\. - Y\. Li, X\. Yue, Z\. Liao, and H\. Sun \(2024\)AttributionBench: how hard is automatic attribution evaluation?\.InFindings of the Association for Computational Linguistics: ACL 2024,External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.886)Cited by:[§1](https://arxiv.org/html/2607.20527#S1.p2.1),[§2\.3](https://arxiv.org/html/2607.20527#S2.SS3.p1.1),[§2\.4](https://arxiv.org/html/2607.20527#S2.SS4.p1.1)\. - X\. Liu, H\. Yu, H\. Zhang,et al\.\(2024\)AgentBench: evaluating LLMs as agents\.InProceedings of ICLR,Note:arXiv:2308\.03688Cited by:[§2\.2](https://arxiv.org/html/2607.20527#S2.SS2.p1.1)\. - A\. M\. Bran, S\. Cox, O\. Schilter,et al\.\(2024\)Augmenting large language models with chemistry tools\.Nature Machine Intelligence6\.External Links:[Document](https://dx.doi.org/10.1038/s42256-024-00832-8)Cited by:[§2\.1](https://arxiv.org/html/2607.20527#S2.SS1.p1.1)\. - C\. Ma, J\. Zhang,et al\.\(2024\)AgentBoard: an analytical evaluation board of multi\-turn LLM agents\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2\.2](https://arxiv.org/html/2607.20527#S2.SS2.p1.1)\. - D\. Manheim and S\. Garrabrant \(2018\)Categorizing variants of goodhart’s law\.External Links:1803\.04585Cited by:[§8](https://arxiv.org/html/2607.20527#S8.p1.1)\. - L\. Messeri and M\. J\. Crockett \(2024\)Artificial intelligence and illusions of understanding in scientific research\.Nature627\.External Links:[Document](https://dx.doi.org/10.1038/s41586-024-07146-0)Cited by:[§2\.1](https://arxiv.org/html/2607.20527#S2.SS1.p1.1)\. - G\. Mialon, C\. Fourrier, C\. Swift,et al\.\(2024\)GAIA: a benchmark for general AI assistants\.InProceedings of ICLR,Note:arXiv:2311\.12983Cited by:[§2\.2](https://arxiv.org/html/2607.20527#S2.SS2.p1.1)\. - C\. Mohri and T\. Hashimoto \(2024\)Language models with conformal factuality guarantees\.InProceedings of ICML,Note:arXiv:2402\.10978Cited by:[§2\.5](https://arxiv.org/html/2607.20527#S2.SS5.p1.1)\. - A\. Panickssery, S\. R\. Bowman, and S\. Feng \(2024\)LLM evaluators recognize and favor their own generations\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2404\.13076Cited by:[§2\.3](https://arxiv.org/html/2607.20527#S2.SS3.p1.1)\. - S\. G\. Patilet al\.\(2025\)The Berkeley function calling leaderboard \(BFCL\): from tool use to agentic evaluation\.InProceedings of ICML,Cited by:[§2\.2](https://arxiv.org/html/2607.20527#S2.SS2.p1.1)\. - V\. Quach, A\. Fisch, T\. Schuster,et al\.\(2024\)Conformal language modeling\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2306\.10193Cited by:[§2\.5](https://arxiv.org/html/2607.20527#S2.SS5.p1.1)\. - H\. Rashkin, V\. Nikolaev, M\. Lamm,et al\.\(2023\)Measuring attribution in natural language generation models\.Computational Linguistics49\(4\)\.External Links:[Document](https://dx.doi.org/10.1162/coli%5Fa%5F00486)Cited by:[§2\.4](https://arxiv.org/html/2607.20527#S2.SS4.p1.1)\. - K\. Shi, W\. Sun, Z\. Zhang, L\. Sun, N\. V\. Chawla, and Y\. Ye \(2026\)CiteAudit: you cited it, but did you read it? a benchmark for verifying scientific references in the LLM era\.External Links:2602\.23452Cited by:[§2\.4](https://arxiv.org/html/2607.20527#S2.SS4.p1.1)\. - N\. Shinn, F\. Cassano, E\. Berman,et al\.\(2023\)Reflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2\.1](https://arxiv.org/html/2607.20527#S2.SS1.p1.1)\. - M\. D\. Skarlinski, S\. Cox, J\. M\. Laurent,et al\.\(2024\)Language agents achieve superhuman synthesis of scientific knowledge\.Note:PaperQA2External Links:2409\.13740Cited by:[§1](https://arxiv.org/html/2607.20527#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20527#S2.SS1.p1.1)\. - K\. Swanson, W\. Wu, N\. L\. Bulaong,et al\.\(2025\)The virtual lab of AI agents designs new SARS\-CoV\-2 nanobodies\.Nature\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09442-9)Cited by:[§2\.1](https://arxiv.org/html/2607.20527#S2.SS1.p1.1)\. - S\. Tanet al\.\(2025\)JudgeBench: a benchmark for evaluating LLM\-based judges\.InProceedings of ICLR,Note:arXiv:2410\.12784Cited by:[§2\.3](https://arxiv.org/html/2607.20527#S2.SS3.p1.1)\. - P\. Verga, S\. Hofstatter, S\. Althammer,et al\.\(2024\)Replacing judges with juries: evaluating LLM generations with a panel of diverse models\.arXiv:2404\.18796\.Cited by:[§2\.3](https://arxiv.org/html/2607.20527#S2.SS3.p1.1)\. - J\. Wallat, M\. Heuss, M\. de Rijke, and A\. Anand \(2024\)Correctness is not faithfulness in RAG attributions\.External Links:2412\.18004Cited by:[§1](https://arxiv.org/html/2607.20527#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20527#S2.SS1.p1.1)\. - H\. Wang, T\. Fu, Y\. Du,et al\.\(2023\)Scientific discovery in the age of artificial intelligence\.Nature620\.External Links:[Document](https://dx.doi.org/10.1038/s41586-023-06221-2)Cited by:[§1](https://arxiv.org/html/2607.20527#S1.p1.1)\. - K\. Wu, E\. Wu, K\. Wei,et al\.\(2025\)An automated framework for assessing how well LLMs cite relevant medical references\.Nature Communications16\.Note:SourceCheckupExternal Links:[Document](https://dx.doi.org/10.1038/s41467-025-58551-6)Cited by:[§1](https://arxiv.org/html/2607.20527#S1.p1.1),[§2\.4](https://arxiv.org/html/2607.20527#S2.SS4.p1.1)\. - Y\. Xu, P\. Qi, J\. Chen,et al\.\(2025\)CiteEval: principle\-driven citation evaluation for source attribution\.InProceedings of the 63rd Annual Meeting of the ACL,External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1574)Cited by:[§2\.4](https://arxiv.org/html/2607.20527#S2.SS4.p1.1)\. - S\. Yan, J\. Gu, Y\. Zhu, and Z\. Ling \(2024\)Corrective retrieval augmented generation\.InarXiv:2401\.15884,Cited by:[§2\.1](https://arxiv.org/html/2607.20527#S2.SS1.p1.1)\. - S\. Yaoet al\.\(2024\)τ\\tau\-Bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv:2406\.12045\.Cited by:[§2\.2](https://arxiv.org/html/2607.20527#S2.SS2.p1.1)\. - X\. Yue, B\. Wang, Z\. Chen,et al\.\(2023\)Automatic evaluation of attribution by large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2023,Note:AttrScoreExternal Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.307)Cited by:[§2\.4](https://arxiv.org/html/2607.20527#S2.SS4.p1.1),[§4\.1](https://arxiv.org/html/2607.20527#S4.SS1.p1.1)\. - W\. Zhang, G\. Dai, D\. Luo, Y\. Wang, and C\. Ye \(2026\)From hallucination to certainty: meta\-knowledge guided self\-correcting large language models\.ACM Transactions on Intelligent Systems and Technology\.External Links:[Document](https://dx.doi.org/10.1145/3797906)Cited by:[§2\.1](https://arxiv.org/html/2607.20527#S2.SS1.p1.1)\. - L\. Zheng, W\. Chiang, Y\. Sheng,et al\.\(2023\)Judging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.InNeurIPS Datasets and Benchmarks Track,Note:arXiv:2306\.05685Cited by:[§2\.3](https://arxiv.org/html/2607.20527#S2.SS3.p1.1)\. ## Supplementary Material This supplement holds the detailed statistics and the reproducibility specification kept out of the main text\. Section numbers \(S1–S6\) are referenced from the main paper\. All numbers are produced by the scripts in the released kit from the result files underresults/\. ## Appendix S1Verifier validation, prompt selection, and held\-out re\-validation Table[S1](https://arxiv.org/html/2607.20527#A1.T1)gives the full per\-verifier metrics on the SciFact gold set \(n=208n=208\) used to draw the recall–specificity frontier \(main Figure 1, Table 1\)\. Cohen’sκ\\kappais reported with a percentile\-bootstrap 95% CI \(2000 resamples\)\. Table S1\.Per\-verifier gold metrics on SciFact \(n=208n=208\)\. This is the development set on which the AttrScore prompt was selected; absolute AttrScore numbers here are therefore optimistic \(see Table[S2](https://arxiv.org/html/2607.20527#A1.T2)\)\.#### Prompt selection and held\-out re\-validation\. The AttrScore verifier prompt was chosen on then=208n=208development set for supported\-class recall\. To avoid reporting a selection\-biased number, we re\-validate the adopted prompt \(“OURS”\) and a canonical alternative \(“CANON”\) on a disjoint held\-out gold split \(n=802n=802\)\. Table[S2](https://arxiv.org/html/2607.20527#A1.T2)shows that OURS keeps the highest recall on held\-out \(0\.942\), the safety axis, while CANON attains a higherκ\\kappa\(0\.567\) at lower recall \(0\.869\)\. We adopt the high\-recall operating point deliberately: for a citation guard, discarding a genuine citation is costlier than an extra review\. Main\-text numbers for the adopted verifier use the held\-out values\. Table S2\.AttrScore prompt selection \(dev,n=208n=208\) versus held\-out re\-validation \(n=802n=802\)\. The dev ranking byκ\\kappadoes not carry to held\-out; we select on recall \(the safety axis\), notκ\\kappa\. ## Appendix S2Verifier agreement, full statistics On 300 deployment citations, the supported class is highly prevalent \(AttrScore 0\.957, DeBERTa 0\.823, GPT\-4o 0\.933\), so Cohen’sκ\\kappais depressed by the base\-rate paradox even when observed agreement is high\. We therefore report observed agreement, Gwet’s AC1, and class\-specific \(positive and negative\) agreement alongsideκ\\kappa\(Table[S3](https://arxiv.org/html/2607.20527#A2.T3)\)\. All three verifiers see the full cited passage: AttrScore and DeBERTa via 600/300 window tiling, GPT\-4o via its long context \(no truncation; this fixes a prior one\-sided head\-slice for GPT\-4o\)\. The diagnostic that matters is*negative\-specific*agreement, the agreement on which citations to flag: it is only 0\.27–0\.30, whereas positive\-specific agreement \(on what is supported\) is 0\.90–0\.96 and AC1 is 0\.78–0\.91\. Table S3\.Pairwise verifier agreement on 300 deployment citations \(symmetric full\-passage protocol\)\.κ\\kappaand negative\-specific agreement with percentile\-bootstrap 95% CIs \(2000 resamples\)\. ## Appendix S3Conformal calibration\-size ablation The main text reports marginal coverage over repeated held\-out splits\. Table[S4](https://arxiv.org/html/2607.20527#A3.T4)shows why the finite\-sample correction matters when gold is scarce: at a calibration size of 25, a naive empirical\-quantile threshold under\-covers the nominal target, while the conformal \(finite\-sample corrected\) rule maintains it\. The gap closes by a calibration size of about 200\. Estimated overB=1000B=1000random calibration/test splits \(testn=200n=200, seed 0\); unsupported\-class scores from QASA gold\. Table S4\.Achieved catch rate \(corrected conformal vs naive empirical\-quantile threshold\) by calibration size, at two targets\. Naive under\-covers at small calibration sizes; the two coincide as size grows\.This coverage is marginal over the calibration draw; on an individual split it can fall below target, the standard conformal caveat\. We recommend recalibrating on a small labeled sample from the target setting when the deployment distribution differs from the calibration distribution\. ## Appendix S4Re\-attribution is a swappable, commodity slot All rankers are evaluated on the full QASA test set \(n=1375n=1375, 100% coverage\), with the same gold definition, recall@kk, and query, differing only in the ranker\. Table[S5](https://arxiv.org/html/2607.20527#A4.T5)reports recall@kkwith Wilson 95% CIs\. Every ranker far exceeds the 3% random baseline\. The open generators, run locally on GPU and asked to re\-attribute their own citations, are the most accurate but vary by model \(0\.51 to 0\.76\); a deterministic lexical BM25 on CPU is close and reliable \(0\.69\); the verifier’s own attribution score is a weaker ranker \(0\.58\)\. Table S5\.Re\-attribution recall@kkon QASA gold \(n=1375n=1375, 100% coverage\); recall@1 with Wilson 95% CIs\. Generators run locally on GPU; BM25 runs on CPU\. All far exceed the 3% random baseline\. recall@1 carries Wilson 95% CIs; recall@3 and recall@5 CIs are within±0\.02\\pm 0\.02–0\.030\.03of the point estimates \(n=1375n=1375\)\.#### Where the generator’s advantage lies \(lexical\-overlap stratification\)\. BM25 ranks by word overlap, so its accuracy should track the lexical similarity between the answer and the gold passage\. We stratify the 1375 items by the maximum token\-Jaccard overlap between the answer and its gold passage and compare BM25 with the best generator, Gemma\-4\-26B \(Table[S6](https://arxiv.org/html/2607.20527#A4.T6)\)\. On the high\-overlap half the two are tied \(0\.886 each\); the generator’s advantage appears only on the low\-overlap half \(0\.64 versus 0\.49\) and is largest on the lowest\-overlap quartile \(0\.53 versus 0\.30\), exactly the items where support must be read for entailment rather than matched by words\. The generator’s edge over BM25 is statistically significant overall \(McNemar exact two\-sidedp=1\.4×10−9p=1\.4\\times 10^\{\-9\}; 177 Gemma\-only versus 80 BM25\-only discordant hits atk=1k=1\), but it is concentrated on the entailment tail, so a free CPU ranker is a defensible default and a generator is a drop\-in upgrade where that tail matters\. Table S6\.Lexical\-overlap stratification: BM25 versus the best generator \(Gemma\-4\-26B\), recall@1 on QASA gold \(n=1375n=1375\), by answer–gold token\-Jaccard overlap\. They tie when words overlap; the generator’s advantage is confined to the low\-overlap \(entailment\) tail\. #### A lexical ranker cannot judge support versus contradiction\. Re\-attribution and verification are separate components because lexical overlap cannot distinguish a supporting passage from a contradicting one: both share the claim’s vocabulary\. On SciFact, supported and refuted claim–evidence pairs have almost identical answer–evidence Jaccard overlap \(0\.064 versus 0\.055\), so a BM25 ranker would surface a contradicting passage as readily as a supporting one\. The attribution verifier separates them cleanly \(meanP\(attributable\)P\(\\text\{attributable\}\)0\.83 on supported versus 0\.50 on refuted\)\. The guard therefore uses BM25 only to*propose*a candidate and the verifier to*judge*whether the proposed passage actually supports the claim before any pointer is moved\. ## Appendix S5Reproducibility specification #### Generators\. Four open instruction\-tuned models, all run locally with vLLM on a single GPU at temperature 0 \(greedy\):google/gemma\-4\-31b\-it\(dense\),google/gemma\-4\-26b\-a4b\-it\(MoE\),Qwen/Qwen3\.6\-27B\-FP8\(dense, Mamba\-hybrid\),Qwen/Qwen3\.6\-35B\-A3B\-FP8\(MoE\)\. The exact HuggingFace snapshot revisions are pinned and recorded in the kit \(e\.g\. Qwen3\.6\-35B95a723d0, Qwen3\.6\-27Be89b16eb, Gemma\-4\-31B35487898\)\. Full\-passage prompts up to the model context window, no truncation\. Conditions: bare \(no retrieval\), OpenScholar, PaperQA2; retrieved context held fixed, published prompts reproduced\. #### Verifiers\. AttrScore\-3B \(osunlp/attrscore\-flan\-t5\-xl\), DeBERTa\-NLI \(MoritzLaurer/DeBERTa\-v3\-large\-mnli\-fever\-anli\-ling\-wanli\), GPT\-4o \(openai/gpt\-4ovia OpenRouter, temperature 0; a non\-deterministic API baseline whose raw per\-citation verdicts are frozen in the kit for audit\), RAGAS faithfulness, OpenScholar post\-hoc\. Support is judged over the full passage via 600\-char windows at stride 300 \(supported if any window is attributable\); claims capped at 600 characters symmetrically\. #### Gold data\. SciFact \(verifier validation,n=208n=208test /n=802n=802held\-out\), QASA \(re\-attribution and conformal calibration, full test setn=1375n=1375, human\-labeled supporting passages\), PubMedQA \(scope control,n=843n=843\), and ScholarQABench multi\-paper tasks \(the deployment outputs: the four generators’ answers under each pipeline\)\. The per\-cell deployment\-citation samples \(250 per model×\\timespipeline for the conformal dial, 300 for the agreement analysis\) are drawn in file order and frozen in the kit\. Dataset licenses and redistribution status of the derived score files are listed in the kit’sDATA\_LICENSESfile\. #### Calibration\. Split\-conformal with finite\-sample quantilek=⌈\(ncal\+1\)\(1−α\)⌉k=\\lceil\(n\_\{\\text\{cal\}\}\+1\)\(1\-\\alpha\)\\rceil; calibration/test splits and all random seeds are fixed \(seed 0\) and recorded\. #### Compute\. The verifier and guard run on one 80GB GPU or on CPU; the full kit reproduces in about two hours\. Code, configs, prompts, seeds, and the figure\-generation script are released under the MIT license at[https://github\.com/GooTec/citation\-guard](https://github.com/GooTec/citation-guard), tagged releasev0\.2\.0\(commitb907985, the hash\-anchored citable snapshot for this paper\); a Zenodo DOI will be minted from this tag for the camera\-ready\. #### What reproduces\. The shippedreproduce/kit is a cross\-domain demonstration on a public BioASQ slice \(n=200n=200\), not the headline\. The main\-text artifacts are regenerated by the experiment scripts from the frozen score files underresults/: the verifier frontier and matched\-catch AUC \(SciFact\), the per\-model unsupported rate \(deployment outputs×\\timesfive verifiers\), the conformal coverage and recall–cost dial, the coverage\-transfer analysis \(S6\), and re\-attribution recall@kk\(QASAn=1375n=1375\)\. Each headline number traces to a named result file; the QASA calibration entry point is included\. ## Appendix S6Calibration\-negative difficulty and coverage transfer The conformal bound is distribution\-free only under exchangeability between the calibration and deployment negative distributions\. We quantify how much this matters by scoring two negative sets with the same adopted verifier \(AttrScore\-3BP\(attributable\)P\(\\text\{attributable\}\), window\-max\): the QASA in\-paper distractors used for calibration, and SciFact’s human\-labeled*unsupported*claim–evidence pairs, which are harder because the evidence is on topic but does not support the claim\. The distractors score low \(mean0\.200\.20, median0\.170\.17\); the human\-labeled negatives score far higher \(mean0\.500\.50, median0\.530\.53\), a0\.300\.30gap in mean verifier score\. Table[S7](https://arxiv.org/html/2607.20527#A6.T7)shows the resulting coverage transfer: a thresholdτ\\taucalibrated on the distractors for a target catch rate, applied to the harder negatives, under\-covers substantially \(e\.g\.0\.90→0\.370\.90\\rightarrow 0\.37\), because many harder negatives score above aτ\\tautuned on easy ones\. This is the standard exchangeability caveat made quantitative\. The split\-conformal method remains valid; what it requires is a calibration set drawn from the deployment negative distribution\. Recalibratingτ\\tauon representative \(harder\) negatives restores the target catch rate at a higher flag rate\. We are not aware of prior conformal\-factuality work that quantifies this calibration\-negative\-difficulty condition, which we regard as part of the contribution\. Table S7\.Coverage transfer from easy calibration negatives \(QASA in\-paper distractors,n=1375n=1375\) to harder human\-labeled negatives \(SciFact unsupported,n=71n=71\), same verifier and score\.τ\\tauis calibrated on the distractors for each target catch rate; the achieved catch on the human\-labeled negatives is far below target\.The Wilson 95% CIs \(SciFact unsupportedn=71n=71\) confirm the under\-coverage is not a small\-sample artifact\. The SciFact refuted pairs are a deliberately hard \(near\-upper\-bound\) proxy, in which the evidence is topically central but contradicts the claim; real deployment negatives range between these and the easy distractors, so the achieved catch in practice lies between the two columns\. Restoring a90%90\\%catch on the hard negatives requiresτ=0\.83\\tau=0\.83and a≈46%\\approx 46\\%flag rate, i\.e\. the verifier’s separation, not the conformal layer, becomes the binding constraint in the hardest regime, where stronger verifiers or human review are warranted\. #### Recalibration recipe\. To deploy with a meaningful guarantee: \(1\) label≳200\\gtrsim 200flagged citations from the target system as supported or unsupported \(the S3 ablation shows the finite\-sample correction has converged by≈200\\approx 200\); \(2\) run the kit’s calibration entry point on these target negatives to obtainτ\\tauand its Wilson flag\-rate CI; \(3\) deploy at thatτ\\tauand read off the review budget\. The QASA\-calibrated threshold should not be inherited\.
Similar Articles
Benchmarking Agentic Review Systems
This paper benchmarks agentic review systems for peer review, evaluating open-source and proprietary systems on research papers. The best configuration achieves 83.0% pairwise accuracy and catches 71.6% of injected errors, but user feedback highlights issues with false positives and nitpicks.
Faithful by Construction: Claim-Anchored Attribution for Multi-Document Summarization
This paper introduces CAMS, a modular multi-document summarization framework that extracts atomic claims with token-level provenance, clusters equivalent claims, and rewrites them into summaries with fine-grained, multi-source traceability, significantly improving faithfulness and citation precision.
Explicit Evidence Grounding via Structured Inline Citation Generation
This paper introduces FullCite, a framework for generating structured inline citations that link each claim to both its source document and specific evidence spans. Evaluated on three QA benchmarks (ASQA, BioASQ, ExpertQA), it finds that while LLMs are good at document-level attribution, they struggle with precise evidence span identification.
Trust, but Don't Verify: Epistemic Blind Spots in LLM Source Evaluation
This paper identifies a failure mode in LLMs where they do not verify the validity of numerical statistics when synthesizing multiple sources, instead relying on the stylistic markers of analytical rigor. The authors term this 'epistemic alignment' and show that it persists across models and domains, resisting prompting-based mitigations.
Beyond Component Testing: Validating Agentic AI Systems
This survey synthesizes 257 papers on validating agentic AI systems, proposing a five-dimension taxonomy covering behavioral, safety, temporal, regulatory, and multi-agent concerns. It identifies gaps in temporal validity, runtime evidence maintenance, regulatory legibility, and open-ended multi-agent assurance, arguing that trustworthy deployment requires validating trajectories in context.