What Changes When Fact-Verification Scores Improve? Evidence and Answer Accounting Across Trained Verifiers and LLMs

# What Changes When Fact-Verification Scores Improve?Evidence and Answer Accounting Across Trained Verifiers and LLMs
Source: [https://arxiv.org/html/2609.27064](https://arxiv.org/html/2609.27064)
## What Changes When Fact\-Verification Scores Improve? Evidence and Answer Accounting Across Trained Verifiers and LLMsThanks:The authors originated the main research question and the two study designs\. ChatGPT, Claude, and Codex assisted with implementation, execution, automated checks, interpretation, and manuscript drafting and revision, and suggested some post\-hoc analyses\.

Yingrui Li22footnotemark:2

###### Abstract

A joint fact\-verification score assesses answers and submitted evidence together\. When the score improves, how much of the gain remains if the answers are held fixed? OnFEVEROUS, strict score is the percentage of claims with a correct answer and a complete annotated evidence group in the submitted evidence\. Across four trained DeBERTa checkpoints and 7,890 claims, replacing DCUF evidence with UnifEE evidence raises strict score by 9\.61 percentage points, compared with 1\.96 percentage points in answer accuracy\. The paired 95% interval for the strict\-score gain is \[8\.77, 10\.43\], conditional on these checkpoints\. Replacing only the evidence passed to the scorer accounts for 7\.92 or 9\.08 percentage points when we retain the answers generated from DCUF or UnifEE evidence, respectively\. To examine how this evidence gain depends on evaluation choices, we generate 470,400 responses from two 8B LLMs on FEVER, FEVEROUS and SciFact under two answer formats and two context budgets\. Increasing context from 256 to 2,048 tokens raises the fixed\-answer evidence gain on FEVEROUS by 3\.84 and 3\.10 percentage points for Qwen and Llama, respectively\. The effects fall short of the prespecified cross\-dataset criterion, while some intervals extend beyond the two\-point small\-effect bound\. Post\-hoc analyses quantify changes in answers and submitted evidence, and show when aggregate accuracy and evidence\-coverage rates miss the claim\-level pattern\. The four answer–evidence score combinations reveal changes that endpoint and aggregate metrics leave unresolved\.

## 1Introduction

A fact\-verification system returns an answer and supporting evidence\. Joint benchmarks award credit when both satisfy the scoring requirements\([Thorne et al\., 2018](https://arxiv.org/html/2609.27064#bib.bib19);[Aly et al\., 2021](https://arxiv.org/html/2609.27064#bib.bib1)\)\. A higher score can reflect better answers, better evidence, or a change in which claims have both\. The total gain leaves these contributions unresolved\.

For example,[Hu et al\. \(2023, Table 2 and Section 3\.4\)](https://arxiv.org/html/2609.27064#bib.bib10)report UnifEE development\-set gains over DCUF of 9\.09 percentage points in FEVEROUS score and 0\.76 in answer accuracy, alongside evidence\-retrieval metrics\. We examine how much joint gain survives at fixed answers by recombining saved answers and evidence submissions\.

In Study 1, switching from DCUF to UnifEE evidence with each trained checkpoint held fixed raises mean strict score by 9\.61 percentage points and answer accuracy by 1\.96\. Keeping the answers generated from DCUF evidence fixed while replacing only the scorer’s evidence retains 7\.92 percentage points; updating the answers adds 1\.69\. Reversing the steps assigns 9\.08 percentage points to evidence and 0\.53 to answers\.

The next question is whether this evidence gain depends on how much evidence a model sees or how it returns an answer\. Study 2 tests this with fixed\-weight LLMs, two formats and two context budgets, using BM25 and BGE evidence orderings on three benchmarks\. FEVEROUS shows context dependence; the other datasets fall short of the prespecified replication criterion\.

#### Contributions\.

Our empirical contribution is to quantify the evidence component of a source gain, test its sensitivity to LLM interfaces, and explain when aggregate rates recover it\. The2×22\\times 2comparison reveals a concrete contrast: marginal accuracy and evidence coverage closely approximate one trained\-verifier allocation but miss most of the FEVER/Qwen context effect\. The missing information is which claims combine correct answers with eligible evidence\.

A separate retrospective example illustrates the related trade\-off between class metrics: a Not Enough Information \(NEI\) override improves macro\-F1 but lowers answer accuracy and strict score\. We retain its reconstruction and unresolved threshold history in Appendix[A](https://arxiv.org/html/2609.27064#A1)\.

The2×22\\times 2comparison requires saving answers and submitted evidence under both source conditions\. Two additional scoring passes provide the off\-diagonal scores and complete the comparison\.

## 2Related Work

#### Answer and evidence evaluation\.

FEVER combines claim labels with Wikipedia evidence\([Thorne et al\., 2018](https://arxiv.org/html/2609.27064#bib.bib19)\); FEVEROUS adds structured sources and requires evidence for NEI claims\([Aly et al\., 2021](https://arxiv.org/html/2609.27064#bib.bib1)\)\. SciFact evaluates scientific claims through abstract labels and rationale sentences\([Wadden et al\., 2020](https://arxiv.org/html/2609.27064#bib.bib20)\)\. DCUF develops text–table fusion, while UnifEE improves evidence extraction and reports both answer accuracy and joint scores\([Hu et al\., 2022](https://arxiv.org/html/2609.27064#bib.bib9);[Hu et al\., 2023](https://arxiv.org/html/2609.27064#bib.bib10)\)\. We use their evidence outputs with our own trained checkpoints\. Our question concerns the score changes under this intervention, rather than reproducing their original end\-to\-end systems\. In Study 2, BM25\([Robertson & Zaragoza, 2009](https://arxiv.org/html/2609.27064#bib.bib15)\)and a BGE reranker\([Xiao et al\., 2023](https://arxiv.org/html/2609.27064#bib.bib22)\)reorder the same candidate pools\.

#### Interfaces and evidence use\.

Prompt formatting and option identifiers can change model outputs\([Sclar et al\., 2024](https://arxiv.org/html/2609.27064#bib.bib17);[Zheng et al\., 2024](https://arxiv.org/html/2609.27064#bib.bib23)\); strict answer matching can combine semantic errors with output\-format failures\([Hua et al\., 2025](https://arxiv.org/html/2609.27064#bib.bib11)\)\. Long\-context performance also depends on information placement\([Liu et al\., 2024](https://arxiv.org/html/2609.27064#bib.bib12)\), including in retrieval\-augmented fact checking\([Bernardelle et al\., 2026](https://arxiv.org/html/2609.27064#bib.bib2)\)\. We study how these evaluation choices affect the gain from replacing scored evidence at fixed answers\. Attribution and citation work similarly distinguishes response correctness from evidential support\([Buchmann et al\., 2024](https://arxiv.org/html/2609.27064#bib.bib3);[Gao et al\., 2023](https://arxiv.org/html/2609.27064#bib.bib6)\)\. Claim\-only artifacts in FEVER show why answer correctness alone is insufficient to establish evidence use\([Schuster et al\., 2019](https://arxiv.org/html/2609.27064#bib.bib16)\)\. Our LLM experiments use Qwen3 and Llama 3 models\([Qwen Team, 2025](https://arxiv.org/html/2609.27064#bib.bib14);[Grattafiori et al\., 2024](https://arxiv.org/html/2609.27064#bib.bib7)\)\.

#### Decomposition and uncertainty\.

Order\-dependent decompositions and averaging over orders are established methods\([Shorrocks, 2013](https://arxiv.org/html/2609.27064#bib.bib18)\)\. Averaging the two evidence components here gives the two\-player Shapley allocation\. We contribute empirical comparisons of these components, not a new attribution identity\. Paired inference follows each metric’s dependence structure\([Dror et al\., 2018](https://arxiv.org/html/2609.27064#bib.bib5)\)\. Study 2 resamples claim families and decoding seeds, resembling crossed\-factor resampling\([Owen, 2007](https://arxiv.org/html/2609.27064#bib.bib13)\); it estimates uncertainty for fixed models\. We distinguish an interval that includes zero from evidence that an effect is small\([Card et al\., 2020](https://arxiv.org/html/2609.27064#bib.bib4)\)\.

## 3Decomposing Answer and Evidence Changes

We call the predicted class the*answer*\. An*official score*uses the benchmark’s own scoring implementation and submission limits\.*Evidence eligibility*means that a submitted evidence set satisfies its evidence requirement for the gold answer\. This concerns annotated evidence identifiers; semantic relevance and the evidence visible to the model are separate properties\.

We score each set of saved answers with evidence from both sources\. LetSa​bS\_\{ab\}denote the resulting corpus score on a 0–100 scale:aaidentifies the source used to generate answers, andbbidentifies the submitted evidence\. Changes are in percentage points for strict score and answer accuracy, and score points for F1; we use “points” as shorthand for both below\. We build a separate table for each model, input condition and replicate\. Source 0 is DCUF in Study 1 and BM25 in Study 2; source 1 is UnifEE and BGE, respectively\. We generate answers separately under both sources\. The diagonal entries are the observed endpoint scores, while the off\-diagonal entries reuse those answers with the other evidence submission\.

Table 1:The2×22\\times 2score table\. Each row holds answers fixed; each column holds submitted evidence fixed\.The endpoint gainGGhas two exact decompositions:

G\\displaystyle G=S11−S00=\(S01−S00\)⏟E0\+\(S11−S01\)⏟A1,\\displaystyle=S\_\{11\}\-S\_\{00\}=\\underbrace\{\(S\_\{01\}\-S\_\{00\}\)\}\_\{E\_\{0\}\}\+\\underbrace\{\(S\_\{11\}\-S\_\{01\}\)\}\_\{A\_\{1\}\},\(1\)=\(S10−S00\)⏟A0\+\(S11−S10\)⏟E1\.\\displaystyle=\\underbrace\{\(S\_\{10\}\-S\_\{00\}\)\}\_\{A\_\{0\}\}\+\\underbrace\{\(S\_\{11\}\-S\_\{10\}\)\}\_\{E\_\{1\}\}\.\(2\)E0E\_\{0\}is the*evidence gain*with source\-0 answers held fixed, andE1E\_\{1\}uses source\-1 answers\. The first order changes evidence then answers; the second changes answers then evidence\. Their difference is the interaction

I=S11−S10−S01\+S00=E1−E0\.I=S\_\{11\}\-S\_\{10\}\-S\_\{01\}\+S\_\{00\}=E\_\{1\}\-E\_\{0\}\.\(3\)Averaging the orders assigns\(E0\+E1\)/2\(E\_\{0\}\+E\_\{1\}\)/2to evidence\. These equalities hold for nonlinear corpus metrics as well as accuracy\. They allocate score changes, not causal effects on reasoning\. We use additive score points because component\-to\-total ratios are unstable when the total is small\. The answers are fixed within each evidence comparison; another input condition may produce different answers\.

#### When marginal rates suffice\.

For strict accuracy, the score of a claim is the product of answer correctness and evidence eligibility\. The off\-diagonal scores follow exactly from these two indicators\. At input conditioncc, letai​\(c\)a\_\{i\}\(c\)denote source\-0 answer correctness, averaged over the reported replicates, and letdi​\(c\)=q1​i​\(c\)−q0​i​\(c\)d\_\{i\}\(c\)=q\_\{1i\}\(c\)\-q\_\{0i\}\(c\)be the difference between the two evidence\-eligibility indicators\. Then

E0​\(c\)=100​a⁡\(c\)​d​\(c\)¯=100​\[a⁡\(c\)¯​d⁡\(c\)¯\+Covn⁡\(a⁡\(c\),d⁡\(c\)\)\],E\_\{0\}\(c\)=100\\,\\overline\{a\(c\)d\(c\)\}=100\\left\[\\overline\{a\(c\)\}\\,\\overline\{d\(c\)\}\+\\operatorname\{Cov\}\_\{n\}\(a\(c\),d\(c\)\)\\right\],\(4\)where the covariance uses denominatornn\. Multiplying marginal answer accuracy by the eligibility difference omits this covariance: whether eligibility changes on the same claims that the model answers correctly\. We apply this diagnostic in both studies\. SciFact uses corpus F1, so we score its four combinations directly rather than apply the strict\-accuracy factorization\.

## 4Study 1: Evidence Replacement with Trained Verifiers

### 4\.1Design

We evaluate four trained DeBERTa\-large\-MNLI checkpoints\([He et al\., 2021](https://arxiv.org/html/2609.27064#bib.bib8);[Williams et al\., 2018](https://arxiv.org/html/2609.27064#bib.bib21)\)on DCUF and UnifEE evidence for the same 7,890 FEVEROUS development claims\. Each checkpoint generates answers from both sources\. We hold its weights fixed, then score both answer vectors with both evidence submissions\. For the FEVEROUS strict score, answer correctness and evidence eligibility factorize, so the off\-diagonal values are exact recombinations of saved indicators\.

All four checkpoints start from the same model revision and are fine\-tuned on DCUF\-derived inputs\. Each sees 16,000 distinct training examples, updates all 406\.2 million parameters, and is evaluated at the fixed final update of 1,000\. Inputs are limited to 512 serialized tokens: a 64\-token claim block and up to eight resolved evidence units, each capped at 96 tokens, selected from the first 30 candidate IDs\. We include optional context when available, omit unresolved units, and use a sentinel for empty inputs\. Scoring preserves provider order with at most five nonstructured and 25 structured items\. Appendix[A](https://arxiv.org/html/2609.27064#A1)gives the full training and input specification\.

The comparison controls verifier weights while changing the evidence source\. It also changes content, ordering and the distribution of inputs relative to training\. We therefore interpret the score split for these DCUF\-trained checkpoints\. The development population had been inspected previously, making this a retrospective comparison under a fixed execution rule\.

### 4\.2A Large Joint Gain with a Smaller Accuracy Gain

Table 2:Study 1 mean FEVEROUS strict scores over four trained checkpoints\. Each row uses answers generated from the named evidence source and scores them with either evidence submission\.Strict score rises from 32\.41 to 42\.02, a mean gain of 9\.61 points with paired 95% interval \[8\.77, 10\.43\]\. Answer accuracy increases from 68\.09 to 70\.05 \(1\.96 points\), and macro\-F1 increases by 1\.44 points\. The four cells in Table[2](https://arxiv.org/html/2609.27064#S4.T2)identify the part of the larger joint gain that survives at fixed answers\.

Changing evidence first contributesE0=7\.92E\_\{0\}=7\.92\{\}points; changing answers next contributesA1=1\.69A\_\{1\}=1\.69\{\}\. Reversing the order assignsA0=0\.53A\_\{0\}=0\.53\{\}points to answers andE1=9\.08E\_\{1\}=9\.08\{\}to evidence\. Table[3](https://arxiv.org/html/2609.27064#S4.T3)gives the component intervals\. The 1\.15\-point difference between evidence gains reflects the interaction between which answers are correct and which submissions are eligible\.

Table 3:Study 1 score decomposition in points\. All six quantities are contrasts of the same four score cells\. Pointwise 95% intervals use 10,000 paired claim\-bootstrap draws conditional on the four trained checkpoints\. The five component/interaction intervals were added post hoc; the total\-gain interval reproduces the original analysis\.We use 10,000 paired claim\-bootstrap draws with the original RNG seed 20261099\. Each draw shares its claim sample across both sources and all four checkpoints\. The five component/interaction intervals are post\-hoc, pointwise 95% summaries conditional on those checkpoints; the total\-gain interval reproduces the original analysis\. They describe each estimate separately and carry no simultaneous\-coverage guarantee\. A claim is the sampling unit; the 31,560 checkpoint/claim rows reuse each claim four times\.

Every checkpoint predicts only SUPPORTS or REFUTES\. The 501 NEI\-gold claims consequently contribute zero to all four strict scores\. The entire gain comes from 3,908 SUPPORTS and 3,481 REFUTES claims, contributing 5\.93 and 3\.69 points respectively on the 7,890\-claim denominator \(Appendix[A](https://arxiv.org/html/2609.27064#A1)\)\. The decomposition remains exact, but its size need not describe a verifier that handles all three classes well\.

The primary population includes 239 claims with an unresolved input unit under at least one source\. Restricting to the predeclared 7,651 both\-source\-resolved claims gives a gain of 9\.42 \[8\.62, 10\.25\]\. All four checkpoint\-specific endpoint gains are positive\. Appendix[A](https://arxiv.org/html/2609.27064#A1)reports these sensitivities and the original metric intervals\.

### 4\.3How Much Do Marginal Rates Explain?

The evidence\-eligibility rate increases from 43\.22% to 55\.07%\. These aggregate rates closely agree with the development\-set evidence recalls reported by[Hu et al\. \(2023, Table 2\)](https://arxiv.org/html/2609.27064#bib.bib10): 43\.22% and 55\.08%, respectively\. Multiplying our 11\.85\-point eligibility difference by source\-0 answer accuracy \(68\.09%\) predicts an evidence gain of 8\.07 points\. Equation[4](https://arxiv.org/html/2609.27064#S3.E4)gives the residual: a covariance contribution of \-0\.14 points, yielding the observed 7\.92\. With source\-1 answers, the corresponding product is 8\.30 and the covariance contribution is 0\.78, yielding 9\.08\.

Thus marginal rates approximate one order well but are less accurate for the other\. The covariance records the overlap between correct answers and changing evidence eligibility\. In Study 2 we use the same diagnostic to test whether marginal changes explain the context effect\.

### 4\.4From Source Gains to Interface Sensitivity

Changing only the evidence passed to the scorer preserves 7\.92 or 9\.08 of the trained\-verifier comparison’s 9\.61\-percentage\-point gain\. Does this evidence gain depend on how much evidence a model sees or how it returns an answer? We examine that question with fixed\-weight LLMs, BM25 and BGE evidence orderings, and three benchmarks \(Table[4](https://arxiv.org/html/2609.27064#S4.T4)\)\.

Table 4:The two study designs\. Training seeds in Study 1 and decoding seeds in Study 2 support different uncertainty statements; effects are reported separately\.The studies also have distinct but historically related populations\. Study 2’s FEVEROUS panel is disjoint by ID from Study 1’s development claims, but comes from a previously used checkpoint\-validation pool\. An earlier LLM development study using DCUF/UnifEE failed its quality gates and stopped before primary evaluation\. The completed BM25/BGE study follows its own later protocol\. Appendix[B](https://arxiv.org/html/2609.27064#A2)records this history and the population joins\.

## 5Study 2: Frozen LLM Evaluation

We useE0E\_\{0\}, the evidence gain with answers generated from BM25 evidence held fixed, as the primary outcome\. We specified the format and context contrasts and the decision rule before generating the primary responses; the later analyses examine the resulting patterns\.

### 5\.1Data and Sampling Units

Table[5](https://arxiv.org/html/2609.27064#S5.T5)gives the evaluation populations\. We select 2,000 FEVER development claims by a deterministic hash and 2,000 eligible FEVEROUS claims from a historical verifier\-validation pool\. The FEVEROUS selection excludes families linked to reserved or previously inspected development cases\. SciFact includes all 300 development claims, each evaluated with three retrieved abstracts\.

Table 5:Study 2 populations\. A family groups related claims and is resampled as one unit\. SciFact’s three abstracts for a claim remain together\.DatasetClaimsUnitsFamiliesOfficial metricFEVER2,0002,0001,963Strict scoreFEVEROUS2,0002,0001,987Strict scoreSciFact300900247Abstract\-rationalized F1We group claims into families using normalized exact matches, narrowly defined numeric/negation mutations, available citation links, and flagged near duplicates from each claim’s top 50 lexical neighbors\. The supplement records the assignments\. This reduces known dependence, although semantic duplicates may remain\. We froze the protocol internally after development and before the primary runs; its hashed record is included, without external preregistration\.

### 5\.2Evidence and Answer Interfaces

Each evaluation unit has one candidate pool and two orderings: BM25 andBAAI/bge\-reranker\-base\. For Wikipedia we retrieve five pages and retain at most 100 evidence units; for SciFact we retrieve three abstracts and rank their sentences\. BGE reranks within these pools\. We assemble whole evidence units in order until adding the next unit would exceed either model’s evidence\-token budget\. Both models therefore see the same evidence text, with model\-specific chat formatting\. Appendix[C](https://arxiv.org/html/2609.27064#A3)gives retrieval queries, tie breaks, model revisions and truncation details\.

The scorer receives a prefix of the visible evidence, limited to five FEVER sentences, five FEVEROUS sentences and 25 cells, or three rationale sentences per SciFact abstract\. Thus increasing context can change both the model’s input and the evidence submitted for scoring\. Gold annotations are used for evaluation and prespecified controls; primary retrieval misses are retained\. Gold SciFact abstracts outside the retrieved set remain in the relevant\-abstract denominator\.

We request either one code \(A, B or C, mapped to the class labels\) or one exact benchmark label string\. After stripping whitespace, every other response isINVALID\. We retain these responses and apply the specified scorer\. Prompt\-building functions, mappings and scorer code are public in the supplement; fully instantiated prompt records are archived separately\.

The evidence rules differ across benchmarks\. FEVER exempts correctly predicted NEI answers from evidence matching; the FEVEROUS implementation requires an annotated group even for NEI\. SciFact’s abstract\-rationalized F1 is2​C/\(P\+R\)2C/\(P\+R\):CCcounts correctly labeled and rationalized abstracts,PPsubmitted non\-NEI abstracts, andRRrelevant gold abstracts\. A SciFact NEI or invalid answer omits that abstract from the submission\. Invalid responses still count as incorrect in the separate answer\-accuracy diagnostic, but their effect on the joint metric depends on its precision/recall denominator\.

### 5\.3Models and Controls

We run Qwen3\-8B and Llama\-3\.1\-8B\-Instruct with fixed weights in bfloat16 and a 16\-token output limit\. Qwen uses its non\-thinking chat option\. Each condition includes greedy decoding and five sampled replicates \(seeds 1103, 2207, 3301, 4409 and 5519\), with temperature 0\.7, top\-pp0\.8, top\-kk20 and repetition penalty 1\.0\. A deterministic hash of the dataset, claim, document and replicate sets each unit’s seed, shared across sources, formats and budgets\.

The primary matrix contains

4,900​units×2​sources×2​contexts×2​formats×2​models×6​replicates=470,4004\{,\}900\\ \\text\{units\}\\times 2\\ \\text\{sources\}\\times 2\\ \\text\{contexts\}\\times 2\\ \\text\{formats\}\\times 2\\ \\text\{models\}\\times 6\\ \\text\{replicates\}=470\{,\}400responses\. Prespecified greedy controls test evidence removal, code mappings, cross\-claim evidence swaps, complete annotated evidence where it fits, and evidence order\. They require 42,834 additional generations after reuse of identical requests\. Appendix[F](https://arxiv.org/html/2609.27064#A6)reports each control’s population, feasibility and results\. All primary responses, including parsing failures, are saved; subsequent analyses reuse them\.

### 5\.4Inference and Decision Rule

For each dataset–model pair, we average the format and context differences inE0E\_\{0\}over the other factor and five sampled seeds:

Dformat\\displaystyle D\_\{\\mathrm\{format\}\}=15​∑s12​∑c\[E0​\(labels,c,s\)−E0​\(codes,c,s\)\],\\displaystyle=\\frac\{1\}\{5\}\\sum\_\{s\}\\frac\{1\}\{2\}\\sum\_\{c\}\[E\_\{0\}\(\\mathrm\{labels\},c,s\)\-E\_\{0\}\(\\mathrm\{codes\},c,s\)\],\(5\)Dcontext\\displaystyle D\_\{\\mathrm\{context\}\}=15​∑s12​∑f\[E0​\(f,2048,s\)−E0​\(f,256,s\)\]\.\\displaystyle=\\frac\{1\}\{5\}\\sum\_\{s\}\\frac\{1\}\{2\}\\sum\_\{f\}\[E\_\{0\}\(f,2048,s\)\-E\_\{0\}\(f,256,s\)\]\.\(6\)These form the 12 primary contrasts\. Greedy outputs are descriptive references\. We also partition the evidence gains by whether the two sources yield identical valid answers; for SciFact, all three abstract labels must match\. Each partition keeps the relevant full\-corpus denominator, so its contributions sum back to the original metric\. We do not calculate a separate F1 for each subgroup\.

For each panel, 50,000 bootstrap draws sample whole claim families and independently sample five decoding\-seed indices with replacement\. Each draw uses the same samples in every paired condition\. We aggregate the scoring counts, compute the corpus metric for each sampled seed, then average\. In particular, SciFact F1 is computed from corpus totals\. The recorded implementation uses NumPy PCG64 seed 20260916 and batches of 128\. Linear percentile bounds at0\.05/\(2×12\)0\.05/\(2\\times 12\)and1−0\.05/\(2×12\)1\-0\.05/\(2\\times 12\)apply Bonferroni adjustment across the primary family\. Coverage is approximate and conditional on the selected models and family construction\.

The protocol calls an effect*replicated dependence*when one factor has the same direction, magnitude of at least two score points, an adjusted interval excluding zero, and at least four of five agreeing seed signs for both models on the same two or more datasets, including SciFact\. It calls all effects small \(*bounded stability*\) when all 12 adjusted intervals lie inside\[−2,\+2\]\[\-2,\+2\]\. All other outcomes receive the recorded label*restricted/mixed evidence*\. These labels describe prespecified decision rules, not new kinds of statistical evidence\.

## 6LLM Results

### 6\.1Primary Contrasts

Figure[1](https://arxiv.org/html/2609.27064#S6.F1)shows the 12 primary contrasts; Appendix[D](https://arxiv.org/html/2609.27064#A4)gives the exact values\.

Figure 1:Primary format and context effects on the fixed\-answer evidence gain, with Bonferroni\-adjusted intervals\. Dashed lines mark the prespecified two\-point threshold\.Increasing context raisesE0E\_\{0\}on FEVEROUS by 3\.84 points for Qwen \[1\.83, 5\.91\] and 3\.10 for Llama \[1\.20, 5\.01\]; all five seed effects are positive\. Both panels meet the local magnitude, interval and sign criteria\. Neither factor, however, meets the required cross\-dataset pattern\. SciFact is required by that rule and qualifies for neither model; FEVER also fails the common two\-point criterion\. The same token increase retains different amounts of material on each dataset, as Section[7](https://arxiv.org/html/2609.27064#S7)explains\.

The smaller positive contrasts are Qwen’s FEVER context effect \(1\.17, \[0\.52, 1\.89\]\) and Llama’s FEVEROUS format effect \(1\.16, \[0\.34, 1\.98\]\)\. All other format intervals include zero\. SciFact’s adjusted context intervals are \[\-1\.65, 1\.83\] for Qwen and \[\-0\.63, 3\.17\] for Llama\. The latter includes a FEVEROUS\-sized effect, while the former excludes a positive two\-point effect\.

Several intervals extend beyond\[−2,\+2\]\[\-2,\+2\]\. We can therefore establish neither the required replication nor that all effects lie within the small\-effect bound\. The recorded outcome is*restricted/mixed evidence*: context dependence on FEVEROUS with inconclusive or smaller effects elsewhere\.

### 6\.2Endpoint Gains and Decomposition Order

Evidence gain and endpoint performance can move in opposite directions\. For Qwen on FEVER, increasing context raisesE0E\_\{0\}by 1\.17 points while the BGE endpointS11S\_\{11\}falls by 1\.83 points \(Table[6](https://arxiv.org/html/2609.27064#S6.T6)\)\. The source gainS11−S00S\_\{11\}\-S\_\{00\}increases by 1\.65 points\. On FEVEROUS, Qwen’sE0E\_\{0\}rises from 2\.28 to 6\.11 while its BGE endpoint rises by 13\.68\. A gain from changing scored evidence, a gain from changing sources, and the performance at an endpoint answer different questions\.

Table 6:The four scores and evidence gain by context budget, averaged over both answer formats and five sampled seeds\. Scores use the same input and scoring budget within each row\.Table 7:Context effects on the two evidence gains and their average\. Intervals use 1,000,000 draws with adjustment over the 60\-comparison exploratory family\. Full format and endpoint comparisons are in the supplement\.The choice of fixed answers also matters\. On FEVEROUS, Qwen’s context contrast is 3\.84 underE0E\_\{0\}and 2\.04 underE1E\_\{1\}; Llama’s is 3\.10 and 2\.30\. Averaging the two orders gives 2\.94 and 2\.70\. Table[7](https://arxiv.org/html/2609.27064#S6.T7)uses 1,000,000 paired bootstrap draws and adjustment over a 60\-comparison exploratory family\. The crossed\-budget and direct interaction analyses below form a separate 42\-comparison post\-hoc family\. These retain the original resampling units; their guarantees do not combine across families\. Appendix[K](https://arxiv.org/html/2609.27064#A11)records the numerical\-precision checks\.

For a concrete example, Qwen gives the same correctSUPPORTSanswer for FEVEROUS claim 236 under both sources, but BGE omits a required sentence\. The four scores are\(1,0,1,0\)\(1,0,1,0\)\. Claim 472 gives the reverse pattern,\(0,1,0,1\)\(0,1,0,1\)\. These post\-hoc, lowest\-ID examples illustrate how evidence alone can change credit; Appendix[G](https://arxiv.org/html/2609.27064#A7)provides the traces and selection rules\.

### 6\.3Crossing Input and Scoring Budgets

A larger context budget can change both the generated answer and the evidence reaching the scorer\. We examine their contributions by crossing the two budgets\. LetTa​b​\(u,v\)T\_\{ab\}\(u,v\)score source\-aaanswers generated at input budgetuuwith source\-bbevidence submitted at scoring budgetvv\. The original conditions haveu=vu=v\. DefineF⁡\(u,v\)=T01​\(u,v\)−T00​\(u,v\)F\(u,v\)=T\_\{01\}\(u,v\)\-T\_\{00\}\(u,v\), averaging the same formats and five per\-seed corpus scores\. For short and long budgetss,ls,l,

F⁡\(l,l\)−F⁡\(s,s\)\\displaystyle F\(l,l\)\-F\(s,s\)=\[F⁡\(l,s\)−F⁡\(s,s\)\]\+\[F⁡\(l,l\)−F⁡\(l,s\)\]\\displaystyle=\[F\(l,s\)\-F\(s,s\)\]\+\[F\(l,l\)\-F\(l,s\)\]\(7\)=\[F⁡\(s,l\)−F⁡\(s,s\)\]\+\[F⁡\(l,l\)−F⁡\(s,l\)\]\.\\displaystyle=\[F\(s,l\)\-F\(s,s\)\]\+\[F\(l,l\)\-F\(s,l\)\]\.\(8\)The first order changes input at fixed short scoring evidence, then scoring evidence at fixed long\-input answers\. The second reverses the steps\. Both use the saved outputs\.

On FEVEROUS, the scoring\-budget component for Qwen is 2\.91 points when input changes first, or 1\.79 when scoring changes first, out of the 3\.84\-point context contrast\. Llama’s values are 2\.56 and 2\.23 out of 3\.10\. The remainder comes from changing the answers at fixed scoring evidence\. For FEVER/Qwen, scoring changes contribute only 0\.08 or 0\.10 points, compared with input components of 1\.07 and 1\.09\. SciFact has unchanged rationale eligibility at both budgets, making its scoring components and input–scoring interaction exactly zero for these submissions\.

Table 8:FEVEROUS context\-effect decomposition in score points\. Input\-first holds scoring evidence at the short budget for the input change; scoring\-first holds answers at the short input budget for the scoring change\. Intervals use 1,000,000 draws and the 42\-comparison adjustment\. Appendix[L](https://arxiv.org/html/2609.27064#A12)gives all datasets\.The intervals use 1,000,000 paired draws and a separate 42\-comparison post\-hoc adjustment \(tail probability0\.05/840\.05/84\)\. The plan was recorded after inspecting earlier results and before these calculations\. Full results for all datasets appear in Appendix[L](https://arxiv.org/html/2609.27064#A12)\.

#### What the intervals support\.

Llama’s FEVEROUS scoring components exclude zero in both orders\. Qwen’s input components exclude zero in both orders; its scoring component does so only when scoring changes second\. Input–scoring interaction intervals include zero in every panel\. The direct context change inE1−E0E\_\{1\}\-E\_\{0\}is \-1\.80 \[\-3\.05, \-0\.64\] for Qwen on FEVEROUS and \-0\.80 \[\-2\.05, 0\.41\] for Llama \(Table[24](https://arxiv.org/html/2609.27064#A8.T24)\), supporting an order difference for Qwen\. In the separate 60\-comparison family, both FEVEROUS averaged\-order intervals exclude zero and bothE1E\_\{1\}intervals include it\. The direct paired contrast, rather than these separate zero\-exclusion outcomes, is the relevant comparison of orders\.

#### Where the FEVER input component occurs\.

For FEVER/Qwen, BGE adds evidence eligibility on 104 claims at 256 tokens and 108 at 2,048; 103 are common to both sets\. Their 109\-claim union accounts arithmetically for the full input contribution\. Within this post\-hoc group, accuracy with BM25 evidence rises from 38\.72% to 59\.54%, while accuracy with BGE evidence changes from 64\.04% to 62\.20% \(Table[9](https://arxiv.org/html/2609.27064#S6.T9)\)\. The group’s contributions, using the full 2,000\-claim denominator, are 1\.065 and 1\.085 points under short and long scoring budgets; the remaining claims net to zero\. This localizes the score change without requiring an account of the model’s internal evidence use\.

Table 9:FEVER/Qwen answer accuracy in the post\-hoc union of 109 claims where BGE adds eligibility: 104 at the short budget, 108 at the long budget and 103 at both\. Means include both formats and five sampled seeds\.
#### Marginal changes miss the FEVER effect\.

Applying Equation[4](https://arxiv.org/html/2609.27064#S3.E4)at each budget reveals a contrast with Study 1\. On FEVER/Qwen, the product of marginal answer accuracy and eligibility differences predicts a 0\.110\-point context change, compared with 1\.165 observed\. The changing covariance supplies 1\.055 points\. On FEVEROUS/Llama the same product predicts 3\.068 points, close to the observed 3\.100 \(Table[10](https://arxiv.org/html/2609.27064#S6.T10)\)\. Agreement for a change can occur even when the covariance at each budget is nonzero\. The important question is which claims combine correct answers with improved eligibility\.

Table 10:Product and covariance contributions to the context effect in strict\-score points\. The last two terms of Equation[4](https://arxiv.org/html/2609.27064#S3.E4), differenced between budgets, sum to the observed effect before rounding\.

### 6\.4Answer\-Format Sensitivity

The format contrast combines answer choice with compliance with the requested output format\. On FEVEROUS, Llama produces at least one invalid response in 13\.97% of claim\-condition observations with codes, compared with 8\.16% with label strings \(Table[25](https://arxiv.org/html/2609.27064#A9.T25)\)\. Each observation covers both sources; the rates pool both budgets and all six replicates\. As a sensitivity check, we apply a conservative alternative parser to every saved response, accepting unambiguous surface variants without consulting gold labels\. FEVEROUS context estimates remain positive and close to their original values\. Appendix[I](https://arxiv.org/html/2609.27064#A9)reports the remaining failures and definitions; Appendix[F](https://arxiv.org/html/2609.27064#A6)gives the code\-mapping and other controls\.

## 7Why the Context Change Differs Across Datasets

The two budgets add different amounts of visible evidence across benchmarks\. At 256 tokens, the average retained fraction is about 5% on FEVEROUS, 30% on FEVER and 65% on SciFact; at 2,048 tokens it is about 44%, 100% and 100%\. Some material remains excluded at the larger budget for 1,998 of 2,000 FEVEROUS claims\. Its structural categories include 739 mixed sentence/table claims and 463 table\-or\-cell claims\. Thus the replication rule tests a common token\-budget change, not an equal increase in information\.

Figure 2:Visible evidence retained and evidence eligibility under the official submission limits\. A claim is in “BGE adds” if BGE improves eligibility at either budget and worsens it at neither; “BGE loses” reverses the comparison\. Corrected profiles use normalized evidence identifiers \(Appendix[J](https://arxiv.org/html/2609.27064#A10)\)\.Figure[2](https://arxiv.org/html/2609.27064#S7.F2)also distinguishes visible evidence from eligible submissions\. We corrected the FEVEROUS diagnostic to use normalized page identifiers and the evidence actually submitted within the scoring limits\. This changes the profiles but leaves primary scores and intervals unchanged\. SciFact additionally has 37 recorded document\-retrieval misses among 300 claims\. Its profile counts the eligible retrieved abstracts, and its corpus F1 differs from the two strict\-accuracy metrics\. These profiles characterize the different interventions; Appendix[J](https://arxiv.org/html/2609.27064#A10)records the corrections and per\-benchmark definitions\.

## 8Discussion and Limitations

#### What the two studies establish\.

In Study 1’s DCUF\-trained checkpoints, which predict no NEI, a 9\.61\-point joint\-score gain contains 7\.92 or 9\.08 points from evidence replacement at fixed answers\. In Study 2’s two LLMs on FEVEROUS, this evidence gain changes with context budget\. The studies provide complementary tests of the measurement question, but they use different sources, models and populations\. Study 2 therefore leaves open whether the same interface dependence holds for Study 1’s trained verifiers\.

Both findings are consistent with the purpose of a joint metric: improving eligible evidence should improve the score\. The added insight is how much of a particular gain remains without changing the answers, and which claims produce it\. Marginal rates closely approximate Study 1’sE0E\_\{0\}, but the covariance term accounts for most of the FEVER/Qwen context contrast\. Appendix[M](https://arxiv.org/html/2609.27064#A13)describes the saved\-output replays\.

#### What to report\.

For a source comparison, retain each claim’s answers and submitted evidence under both conditions\. The official scorer can then produce the two off\-diagonal scores without more inference\. Report the four scores, both decomposition orders and answer accuracy, with class metrics and invalid\-response rates where relevant\. Distinguish model\-visible evidence, submitted evidence and eligibility under the scoring rule\. This requires access to both sets of outputs: an endpoint\-only leaderboard cannot reconstruct the decomposition after the fact\. For nonlinear metrics, retain the sufficient counts and recompute the corpus score\.

#### Limits of Study 1\.

All checkpoints were trained on DCUF\-derived inputs\. Switching to UnifEE therefore also changes the input distribution, potentially affecting the answer components; the evidence share should not be extrapolated to source\-symmetric or stronger verifiers\. The 501 NEI\-gold claims contribute zero to every cell, so the gains describe SUPPORTS/REFUTES performance on the full\-population denominator\. Source replacement also changes content, order, context, resolution and truncation together\. The intervals condition on four fixed checkpoints\. Saved predictions and historical reload records support the recorded comparison, while missing weights and runtime inputs limit full model reproduction\. The retrospective override in Appendix[A](https://arxiv.org/html/2609.27064#A1)shares Study 1’s claims and has unresolved threshold\-selection history\.

#### Limits of Study 2\.

We evaluate three English benchmarks, two fixed 8B models, two budgets and one reranker within fixed candidate pools\. Decoding seeds do not capture training or model\-selection uncertainty\. The FEVEROUS panel comes from a historically exposed checkpoint\-validation pool; residual semantic duplication and pretraining overlap are unknown\. Evidence eligibility measures agreement with annotated identifiers, without new human relevance judgments\. SciFact has 247 families and a different corpus metric; its 809 training claims lie outside the selected population\.

The output\-format sensitivity retains residual invalid responses and code\-mapping effects\. Many invalid Llama responses reach the output limit, as Appendix[H](https://arxiv.org/html/2609.27064#A8)quantifies\. The post\-hoc budget and covariance analyses describe these saved outputs rather than model attention\. Earlier source\-matched LLM development gates failed; Appendix[B](https://arxiv.org/html/2609.27064#A2)distinguishes that development history from the completed study\.

## 9Conclusion

Joint fact\-verification scores combine answer quality with evidence requirements\. In four DCUF\-trained verifiers with zero NEI recall, most of a 9\.61\-point source gain remains when answers are held fixed\. In two LLMs on FEVEROUS, the fixed\-answer evidence gain changes with context, while the prespecified cross\-dataset criterion is unmet\. The four answer–evidence score combinations expose these differences and reveal when aggregate accuracy and evidence\-coverage rates are insufficient\. They provide a reproducible account of what changed in a score, alongside the benchmark’s intended endpoint measure\.

## References

- Aly et al\. \(2021\)Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal\.The fact extraction and verification over unstructured and structured information shared task\.In*Proceedings of the Fourth Workshop on Fact Extraction and Verification*, pp\. 1–13, 2021\.doi:10\.18653/v1/2021\.fever\-1\.1\.
- Bernardelle et al\. \(2026\)Pietro Bernardelle, Stefano Civelli, Kevin Roitero, and Gianluca Demartini\.Context shapes LLMs retrieval\-augmented fact\-checking effectiveness\.*arXiv preprint arXiv:2602\.14044*, 2026\.
- Buchmann et al\. \(2024\)Jan Buchmann, Xiao Liu, and Iryna Gurevych\.Attribute or abstain: Large language models as long document assistants\.In*Proceedings of EMNLP*, pp\. 8113–8140, 2024\.doi:10\.18653/v1/2024\.emnlp\-main\.463\.
- Card et al\. \(2020\)Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky\.With little power comes great responsibility\.In*Proceedings of EMNLP*, pp\. 9263–9274, 2020\.doi:10\.18653/v1/2020\.emnlp\-main\.745\.URL[https://aclanthology\.org/2020\.emnlp\-main\.745/](https://aclanthology.org/2020.emnlp-main.745/)\.
- Dror et al\. \(2018\)Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart\.The hitchhiker’s guide to testing statistical significance in natural language processing\.In*Proceedings of ACL*, pp\. 1383–1392, 2018\.doi:10\.18653/v1/P18\-1128\.URL[https://aclanthology\.org/P18\-1128/](https://aclanthology.org/P18-1128/)\.
- Gao et al\. \(2023\)Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen\.Enabling large language models to generate text with citations\.In*Proceedings of EMNLP*, pp\. 6465–6488, 2023\.doi:10\.18653/v1/2023\.emnlp\-main\.398\.URL[https://aclanthology\.org/2023\.emnlp\-main\.398/](https://aclanthology.org/2023.emnlp-main.398/)\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori et al\.The Llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*, 2024\.
- He et al\. \(2021\)Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen\.DeBERTa: Decoding\-enhanced BERT with disentangled attention\.In*International Conference on Learning Representations*, 2021\.URL[https://openreview\.net/forum?id=XPZIaotutsD](https://openreview.net/forum?id=XPZIaotutsD)\.
- Hu et al\. \(2022\)Nan Hu, Zirui Wu, Yuxuan Lai, Xiao Liu, and Yansong Feng\.Dual\-channel evidence fusion for fact verification over texts and tables\.In*Proceedings of NAACL\-HLT*, pp\. 5232–5242, 2022\.doi:10\.18653/v1/2022\.naacl\-main\.384\.
- Hu et al\. \(2023\)Nan Hu, Zirui Wu, Yuxuan Lai, Chen Zhang, and Yansong Feng\.UnifEE: Unified evidence extraction for fact verification\.In*Proceedings of EACL*, pp\. 1150–1160, 2023\.doi:10\.18653/v1/2023\.eacl\-main\.82\.
- Hua et al\. \(2025\)Andong Hua, Kenan Tang, Chenhe Gu, Jindong Gu, Eric Wong, and Yao Qin\.Flaw or artifact? Rethinking prompt sensitivity in evaluating LLMs\.In*Proceedings of EMNLP*, pp\. 19889–19899, 2025\.doi:10\.18653/v1/2025\.emnlp\-main\.1006\.
- Liu et al\. \(2024\)Nelson F\. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang\.Lost in the middle: How language models use long contexts\.*Transactions of the Association for Computational Linguistics*, 12:157–173, 2024\.doi:10\.1162/tacl\_a\_00638\.
- Owen \(2007\)Art B\. Owen\.The pigeonhole bootstrap\.*The Annals of Applied Statistics*, 1\(2\):386–411, 2007\.doi:10\.1214/07\-AOAS122\.URL[https://arxiv\.org/abs/0712\.1111](https://arxiv.org/abs/0712.1111)\.
- Qwen Team \(2025\)Qwen Team\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.
- Robertson & Zaragoza \(2009\)Stephen Robertson and Hugo Zaragoza\.The probabilistic relevance framework: BM25 and beyond\.*Foundations and Trends in Information Retrieval*, 3\(4\):333–389, 2009\.
- Schuster et al\. \(2019\)Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay\.Towards debiasing fact verification models\.In*Proceedings of EMNLP\-IJCNLP*, pp\. 3419–3425, 2019\.doi:10\.18653/v1/D19\-1341\.URL[https://aclanthology\.org/D19\-1341/](https://aclanthology.org/D19-1341/)\.
- Sclar et al\. \(2024\)Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr\.Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting\.In*Proceedings of the Twelfth International Conference on Learning Representations*, 2024\.
- Shorrocks \(2013\)Anthony F\. Shorrocks\.Decomposition procedures for distributional analysis: a unified framework based on the Shapley value\.*The Journal of Economic Inequality*, 11:99–126, 2013\.doi:10\.1007/s10888\-011\-9214\-z\.URL[https://doi\.org/10\.1007/s10888\-011\-9214\-z](https://doi.org/10.1007/s10888-011-9214-z)\.
- Thorne et al\. \(2018\)James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal\.FEVER: a large\-scale dataset for fact extraction and verification\.In*Proceedings of NAACL\-HLT*, pp\. 809–819, 2018\.doi:10\.18653/v1/N18\-1074\.
- Wadden et al\. \(2020\)David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi\.Fact or fiction: Verifying scientific claims\.In*Proceedings of EMNLP*, pp\. 7534–7550, 2020\.doi:10\.18653/v1/2020\.emnlp\-main\.609\.
- Williams et al\. \(2018\)Adina Williams, Nikita Nangia, and Samuel R\. Bowman\.A broad\-coverage challenge corpus for sentence understanding through inference\.In*Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long Papers\)*, pp\. 1112–1122, New Orleans, Louisiana, 2018\. Association for Computational Linguistics\.doi:10\.18653/v1/N18\-1101\.URL[https://aclanthology\.org/N18\-1101/](https://aclanthology.org/N18-1101/)\.
- Xiao et al\. \(2023\)Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian\-Yun Nie\.C\-Pack: Packed resources for general Chinese embeddings\.*arXiv preprint arXiv:2309\.07597*, 2023\.
- Zheng et al\. \(2024\)Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang\.Large language models are not robust multiple choice selectors\.In*International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=shr9PXz7T0](https://openreview.net/forum?id=shr9PXz7T0)\.

## Appendix ARetrospective Case and Trained\-Verifier Records

#### Retrospective override\.

On 7,890 FEVEROUS development claims, an auxiliary four\-model ensemble kept the public DCUF answer unless its mean NEI probability reached 0\.23\. Macro\-F1 rose from 49\.34 to 54\.16 and NEI\-F1 from zero to 17\.67, while accuracy fell from 72\.05 to 69\.01 and strict score from 35\.16 to 34\.78 \(Table[11](https://arxiv.org/html/2609.27064#A1.T11)\)\. Of 597 final NEI predictions, 595 changed and two stayed the same: 54 lose strict correctness, 24 gain it, and 519 remain incorrect\. All 54 losses had eligible evidence according to the saved flags\.

Table 11:Retrospective NEI override: saved answer and strict\-score metrics on 7,890 development claims\.We join the answers and per\-model probabilities by FEVEROUS ID\. Their mean reproduces the ensemble, and the 0\.23 rule reproduces every hybrid answer\. NEI is below the highest\-probability class in 48 of the 54 losses; that class is the gold answer in 39\. It is also below the highest\-probability class in 17 of the 24 gains\. The rule therefore both discards correct modal decisions and recovers gold NEI cases\.

The retained records show development\-informed candidate advancement and a later development sweep\. They leave independent selection of the final threshold, four\-model inclusion and rounded\-mean rule unresolved\. The original weights, successful\-run source snapshot and exact model inputs are unavailable\. This case is a retrospective description of the saved rule\. It reuses seed numbers with Study 1, but uses different trained models\.

#### Study 1 training and inputs\.

The base ismicrosoft/deberta\-large\-mnli, revision7296194b9009373def4f7c5dad292651e4b5cf4e\. The class order is REFUTES, NEI, SUPPORTS\. A label\-independent ID hash partitions 71,291 training examples into 57,293 training, 7,002 checkpoint\-validation and 6,996 reserved\-calibration claims, all disjoint from the 7,890 development IDs\. Each seed sees 16,000 distinct training examples\. AdamW uses learning rate10−510^\{\-5\}, weight decay 0\.01, unweighted cross\-entropy, FP32 microbatch 2 with accumulation 8, 60 warmup steps and decay to zero at update 1,000\. Gradient clipping is 1\.0; gradient checkpointing is enabled\.

The input builder validates and deduplicates the first 30 candidate IDs, selects up to eight resolved units, and applies the block and sequence limits described in Section[4](https://arxiv.org/html/2609.27064#S4)\. Scoring preserves provider order within the 5/25 submission limits\. Five DCUF and 235 UnifEE cases have unresolved direct units, including one shared case\. The 7,651 both\-source\-resolved claims include 15 empty retrievals handled with the sentinel\. The corpus audit checked archive\-member CRCs and lengths; the published whole\-corpus checksum was not independently reverified\.

Historical reload records report unchanged checkpoint tensor hashes and zero reference\-logit difference across source evaluations\. The supplement contains these records, all eight prediction files and the paired labels/scoring flags\. The trained weights and complete original inputs are excluded\. Full model reproduction would require those missing assets\.

Table 12:Study 1 mean endpoint metrics and paired 95% intervals for the 7,890\-claim population\. These are the original 10,000\-draw claim\-bootstrap results, conditional on the four checkpoints\.Table 13:Study 1 endpoint metrics by training seed\. Each checkpoint is evaluated on both evidence sources\.Table 14:Study 1 contributions by gold label\. Each contribution uses the full population of 7,890 claims as its denominator; columns sum to the corresponding overall scores before rounding\. NEI contributes zero because every checkpoint predicts only SUPPORTS or REFUTES\.
#### Interval reproduction and additional summaries\.

The original metric implementation reconstructs all 180 stored interval rows: 18 endpoint metrics for each of four checkpoints and their mean, in both the 7,890\-claim and 7,651\-claim populations\. For each population, it resets PCG64 to 20261099, draws 10,000 paired claim samples, computes each checkpoint’s source difference and averages\. Linear 2\.5th and 97\.5th percentiles give pointwise intervals\. Undefined ratios remain undefined\. The stored CSVs and original implementation are unchanged\.

A separate post\-hoc script applies the same primary\-population draws toE0,A1,A0,E1E\_\{0\},A\_\{1\},A\_\{0\},E\_\{1\}andII\. It reproduces the original total\-gain interval as a cross\-check and reports pointwise 95% intervals for the five new summaries\. These describe sampling variation conditional on the four checkpoints, separately from all LLM adjustment families\. The script also gives the class contributions in Table[14](https://arxiv.org/html/2609.27064#A1.T14)and the product/covariance terms for both evidence gains\. The original May intervals remain archived results, outside this replay\.

## Appendix BDevelopment History and Study Populations

An earlier LLM development study used DCUF/UnifEE evidence and two input contexts on 2,048 claims\. Both model gates failed, as did a revised Llama label interface\. That study stopped before its planned primary evaluation\. Six exploratory scoring blocks can be reconstructed from 24,576 decisions, with bounds retained for one parser\-unsupported evidence ID\. The completed BM25/BGE study uses different candidates and conditions under its own later protocol\.

ID joins show 7,890 shared claims between the retrospective override and Study 1\. The final FEVEROUS LLM panel overlaps neither Study 1’s development IDs nor the early LLM panel’s IDs\. Both LLM panels come from the historical 7,002\-claim checkpoint\-validation split\. The supplement includes the membership projection and join procedure\. This establishes ID disjointness, while the historical exposure, semantic\-duplicate and pretraining limitations remain as described in Section[8](https://arxiv.org/html/2609.27064#S8)\. Detailed execution history and unexecuted plans are retained in the supplementary method notes\.

## Appendix CLLM Configuration and Evidence Construction

Qwen usesQwen/Qwen3\-8B, revisionb968826d9c46dd6066d109eabc6255188de91218; Llama usesmeta\-llama/Llama\-3\.1\-8B\-Instruct, revision0e9e39f249a16976918f6564b8830bc894c89659\. Tokenizer and weight\-file hashes are in the protocol\. We process one request at a time\. The shared evidence text fits both tokenizers; each model applies its own chat serialization and the recorded per\-unit seed\.

For FEVER and FEVEROUS, A/B/C map to SUPPORTS/REFUTES/NOT ENOUGH INFO\. For SciFact they map to SUPPORT/CONTRADICT/NOT\_ENOUGH\_INFO\. After whitespace stripping, a response must equal one code or the appropriate benchmark label\. Other strings are invalid\. FEVEROUS page IDs use Unicode NFD normalization for scoring; display text and raw outputs are retained unchanged\.

Wikipedia candidates come from SQLite FTS5 BM25 retrieval over titles and bodies\. Queries are the OR of unique casefolded alphanumeric claim tokens\. We retrieve five pages, rank their evidence units lexically, and keep at most 100, with ties broken by evidence ID\. SciFact retrieves three abstracts using the same engine and ranks at most 100 sentences per abstract\. BGE reranks within these pools at revisionaf37ed791788201a1cdcf513e0f584f3aa3be105; reranker pairs are limited to 512 tokens and truncation is recorded\. Material assembly stops when the next whole unit would exceed either tokenizer’s budget\. FEVEROUS cells retain their recorded table context and identities\.

The original protocol specifies IDs, families, materials, parsers, models, requests, analysis code and compute limits\. Its statistical code and outputs are unchanged\. Reference code contains the source\-view, candidate and prompt\-construction policies, but the compact archive excludes the full corpora and a self\-contained indexing pipeline\. Fully instantiated prompts and operational logs remain in the private execution archive\.

Input preparation excludes one FEVEROUS source record with an empty page identifier; the annotation audit found no references to such pages\. FEVER has recorded repairs to malformed hyperlink metadata that preserve sentence IDs and text\. Compatibility and packaging changes, including the local analysis timeout, are documented in the supplementary method notes\.

## Appendix DComplete Primary Results

Table 15:All 12 primary effects in score points\. Format is labels minus codes; context is 2,048 minus 256 tokens\. Seed signs use each unrounded effect and the direction of the mean\.Table[15](https://arxiv.org/html/2609.27064#A4.T15)gives every prespecified contrast shown in Figure[1](https://arxiv.org/html/2609.27064#S6.F1)\.

## Appendix EPer\-Seed Results

Table 16:Primary effects by sampled decoding seed, in score points\. Displayed zeros may be small nonzero values before rounding\.Each entry uses one sampled replicate’s corpus score\. Sign counts use unrounded values\. Greedy results and all 48 condition\-level score tables are intables/native\-score\-table\.csvin the LLM supplement\.

## Appendix FControl Populations and Results

Controls use 64 deterministic hash\-selected claims per dataset, expanded to three abstracts per SciFact claim where required\. They remove evidence, permute the code mapping, swap evidence between eligible claims, reorder evidence units, or supply a complete annotated group that fits the budget\. The controls are descriptive and were not used to retune primary conditions\.

Table 17:Planned and feasible control evaluations and the number of additional requests\. Identical requests are reused; 56 SciFact requests reuse primary outputs\.Of 46,848 planned control evaluations, 44,608 are feasible: 1,848 lack a fitting annotated group and 392 lack five distinct nonidentity orders\. Repeated identical evaluations share requests, leaving 42,890 unique requests\. Of these, 56 reuse primary outputs and 42,834 require additional generation\. The case\-level export records feasibility, answer correctness and agreement with the canonical control\.

Table 18:Descriptive control results \(percent\)\. Claim\-only and Gold report answer accuracy on their respective feasible subsets\. Mapping, Swap and Order report agreement with the corresponding canonical control\. Full counts are in the supplement\.Qwen returns NEI for every claim\-only evaluation; Llama returns NEI or an invalid answer in all but one\. These prompts request evidence\-grounded answers and remove the evidence, so the result is consistent with abstention\. It does not measure how much relevant knowledge the model stores\. Code\-mapping disagreement likewise identifies interface sensitivity without isolating its cause\.

## Appendix GExamples with Unchanged Answers

For FEVEROUS claim 236, “Mizmaar released a second album in 2007 which launched worldwide and two of the tracks became instant hits,” Qwen returnsSUPPORTSunder both sources in the greedy, 2,048\-token label condition\. The annotated group containsMizmaarsentences 4, 18, 19 and 20\. BM25 submits \[13, 4, 19, 18, 20\]; BGE submits \[4, 20, 18, 13\] andKashan Admanisentence 14\. BGE omits required sentence 19, giving per\-claim scores\(1,0,1,0\)\(1,0,1,0\)\. Claim 472 has unchanged valid answers and the reverse scores\(0,1,0,1\)\(0,1,0,1\)\. We selected the lowest\-ID eligible example in each direction post hoc\. Full evidence lists and responses accompany the records\.

Table 19:Twelve selected greedy examples\. Entries are changes in the credited numerator when scored evidence changes\. C/L denote codes/labels; Ex\./Ctr\. denote example/counterexample\.Table[19](https://arxiv.org/html/2609.27064#A7.T19)retains the twelve examples selected under the earlier fixed policy\. Each cell is a change in the credited numerator under greedy decoding when BM25 answers are scored with BGE instead of BM25 evidence\. Code\-to\-label contrasts subtract the two code entries from the two label entries; context contrasts subtract the two short\-budget entries from the two long\-budget entries\. The units are numerator counts, before conversion to corpus\-score points\. The supplement includes all traces and record hashes\.

## Appendix HSecondary Answer and Evidence Diagnostics

Table 20:Budget\-averaged score decomposition and the contribution from identical valid answers, in points\. Partition contributions use the full\-corpus denominator\.The unchanged\-answer contribution in Table[20](https://arxiv.org/html/2609.27064#A8.T20)uses the full\-corpus denominator\. This lets contributions from the partitions sum to the total evidence gain, including for corpus F1\. Negative answer components can offset positive evidence components\.

Table 21:Individual responses, including greedy\. Surface variants and unambiguous leading\-label explanations are accepted by the alternative parser; Remaining covers its residual failures\. The final column counts originally invalid responses with 16 tokens and overlaps the other categories\.Table 22:Invalid Llama responses truncated at the generation limit: 16 tokens with no terminal token \(IDs 128001, 128008 or 128009\)\. The percentage uses invalid responses as its denominator\. Both sampled\-only and including\-greedy results are shown\.The Llama configuration sets a 16\-token output limit and terminal IDs 128001, 128008 and 128009, without custom stopping criteria or strings\. It saves the generated token slice before removing special tokens\. A 16\-token output with no terminal token is therefore truncated at the generation limit\. Table[22](https://arxiv.org/html/2609.27064#A8.T22)identifies that subset, which is slightly smaller than the broad 16\-token count in Table[21](https://arxiv.org/html/2609.27064#A8.T21)\. Many responses already contain explanations that violate the exact\-label rule\. The saved records establish truncation, but whether longer completions would become valid or more accurate requires a different experiment\.

Table 23:Answer accuracy \(LL\), joint score with oracle\-correct answers \(OO\), and observed joint score \(SS\)\. Subscripts 0/1 denote BM25/BGE\. SciFact accuracy uses 900 claim–abstract pairs; itsOOandSSare corpus F1, including gold abstracts missed by retrieval\.Table 24:Direct paired interaction estimates with intervals adjusted over the 42\-comparison post\-hoc family\. The first column compares the two input\-budget components; the others compare evidence\-decomposition orders\. SciFact's input–scoring interaction is zero by construction because rationale eligibility is unchanged across budgets at fixed answers\.The secondary records include the original and corrected coverage diagnostics, parser sensitivity and crossed\-budget analysis\. The later 42\-comparison plan was recorded after earlier results were inspected and before its own calculations\. Original results, exact claim IDs, covariance terms and Monte Carlo batches are retained with their separate analysis plans\.

## Appendix IAlternative Parser and Invalid\-Response Rates

Table 25:Claim\-condition observations flagged for any invalid response, by answer format\. Each observation covers both sources and, for SciFact, all three abstracts\. Rates pool both budgets and six replicates\.Table[25](https://arxiv.org/html/2609.27064#A9.T25)flags a claim\-condition observation when either source’s answer is invalid\. For SciFact, any invalid answer among two sources and three abstracts triggers the flag\. The denominator includes both contexts and six replicates for each format\. A claim may therefore contribute several flagged observations\.

Qwen rarely fails the exact\-label format\. Llama’s failures vary with format: on FEVEROUS, 13\.97% of code observations and 8\.16% of label observations are flagged\. This interface behavior is part of the primary format comparison\.

Table 26:Original and alternative\-parser effects in score points\. The alternative parser accepts unambiguous variants without consulting gold labels and keeps every observation in the original population\.The alternative parser examines every response without consulting its gold label\. It normalizes case, spaces/underscores and surrounding punctuation, and accepts a leading label/code followed by an explanation only when there is one unambiguous interpretation\. Conflicting labels, uncertainty or negation markers, questions, empty strings and other malformed outputs remain invalid\. The FEVEROUS context estimates remain close to their original values\. Residual failures and mapping effects still limit a parser\-independent interpretation\.

## Appendix JEvidence\-Eligibility Correction

The original FEVEROUS diagnostic used raw page IDs and visible evidence\. The corrected version uses the scorer’s NFD\-normalized IDs and the submitted prefix\. Normalization changes visible\-group membership in 210 of 8,000 presentations\. In 497 presentations a complete normalized group is visible but falls outside the submitted prefix\.

Corrected evidence eligibility increases under BGE for 368 FEVEROUS claims and decreases for 159; the corresponding counts are 109/9 for FEVER and 14/9 for SciFact\. A gain means improvement at one or both budgets and deterioration at neither; a loss reverses the definition\. FEVER correct NEI answers require no evidence match, so their eligibility is one even when annotated group coverage is undefined\. SciFact compares the number of eligible retrieved abstracts per claim; abstracts without annotated rationales have undefined group coverage\.

The correction resolves 52 Qwen and 59 Llama claims whose strict successes conflicted with the old flags\. For example, claim 837 has a complete normalized submitted group at 2,048 tokens under both sources\. Re\-parsing and rescoring the raw responses reproduces all original counts, with zero strict successes lacking corrected eligibility\. Original diagnostic files and the correction are both retained; the primary effects and intervals are unchanged\.

## Appendix KSecondary Intervals and Monte Carlo Precision

The primary family uses 50,000 draws and tail probability0\.05/240\.05/24, about 104 expected draws per tail\. The 60\-comparison exploratory family uses tail probability0\.05/1200\.05/120: increasing its draws from 50,000 to 1,000,000 raises that count from about 21 to 417\. The separate 42\-comparison family uses0\.05/840\.05/84, about 595 draws per tail at 1,000,000 draws\. These counts describe numerical resolution of the percentile bounds, not additional experimental observations\. The original primary analysis is retained\.

The million\-draw results pool four independently seeded 250,000\-draw batches, specified before that calculation\. For Llama on FEVEROUS, theE1E\_\{1\}context lower bound is \-0\.075; batch lower bounds range from \-0\.079 to \-0\.065\. The bound is close to zero\. Table[27](https://arxiv.org/html/2609.27064#A11.T27)reports batch endpoints, while direct order comparisons appear in Table[24](https://arxiv.org/html/2609.27064#A8.T24)\.

Table 27:Lower and upper percentile endpoints for predetermined 250,000\-draw batches at the 60\-comparison tail probabilities\. The reported intervals pool all four batches\. All 102 secondary contrast records are retained in the supplement\.
## Appendix LComplete Crossed\-Budget Results

Table 28:All crossed\-budget decompositions with the 42\-comparison adjusted intervals\. The two rows for each panel change input and scoring budgets in opposite orders\. SciFact's scoring component is zero by construction because rationale eligibility and the F1 denominators are unchanged at fixed answers\.Table[28](https://arxiv.org/html/2609.27064#A12.T28)retains all datasets and both orders\. In SciFact, 48 of 1,800 source–abstract pairs change their submitted sentence lists between budgets, but none changes complete\-rationale eligibility\. With answers fixed, the credited\-abstract counts and F1 denominators therefore stay the same\. Its scoring components and input–scoring interactions are exactly zero for every resample; the zeros describe these submissions, rather than establishing a general absence of context effects\.

## Appendix MReproducibility

The supplement has separate trained\-verifier and LLM components\. The former reconstructs the retrospective override, checks eight original prediction files, and reproduces all 180 original Study 1 interval rows\. An additive post\-hoc script computes the five decomposition intervals, gold\-label contributions and covariance summaries reported here\. These replays use saved answers and scoring flags; original historical weights, model\-visible inputs and threshold\-selection records remain incomplete\.

The LLM component contains raw responses, token/evidence IDs, selected gold, scorer implementations, prompt\-building code, family assignments, scoring counts and the original analysis plans\. Fully instantiated prompt records and operational execution logs are archived privately; model weights and complete source corpora are excluded\. Existing counts and statistical records remain unchanged\. The supplement documents each executable check and the boundary between saved\-output scoring and reproducing model inference\.