Human-Anchored Factuality Evaluation with Strategic Annotation

# Human-Anchored Factuality Evaluation with Strategic Annotation
Source: [https://arxiv.org/html/2609.00494](https://arxiv.org/html/2609.00494)
###### Abstract

LLM\-based factuality judges provide scalable evaluation signals, but their metrics are often systematically biased relative to human judgments\. We study human\-anchored factuality evaluation under limited annotation budgets, where judge predictions on the full dataset are combined with human labels on a small selectively sampled subset to obtain statistically valid estimates\. The efficiency of this approach depends critically on which examples receive human annotation: in factuality evaluation, judge\-human misalignment is not driven solely by low confidence, but also by structured failure modes such as incomplete evidence, temporal mismatch, unverifiable claims, and rubric misalignment\. To exploit this structure, we introduce a factuality\-specific annotation policy design pipeline that uses failure\-space analysis \(FSA\) to derive diverse predictive signals for modeling human\-judge misalignment\. On an internal reference\-based factuality evaluation system \(AutoFA\) and RAGTruth, where judge\-predicted estimates substantially underestimate human\-annotated factual accuracy, our FSA\-guided policy improves annotation efficiency over uniform sampling and uncertainty\-driven baselines, achieving effective\-sample\-size gains of 40\.3% on AutoFA and 27\.1% on RAGTruth\.

22footnotetext:Now at Microsoft\.## 1Introduction

Factuality evaluation is a recurring concern when deploying large language models in user\-facing systems\. In production settings, teams rely on factual accuracy to measure system quality \(e\.g\., LLM model, prompts, grounding retrieval\), monitor regressions, and make release decisions from evaluation data that are both timely and statistically reliable\. Human annotation remains the most trusted way to measure factual accuracy, but it is expensive and slow\. Collecting sufficient labels to obtain narrow confidence intervals requires substantial annotation budget and introduces operational latency\. This creates a practical tension between evaluation quality and evaluation efficiency\.

LLM\-as\-a\-judge \(LLMaaJ\) systems offer an attractive way to scale factuality evaluation\([Min et al\., 2023](https://arxiv.org/html/2609.00494#bib.bib6);[Chen et al\., 2023](https://arxiv.org/html/2609.00494#bib.bib7);[Niu et al\., 2024](https://arxiv.org/html/2609.00494#bib.bib5);[Wei et al\., 2024](https://arxiv.org/html/2609.00494#bib.bib18);[Song et al\., 2024](https://arxiv.org/html/2609.00494#bib.bib17)\)\. Given a query, a response, and reference evidence, the judge can decompose the response into claims, verify them against evidence, and produce factuality labels at much larger scale than human annotators\. However, judge\-predicted factuality estimates are not necessarily human\-anchored\([Zheng et al\., 2023](https://arxiv.org/html/2609.00494#bib.bib10);[Wang et al\., 2024](https://arxiv.org/html/2609.00494#bib.bib11)\)\. In reference\-based factuality evaluation, judge errors are often systematic rather than random\. A judge may reject reasonable answers when references are incomplete, stale, conflicting, or indirectly supportive\. It may also apply a more literal verification rubric than human annotators\. As a result, treating judge predictions as ground truth can produce biased factual accuracy estimates, even when the judge is useful as a proxy signal\.

This paper studies factuality evaluation under a budgeted annotation setting\. Given a large set of evaluation items, an LLMaaJ provides proxy factuality predictions for all items, while only a small subset can receive human labels\. The goal is to estimate the human\-defined factual accuracy rate \(FAR\) with valid uncertainty quantification and lower variance than uniformly sampled human annotation alone\. We adopt Active Statistical Inference \(ASI\)\([Zrnic and Candes, 2024](https://arxiv.org/html/2609.00494#bib.bib3)\), which combines judge predictions with human corrections from a sampled subset of examples\. ASI preserves the human\-anchored target, but its efficiency depends critically on the annotation policy – human labels should be allocated to examples where the judge is most likely to disagree with humans\.

Factuality residuals are often structured rather than uniformly distributed across examples\. Existing confidence\-based approaches\([Gligorić et al\., 2025](https://arxiv.org/html/2609.00494#bib.bib15)\)use the judge’s expressed uncertainty as a sampling signal, but judge\-human misalignment is not driven solely by low confidence\. A judge can be confidently misaligned when evidence is incomplete, temporally inconsistent, or only indirectly supportive, and when responses require paraphrase, inference, or rubric tolerance that humans handle differently from the judge\. To exploit this structure, we construct the annotation policy using*failure\-space analysis*\(FSA\), a factuality\-specific policy\-design pipeline that derives predictive signals from judge\-output, evidence quality, task and input strata, and answer\-rubric alignment before converting them into ASI sampling policies\.

We evaluate the framework on two benchmarks: an internal Automatic Factual Evaluation system \(AutoFA\) for virtual assistant responses, and RAGTruth, a public hallucination benchmark with human span\-level annotations\. In both settings, judge\-predicted FAR substantially underestimates the human\-annotated FAR and adopting FSA\-derived policies preserve desired coverage while improving annotation efficiency over uniform sampling and confidence\-based sampling\. Aggregated across budgets, FSA\-guided ASI achieves an average effective sample size \(ESS\) gain of 40\.3% on AutoFA and 27\.1% on RAGTruth\.

To summarize, our contributions are threefold\. First, we formulate reference\-based factuality evaluation as a human\-anchored, judge\-assisted inference problem under limited annotation budget\. Second, we introduce failure\-space analysis as a practical policy\-design layer for identifying factuality\-specific judge\-human residual risk\. Third, we demonstrate on both internal and public factuality settings that FSA\-guided ASI recovers the human\-defined evaluation target while improving annotation efficiency over uniform and confidence\-based alternatives\. Together, these contributions enable reliable and cost\-efficient factuality measurement in practical evaluation pipelines, where fully manual annotation is infeasible and judge\-only metrics can be systematically biased\.

## 2Background

### 2\.1Problem Setup

We first formulate the problem of reference\-based factuality evaluation for short\-form query\-response pairs, while noting the proposed methodology can be naturally extended to long form settings\. Each evaluation item consists of a queryqiq\_\{i\}, a model responseaia\_\{i\}, and the reference evidenceeie\_\{i\}obtained from retrieval, curated sources, or an existing evaluation pipeline\. We write the observable input asXi=\{qi,ai,ei\}X\_\{i\}=\\\{q\_\{i\},a\_\{i\},e\_\{i\}\\\}and denote the human\-annotated factuality label of eachXiX\_\{i\}byYiY\_\{i\}\. In practice,YYis usually not a direct verdict, but verified atomic claims, or detected hallucinated spans\.

In this paper, we focus on*Factual Accuracy Rate*\(FAR\)111If the LLMaaJ’s evidence is the grounding provided to the answer generator, FAR measures faithfulness rate\., which is defined as:

θ=𝔼⁡\[ϕ⁡\(X,Y\)\],\\theta=\\mathbb\{E\}\[\\phi\(X,Y\)\],\(1\)whereϕ⁡\(Xi,Yi\)\\phi\(X\_\{i\},Y\_\{i\}\)denotes the factual correctness for theithi\_\{\\text\{th\}\}item\.

### 2\.2Judge\-Assisted Factual Evaluation under Budgeted Annotation

While FAR from human annotation is usually treated as the ground truth, acquiring human labels is expensive and slow, making it difficult to scale for drawing statistically significant conclusions\. On the other hand, Automatic Factual Evaluation can produce judge predictions at scale but the obtained FAR is generally biased\. This motivates us to study a budgeted, judge\-assisted factual evaluation problem where we combine the abundant judge predictions with limited human labels to form a statistically valid and more efficient estimator\.

GivenNNdata points, an LLMaaJ provides predictions, denoted byf⁡\(X\)f\(X\), for all items, while onlynbgt≪Nn\_\{\\text\{bgt\}\}\\ll Nitems can be sent for human annotation\. We adopt Active Statistical Inference \(ASI\)\([Zrnic and Candes, 2024](https://arxiv.org/html/2609.00494#bib.bib3)\), which selects the labeled subset from an annotation policy, denoted byπ⁡\(X\)\\pi\(X\)\.

Letξi∼Bernoulli​\(πi\)\\xi\_\{i\}\\sim\\text\{Bernoulli\}\(\\pi\_\{i\}\)indicate whether itemiireceives a human label with the inclusion probabilityπi\\pi\_\{i\}\. ASI estimates FAR by:

θ^ASI=1N​∑i=1N\[ϕif\+ξiπi​\(ϕih−ϕif\)\],\\hat\{\\theta\}^\{\\text\{ASI\}\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\[\\phi\_\{i\}^\{f\}\+\\frac\{\\xi\_\{i\}\}\{\\pi\_\{i\}\}\(\\phi\_\{i\}^\{h\}\-\\phi\_\{i\}^\{f\}\)\\right\],whereϕf\\phi^\{f\}andϕh\\phi^\{h\}denote the judge predicted and human annotated factual correctness respectively\.

Signal familyRepresentative signalsJudge\-output artifacts \(F1\)Claim/span structure, verification or hallucination fraction, unverifiable outputs, self\-reflection or confidence\.Evidence and reference quality \(F2\)Reference support strength, reference conflict, source freshness, direct vs\. indirect support\.Task and input strata \(F3\)Task or domain type, query category, time\-sensitive or location context indicators\.Answer\-rubric alignment \(F4\)Claim importance, completeness vs\. correctness, overprecision or auxiliary details, paraphrase/inference tolerance\.Table 1:Failure\-space signal families for residual\-risk policy design\. The table lists representative pre\-labeling signals rather than an exhaustive feature set\.
### 2\.3Statistical Efficiency

Drawing statistically meaningful conclusions requires not only targeting the correct human\-defined metric, but also estimating it with low variance under a limited annotation budget\. We compare three estimators that are central to this paper\. The*classical*estimator,θ^h=1nbgt∑i:ξi=1ϕih\\hat\{\\theta\}^\{h\}=\\frac\{1\}\{n\_\{\\text\{bgt\}\}\}\\sum\_\{i:\\xi\_\{i\}=1\}\\phi\_\{i\}^\{h\}, uses only thenbgtn\_\{\\text\{bgt\}\}human\-labeled examples sampled uniformly at random\. It is unbiased for the human\-defined FAR, but can be noisy under small annotation budget\. The*judge\-predicted*estimator,θ^f=1N​∑i=1Nϕif\\hat\{\\theta\}^\{f\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\phi\_\{i\}^\{f\}, uses judge predictions on allNNexamples and therefore has low variance, but it is biased when the judge is systematically misaligned with human\.

The ASI estimator remains human\-anchored\. However, its variance is policy dependent and scales with𝔼⁡\[Δ2​\(1/π⁡\(X\)−1\)\]\\mathbb\{E\}\[\\Delta^\{2\}\(1/\\pi\(X\)\-1\)\], whereΔ=ϕh−ϕf\\Delta=\\phi^\{h\}\-\\phi^\{f\}is the judge\-human residual\([Zrnic and Candes, 2024](https://arxiv.org/html/2609.00494#bib.bib3)\)\. Therefore, efficient annotation requires assigning higher sampling probabilities to examples with larger expected residuals, which motivates the failure\-space analysis in the next section\.

## 3FSA\-guided Policy Design

Efficient ASI requires allocating human labels to examples with large expected judge\-human residuals\. In the oracle case, the optimal policy is proportional to the conditional residual magnitude,

π⋆​\(X\)∝𝔼⁡\[Δ2∣X\]\.\\pi^\{\\star\}\(X\)\\propto\\sqrt\{\\mathbb\{E\}\[\\Delta^\{2\}\\mid X\]\}\.However, sinceϕh\\phi^\{h\}is unavailable before annotation, the practical policy\-design problem is to predict residual risk from pre\-labeling information: the queryqq, responseaa, reference evidenceee, and judge outputf⁡\(X\)f\(X\)\.

Directly modeling residual risk from raw text is difficult, and generic uncertainty is often insufficient for factuality evaluation\. We therefore introduce*failure\-space analysis*\(FSA\), a factuality\-specific procedure for deriving structured featuresZ⁡\(X,f⁡\(X\)\)Z\(X,f\(X\)\)that expose where judge\-human misalignment is likely to occur\. FSA organizes candidate policy signals into four families, summarized in Table[1](https://arxiv.org/html/2609.00494#S2.T1):*judge\-output artifacts*,*evidence and reference quality*,*task and input strata*, and*answer\-rubric alignment*\. Together, these signals capture not only whether the judge is uncertain, but also whether the evidence is incomplete or stale, whether the query belongs to a difficult slice, and whether the response requires inference or rubric tolerance that the judge may handle differently from humans\.

Operationally, FSA uses a historical labeled set where both judge outputs and human annotations are available\. We compute the residual target\|Δ\|\|\\Delta\|, extract numerical features from pre\-labeling artifacts, and train a policy model to predict residual risk\. The resulting scores are converted into sampling probabilities so that the annotation budget is concentrated on examples where the judge is most likely to disagree with humans\. A pipeline overview is summarized and illustrated in Figure[1](https://arxiv.org/html/2609.00494#S3.F1)\. We then provide a comprehensive demonstration of this pipeline in Section[4\.2](https://arxiv.org/html/2609.00494#S4.SS2)and Section[4\.3](https://arxiv.org/html/2609.00494#S4.SS3)\.

![Refer to caption](https://arxiv.org/html/2609.00494v1/figures/sys_diagram.png)Figure 1:Overview of the pipeline of human anchored factual evaluation\. \[1\]System Development: human experts design and iteratively optimize the LLM judge system, which takes the user query \(qq\), model answer \(aa\), retrieved evidence \(ee\) and predicts the factual correctness\. We consider this step as upstream and remains fixed once finished\. \[2\]Failure Space Analysis \(FSA\) on Calibration Data: historical labeled data is leveraged to analyze the judge failure modes spreading across four main families\. \[3\]Policy Learning: numerical features are extracted based on failure families and are selected with cross\-validation\. A regression model is trained to predict the risk of records based on the selected features, which is further converted to the sampling policy\. High risk records are assigned higher probability to be sampled for human annotation\. \[4\]ASI estimation: given new test dataset of sizeNN,nn\(labeling budget\) of them are sampled with the policy and sent for human annotations, while all data receives judge predictions\. The ASI estimator combines both judge predictions and human labels and produces human anchored factual accuracy rate \(FAR\) and confidence interval \(CI\)\.
## 4Experiments

### 4\.1Evaluation Settings

We study two factuality\-judge settings with different judge architectures, data formats, and judge output structures\.

#### Automatic Factual Accuracy \(AutoFA\)

AutoFA is an internal reference\-based factuality evaluation system for voice assistant responses\. Given a user query and model response, three reference answers are produced by search\-enabled LLMs\. The judge decomposes the response into verifiable claims, fact\-checks each claim against the references, and produces an overall factuality verdict, cf\. SAFE[Wei et al\. \(2024\)](https://arxiv.org/html/2609.00494#bib.bib18)andVeriScore[Song et al\. \(2024\)](https://arxiv.org/html/2609.00494#bib.bib17)\. Both claim\-level and response\-level outputs can betrue,false, orunknown\. We useN=2338N=2338responses and FAR is defined as the fraction of responses rated factually correct among examples with a definitive verdict\.

#### RAGTruth\.

RAGTruth\([Niu et al\., 2024](https://arxiv.org/html/2609.00494#bib.bib5)\)is a public hallucination benchmark with human span\-level annotations\. We useN=2937N=2937GPT\-4\-0613 responses across question answering, summarization, and data\-to\-text generation\. We evaluate Claude Sonnet 4\.6 as a faithfulness judge using the benchmark prompt, which asks the judge to identify hallucinated spans\. For RAGTruth, FAR is the faithfulness rate: the fraction of responses without hallucination\.

#### Self\-Confidence Score\.

Following prior work on confidence\-driven inference\([Gligorić et al\., 2025](https://arxiv.org/html/2609.00494#bib.bib15)\), we collect a self\-reflected confidence score for each record by prompting the judge to assess its confidence in its own previous verdict, yielding a scalar uncertainty signal in\[0,1\]\[0,1\]\.

### 4\.2FSA of Judge Residuals

We first compare judge FAR against human FAR to characterize the residuals that an annotation policy should target\. On AutoFA, the judge underestimates human FAR by13\.313\.3percentage points\. On RAGTruth, the gap is larger: the judge FAR is55\.6%55\.6\\%, compared with a human FAR of86\.2%86\.2\\%\.

#### Typical judge failures\.

In both datasets, judge\-human disagreements are dominated by judge over\-rejection\. On AutoFA,90%90\\%of disagreements occur when the judge marks human\-correct responses asfalse\(43%43\\%\) orunknown\(46%46\\%\)\. Manual inspection identifies two common mechanisms:*evidence gaps*, where references do not directly state a reasonable answer that humans accept, and*overly\-strict verification*, where the judge rejects an otherwise correct response due to unsupported peripheral details\. On RAGTruth,94%94\\%of disagreements occur when the judge flags hallucinations that humans do not annotate, mainly caused by*overly\-literal grounding*: the judge penalizes valid paraphrases or reasonable inferences because they are not explicitly stated in the reference\.

DatasetStratumPrev\.𝔼⁡\[Δ2\]\\mathbb\{E\}\[\\Delta^\{2\}\]LiftAutoFAPartial Verification0\.320\.412\.58Time\-sensitive Query0\.400\.201\.29RAGTruthLow Self\-confidence0\.320\.782\.24Judge flagged output0\.430\.702\.02Table 2:Residual concentration across observable strata\. Prevalence \(Prev\.\) is the fraction of records in the stratum\. Lift\>1\>1indicates above\-average residual density\.
#### Non\-uniform residual structure\.

Table[2](https://arxiv.org/html/2609.00494#S4.T2)shows that residuals concentrate in observable strata derived from judge outputs and task metadata\. We report each stratum’s prevalence, average squared residual𝔼⁡\[Δ2\]\\mathbb\{E\}\[\\Delta^\{2\}\], and*lift*, defined as the ratio between stratum\-level and population\-level𝔼⁡\[Δ2\]\\mathbb\{E\}\[\\Delta^\{2\}\]\. On AutoFA, partial verification, i\.e\., cases where at least one extracted claim is markedunknown, has2\.58×2\.58\\timeslift, while time\-sensitive queries have1\.29×1\.29\\timeslift\. On RAGTruth, low self\-confidence has2\.24×2\.24\\timeslift, and judge\-flagged outputs, i\.e\., cases where the judge returns one or more hallucination spans rather than an empty list, have2\.02×2\.02\\timeslift\. These concentrations show that judge\-human residuals are structured rather than uniformly distributed, motivating the FSA\-derived features in Section[4\.3](https://arxiv.org/html/2609.00494#S4.SS3)\.

FamilyFeatureSpearman\|ρ\|\|\\rho\|AutoFARAGTruthJudgeself\-confidence0\.310\.310\.66\\mathbf\{0\.66\}flag\-reliability–0\.25†0\.25^\{\\dagger\}Evidenceref\-support\-strength0\.57\\mathbf\{0\.57\}0\.220\.22ref\-temporal\-consistency0\.360\.36–Stratatime\-sensitive / task0\.320\.320\.180\.18Rubricinference\-required0\.440\.440\.160\.16qualitative\-language–0\.170\.17Table 3:Feature ranking by FSA family on absolute Spearman correlation \(\|ρ\|\|\\rho\|\) with\|Δ\|\|\\Delta\|\.†Computed on the judge\-flagged subset\.

### 4\.3Residual\-Risk Feature Modeling

Guided by Section[3](https://arxiv.org/html/2609.00494#S3)and the residual patterns in Section[4\.2](https://arxiv.org/html/2609.00494#S4.SS2), we instantiate dataset\-specific features from the artifacts available before human annotation\. For both datasets, we use the judge’s self\-confidence score as a generic uncertainty signal\.

For RAGTruth, where the judge may over\-detect hallucinated spans, we additionally define*flag\-reliability*to assess whether the judge\-flagged spans are plausibly unsupported by the reference\.

For AutoFA, the dominant residuals are tied to reference quality, so we extract*ref\-support\-strength*, which measures how directly the references support the response’s core claims, and*ref\-temporal\-consistency*, which measures whether search\-generated references agree on time\-dependent facts\. We also include task\-level strata, such as time\-sensitive queries in AutoFA and task type in RAGTruth\.

![Refer to caption](https://arxiv.org/html/2609.00494v1/figures/ci_simulation_2x3.png)Figure 2:Coverage, CI width, and effective sample size across annotation budgets\. FSA ASI maintains valid coverage while achieving the largest effective sample size \(ESS\) across both datasets\.Finally, we extract rubric\-alignment features, including*inference\-required*, which measures whether verification requires reasoning beyond literal evidence, and*qualitative\-language*, which captures interpretive or subjective phrasing\. LLM\-extracted features are produced with structured prompts that return integer scores on a 0–3 scale\. These feature\-extraction calls are applied before human annotation and are substantially cheaper than human labeling in our setting\. We provide more details in Appendix[E](https://arxiv.org/html/2609.00494#A5)\.

Table[3](https://arxiv.org/html/2609.00494#S4.T3)reports the absolute Spearman correlation between each feature and the residual magnitude\|Δ\|\|\\Delta\|\. This provides a univariate diagnostic of whether each feature monotonically tracks residual risk\. On AutoFA,*ref\-support\-strength*has the strongest correlation, consistent with evidence sufficiency being the dominant failure mode\. On RAGTruth,*self\-confidence*dominates, while*flag\-reliability*provides an additional signal on the judge\-flagged subset\.

The correlations in Table[3](https://arxiv.org/html/2609.00494#S4.T3)are diagnostic rather than the final policy criterion\. A feature may correlate with\|Δ\|\|\\Delta\|but still be less useful for sampling if it is sparse, redundant, or unstable under limited calibration data\. We therefore split the calibration set into an inner training split for fitting residual\-risk models and an inner validation split for comparing candidate feature configurations\. This procedure selects*ref\-support\-strength*for AutoFA, and*self\-confidence*together with*flag\-reliability*for RAGTruth, reflecting the different residual structures of the two factuality settings\.

### 4\.4Debiasing and Efficiency Evaluation

We evaluate the method along two dimensions: whether ASI recovers the human\-anchored FAR despite judge bias, and whether FSA\-derived policies improve annotation efficiency\.

#### Policy learning\.

Each dataset is split 40/60 into calibration and evaluation sets\. On the calibration set, we train a gradient\-boosted tree \(XGBoost\) to predict\|Δ\|\|\\Delta\|from the selected features, producing a residual\-risk score for each record\. The scores are converted into sampling probabilities by normalization with uniform mixing\([Li et al\., 2025](https://arxiv.org/html/2609.00494#bib.bib4);[Zrnic and Candes, 2024](https://arxiv.org/html/2609.00494#bib.bib3)\), and we apply power\-tuning\([Angelopoulos et al\., 2024](https://arxiv.org/html/2609.00494#bib.bib2)\)to set the judge contribution in the ASI estimator\. Uniform\-mixing and power\-tuning parameters are estimated on the calibration set, with details in Appendix[A](https://arxiv.org/html/2609.00494#A1)\. All reported results are evaluated on the evaluation split\.

#### Estimators for comparison\.

We compare five estimators\.*Classical*uniformly samplesnbgtn\_\{\\text\{bgt\}\}human labels and estimates FAR with only human annotations\.*Judge\-predicted*reports the judge FAR over allNNrecords without human correction\.*Uniform ASI*applies the power\-tuned ASI correction under a uniform policy\.*Confidence ASI*uses only self\-confidence as the sampling signal\.*FSA ASI*uses the FSA\-selected features from Section[4\.3](https://arxiv.org/html/2609.00494#S4.SS3)\. The ASI variants differ only in how they allocate the human\-labeling budget\.

DatasetPolicyCov\.ESS Gain \(%\)AutoFAClassical0\.94±0\.010\.94\{\\scriptstyle\\pm 0\.01\}—Judge\-predicted0\.000\.00—Uniform ASI0\.95±0\.010\.95\{\\scriptstyle\\pm 0\.01\}\+13\.3±4\.1\+13\.3\{\\scriptstyle\\pm 4\.1\}Confidence ASI0\.94±0\.010\.94\{\\scriptstyle\\pm 0\.01\}\+17\.0±4\.3\+17\.0\{\\scriptstyle\\pm 4\.3\}FSA ASI0\.95±0\.010\.95\{\\scriptstyle\\pm 0\.01\}\+40\.3±9\.1\\mathbf\{\+40\.3\}\{\\scriptstyle\\pm 9\.1\}RAGTruthClassical0\.94±0\.010\.94\{\\scriptstyle\\pm 0\.01\}—Judge\-predicted0\.000\.00—Uniform ASI0\.94±0\.010\.94\{\\scriptstyle\\pm 0\.01\}\+11\.3±1\.8\+11\.3\{\\scriptstyle\\pm 1\.8\}Confidence ASI0\.94±0\.000\.94\{\\scriptstyle\\pm 0\.00\}\+22\.6±2\.3\+22\.6\{\\scriptstyle\\pm 2\.3\}FSA ASI0\.94±0\.000\.94\{\\scriptstyle\\pm 0\.00\}\+27\.1±2\.7\\mathbf\{\+27\.1\}\{\\scriptstyle\\pm 2\.7\}Table 4:CI simulation results aggregated across budgetsnbgt∈\{50,100,…,1000\}n\_\{\\mathrm\{bgt\}\}\\in\\\{50,100,\\ldots,1000\\\}\(α=0\.05\\alpha=0\.05, 300 trials per budget\)\. Values report mean±\\pmstandard deviation across budgets\.
#### Metrics and simulation\.

We report coverage, confidence interval width, and effective sample size \(ESS\)\. Coverage is the fraction of confidence intervals containing the human FAR computed on the full evaluation set\. ESS measures efficiency on the scale of uniform annotation: an ESS gain of\+40%\+40\\%means thatnnpolicy\-sampled labels achieve the same variance as1\.4​n1\.4nuniformly sampled labels\. For each budgetnbgt∈\{50,100,…,1000\}n\_\{\\mathrm\{bgt\}\}\\in\\\{50,100,\\ldots,1000\\\}, we run 300 Monte Carlo trials on the evaluation split\. In each trial, we bootstrap the evaluation set, sample human labels according to the corresponding policy, compute the estimator, and construct a Wald confidence interval atα=0\.05\\alpha=0\.05\.

#### Results\.

Table[4](https://arxiv.org/html/2609.00494#S4.T4)and Figure[2](https://arxiv.org/html/2609.00494#S4.F2)summarize the simulation results\. The judge\-predicted estimator has zero coverage on both datasets, confirming that judge FAR is biased relative to the human\-anchored target\. In contrast, all ASI variants maintain near\-nominal coverage across budgets, showing that human\-labeled correction successfully debiases the judge estimate\.

Among the valid estimators, all ASI variants improve efficiency over the classical baseline\. Uniform ASI already provides ESS gains of\+13\.3%\+13\.3\\%on AutoFA and\+11\.3%\+11\.3\\%on RAGTruth while Confidence ASI further improves efficiency to\+17\.0%\+17\.0\\%and\+22\.6%\+22\.6\\%, respectively, by allocating more labels to low\-confidence examples\.

FSA ASI achieves the largest gains, improving ESS by\+40\.3%\+40\.3\\%on AutoFA and\+27\.1%\+27\.1\\%on RAGTruth when aggregated across budgets\. The larger gap over Confidence ASI on AutoFA reflects that self\-confidence is a weaker residual\-risk signal in this setting, while FSA identifies reference support strength as a more informative evidence\-quality feature\. On RAGTruth, the smaller gap is consistent with self\-confidence already capturing much of the residual structure, with flag\-reliability providing an additional gain\.

## 5Related Work

Factuality evaluation is central to assessing LLM outputs when correctness must be verified against retrieved evidence, source documents, or external references\. Prior work has developed automatic factuality and hallucination evaluation methods that decompose responses into claims, atomic facts, or hallucinated spans, and assess whether these units are supported by evidence\([Min et al\., 2023](https://arxiv.org/html/2609.00494#bib.bib6);[Chen et al\., 2023](https://arxiv.org/html/2609.00494#bib.bib7);[Niu et al\., 2024](https://arxiv.org/html/2609.00494#bib.bib5);[Iqbal et al\., 2024](https://arxiv.org/html/2609.00494#bib.bib8);[Song et al\., 2024](https://arxiv.org/html/2609.00494#bib.bib17);[Wei et al\., 2024](https://arxiv.org/html/2609.00494#bib.bib18)\)\. These methods improve scalability, but LLM judges can systematically diverge from human labels due to incomplete evidence, over\-literal grounding, rubric mismatch, and other biases in the judging procedure\([Zheng et al\., 2023](https://arxiv.org/html/2609.00494#bib.bib10);[Wang et al\., 2024](https://arxiv.org/html/2609.00494#bib.bib11);[Gu et al\., 2025](https://arxiv.org/html/2609.00494#bib.bib12)\)\.

Another line of works including Prediction\-Powered Inference and its variants studies how to combine abundant machine predictions with limited human labels while preserving statistically valid inference\([Angelopoulos et al\., 2023](https://arxiv.org/html/2609.00494#bib.bib1);[Angelopoulos et al\., 2024](https://arxiv.org/html/2609.00494#bib.bib2)\)\. Active Statistical Inference extends this idea to budgeted annotation, where examples are sampled according to a policy designed to reduce estimator variance\([Zrnic and Candes, 2024](https://arxiv.org/html/2609.00494#bib.bib3)\)\. Recent work adapts these ideas to LLM evaluation, showing how LLMaaJ metrics can be reported with valid uncertainty quantification\([Gligorić et al\., 2025](https://arxiv.org/html/2609.00494#bib.bib15);[Wu et al\., 2026](https://arxiv.org/html/2609.00494#bib.bib13);[Lee et al\., 2025](https://arxiv.org/html/2609.00494#bib.bib14)\)\. Cost optimal AI evaluation\([Angelopoulos et al\., 2025](https://arxiv.org/html/2609.00494#bib.bib19)\)extends ASI by incorporating the cost of human annotator and LLMaaJ into the optimization objective to derive cost\-aware annotation policy\. Meanwhile, MultiPPI\([Cowen\-Breen et al\., 2026](https://arxiv.org/html/2609.00494#bib.bib20)\)and AM\-PPI\([Brawand et al\., 2026](https://arxiv.org/html/2609.00494#bib.bib21)\)explore settings where multiple predictors are available and how to route between predictors to balance the cost\-performance trade\-off\.

Closest to our work, confidence\-driven inference\([Gligorić et al\., 2025](https://arxiv.org/html/2609.00494#bib.bib15)\)uses verbalized LLM confidence to guide human annotation, which provides a strong generic sampling signal\. However, factuality residuals are not always explained by uncertainty alone: judges can be confidently misaligned when evidence is incomplete or only indirectly supportive\. Our work adds a factuality\-specific policy\-design layer: FSA derives residual\-risk features from judge artifacts, evidence quality, task strata, and rubric alignment, improving annotation efficiency while preserving the human\-anchored ASI target\.

## 6Limitations

Our proposed framework is effective but is constrained by a few limitations:

First, FSA\-guided ASI relies on residual structure learned from historical or calibration data\. Its efficiency depends on this structure remaining stable across the evaluation population, i\.e\., model updates, retrieval or grounding changes, shifts in user queries, or new failure modes can degrade the policy\. Silent high\-confidence errors that are not captured under distribution shift will be under\-sampled by the policy, thus inflate inverse\-probability weights and increase variance\. Uniform mixing can mitigate but does not remove the need for distribution\-shift monitoring and periodic policy recalibration in production deployment\. Specifically, a small uniformly sampled audit stream can test whether recent LLM–human residuals remain concentrated in the regions predicted by the policy\. If this relationship degrades, the system can increase uniform mixing or revert temporarily to Uniform ASI, while acquiring additional human annotations to recalibrate the policy\.

Second, while the four failure families provide useful pre\-annotation signals for predicting disagreement in the studied settings, they do not exhaustively cover all error categories\. Rare but complex cases, for example, failures in multi\-hop reasoning across multiple pieces of evidence, may still occur and may not be captured by FSA\. Depending on the application setting, the FSA rubric may therefore need to be further adapted or extended\.

Third, our formulation also simplifies the annotation process\. We treat the final adjudicated human label as the target and do not explicitly model annotator\-level noise or disagreement\. We also focus on a single human\-annotator rather than a multi\-annotator setting with different costs and reliabilities, such as cheap annotators, expert annotators, or multiple judge models\. Extending FSA\-guided ASI to jointly allocate budget across heterogeneous annotators is an important direction for future work\.

Finally, ASI improves efficiency for a given annotation budget, but it does not determine the budget required for a particular production decision\. In practice, the choice ofnbgtn\_\{\\text\{bgt\}\}depends on cost, latency, acceptable uncertainty, and release\-risk tolerance\. Our results quantify the variance reduction achieved by FSA\-guided sampling, while budget selection itself remains an operational decision\.

## References

- Angelopouloset al\.\(2023\)A\. N\. Angelopoulos, S\. Bates, C\. Fannjiang, M\. I\. Jordan, and T\. ZrnicPrediction\-Powered Inference\.Science382\(6671\),pp\. 669–674\.External Links:[Document](https://dx.doi.org/10.1126/science.adi6000),[Link](https://www.science.org/doi/abs/10.1126/science.adi6000),https://www\.science\.org/doi/pdf/10\.1126/science\.adi6000Cited by:[§5](https://arxiv.org/html/2609.00494#S5.p2.1)\.
- Angelopouloset al\.\(2024\)A\. N\. Angelopoulos, J\. C\. Duchi, and T\. ZrnicPPI\+\+: Efficient Prediction\-Powered Inference\.External Links:2311\.01453,[Link](https://arxiv.org/abs/2311.01453)Cited by:[Appendix A](https://arxiv.org/html/2609.00494#A1.SS0.SSS0.Px5.p1.1),[§4\.4](https://arxiv.org/html/2609.00494#S4.SS4.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.00494#S5.p2.1)\.
- Angelopouloset al\.\(2025\)A\. N\. Angelopoulos, J\. Eisenstein, J\. Berant, A\. Agarwal, and A\. FischCost\-Optimal Active AI Model Evaluation\.External Links:2506\.07949,[Link](https://arxiv.org/abs/2506.07949)Cited by:[§5](https://arxiv.org/html/2609.00494#S5.p2.1)\.
- Brawandet al\.\(2026\)N\. Brawand, N\. Leclerc, A\. Ngo, M\. Peterson, S\. Vishwanath, L\. Alhussein, and B\. WellnerActive Multiple\-Prediction\-Powered Inference\.External Links:2605\.08429,[Link](https://arxiv.org/abs/2605.08429)Cited by:[§5](https://arxiv.org/html/2609.00494#S5.p2.1)\.
- Chenet al\.\(2023\)S\. Chen, Y\. Zhao, J\. Zhang, I\. Chern, S\. Gao, P\. Liu, and J\. HeFELM: Benchmarking Factuality Evaluation of Large Language Models\.InThirty\-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](http://arxiv.org/abs/2310.00741)Cited by:[§1](https://arxiv.org/html/2609.00494#S1.p2.1),[§5](https://arxiv.org/html/2609.00494#S5.p1.1)\.
- Chen and Guestrin \(2016\)T\. Chen and C\. GuestrinXGBoost: A Scalable Tree Boosting System\.InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,KDD ’16,pp\. 785–794\.External Links:[Link](http://dx.doi.org/10.1145/2939672.2939785),[Document](https://dx.doi.org/10.1145/2939672.2939785)Cited by:[Appendix A](https://arxiv.org/html/2609.00494#A1.SS0.SSS0.Px2.p1.1)\.
- Cowen\-Breenet al\.\(2026\)C\. Cowen\-Breen, A\. Agarwal, S\. Bates, W\. W\. Cohen, J\. Eisenstein, A\. Globerson, and A\. FischMultiple\-Prediction\-Powered Inference\.External Links:2603\.27414,[Link](https://arxiv.org/abs/2603.27414)Cited by:[§5](https://arxiv.org/html/2609.00494#S5.p2.1)\.
- Gligorićet al\.\(2025\)K\. Gligorić, T\. Zrnic, C\. Lee, E\. Candès, and D\. JurafskyCan Unconfident LLM Annotations Be Used for Confident Conclusions?\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 3514–3533\.External Links:[Link](https://aclanthology.org/2025.naacl-long.179/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.179),ISBN 979\-8\-89176\-189\-6Cited by:[§1](https://arxiv.org/html/2609.00494#S1.p4.1),[§4\.1](https://arxiv.org/html/2609.00494#S4.SS1.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2609.00494#S5.p2.1),[§5](https://arxiv.org/html/2609.00494#S5.p3.1)\.
- Guet al\.\(2025\)J\. Gu, X\. Jiang, Z\. Shi, H\. Tan, X\. Zhai, C\. Xu, W\. Li, Y\. Shen, S\. Ma, H\. Liu, S\. Wang, K\. Zhang, Y\. Wang, W\. Gao, L\. Ni, and J\. GuoA Survey on LLM\-as\-a\-Judge\.External Links:2411\.15594,[Link](https://arxiv.org/abs/2411.15594)Cited by:[§5](https://arxiv.org/html/2609.00494#S5.p1.1)\.
- Iqbalet al\.\(2024\)H\. Iqbal, Y\. Wang, M\. Wang, G\. N\. Georgiev, J\. Geng, I\. Gurevych, and P\. NakovOpenFactCheck: a Unified Framework for Factuality Evaluation of LLMs\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,D\. I\. Hernandez Farias, T\. Hope, and M\. Li \(Eds\.\),Miami, Florida, USA,pp\. 219–229\.External Links:[Link](https://aclanthology.org/2024.emnlp-demo.23/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-demo.23)Cited by:[§5](https://arxiv.org/html/2609.00494#S5.p1.1)\.
- Leeet al\.\(2025\)C\. Lee, T\. Zeng, J\. Jeong, J\. Sohn, and K\. LeeHow to Correctly Report LLM\-as\-a\-Judge Evaluations\.External Links:2511\.21140Cited by:[§5](https://arxiv.org/html/2609.00494#S5.p2.1)\.
- Liet al\.\(2025\)P\. Li, T\. Zrnic, and E\. CandèsRobust Sampling for Active Statistical Inference\.External Links:2511\.08991,[Link](https://arxiv.org/abs/2511.08991)Cited by:[Appendix A](https://arxiv.org/html/2609.00494#A1.SS0.SSS0.Px3.p1.2),[§4\.4](https://arxiv.org/html/2609.00494#S4.SS4.SSS0.Px1.p1.1)\.
- Lundberg and Lee \(2017\)S\. M\. Lundberg and S\. LeeA unified approach to interpreting model predictions\.InProceedings of the 31st International Conference on Neural Information Processing Systems,NIPS’17,Red Hook, NY, USA,pp\. 4768–4777\.External Links:ISBN 9781510860964Cited by:[Appendix B](https://arxiv.org/html/2609.00494#A2.SS0.SSS0.Px2.p1.1)\.
- Minet al\.\(2023\)S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\. Yih, P\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. HajishirziFActScore: Fine\-grained Atomic Evaluation of Factual Precision in Long Form Text Generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 12076–12100\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.741/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.741)Cited by:[§1](https://arxiv.org/html/2609.00494#S1.p2.1),[§5](https://arxiv.org/html/2609.00494#S5.p1.1)\.
- Niuet al\.\(2024\)C\. Niu, Y\. Wu, J\. Zhu, S\. Xu, K\. Shum, R\. Zhong, J\. Song, and T\. ZhangRAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval\-Augmented Language Models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 10862–10878\.External Links:[Link](https://aclanthology.org/2024.acl-long.585/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.585)Cited by:[Appendix C](https://arxiv.org/html/2609.00494#A3.p1.1),[§1](https://arxiv.org/html/2609.00494#S1.p2.1),[§4\.1](https://arxiv.org/html/2609.00494#S4.SS1.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2609.00494#S5.p1.1)\.
- Songet al\.\(2024\)Y\. Song, Y\. Kim, and M\. IyyerVeriScore: evaluating the Factuality of Verifiable Claims in Long\-form Text Generation\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 9447–9474\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.552/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.552)Cited by:[§1](https://arxiv.org/html/2609.00494#S1.p2.1),[§4\.1](https://arxiv.org/html/2609.00494#S4.SS1.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.00494#S5.p1.1)\.
- Wanget al\.\(2024\)P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, L\. Kong, Q\. Liu, T\. Liu, and Z\. SuiLarge Language Models are not Fair Evaluators\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 9440–9450\.External Links:[Link](https://aclanthology.org/2024.acl-long.511/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.511)Cited by:[§1](https://arxiv.org/html/2609.00494#S1.p2.1),[§5](https://arxiv.org/html/2609.00494#S5.p1.1)\.
- Weiet al\.\(2024\)J\. Wei, C\. Yang, X\. Song, Y\. Lu, N\. Hu, J\. Huang, D\. Tran, D\. Peng, R\. Liu, D\. Huang, C\. Du, and Q\. V\. LeLong\-form Factuality in Large Language Models\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[§1](https://arxiv.org/html/2609.00494#S1.p2.1),[§4\.1](https://arxiv.org/html/2609.00494#S4.SS1.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.00494#S5.p1.1)\.
- Wuet al\.\(2026\)S\. Wu, Y\. Nair, and E\. J\. CandèsEfficient Evaluation of LLM Performance with Statistical Guarantees\.External Links:2601\.20251,[Link](https://arxiv.org/abs/2601.20251)Cited by:[§5](https://arxiv.org/html/2609.00494#S5.p2.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-bench and Chatbot Arena\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23,Red Hook, NY, USA\.Cited by:[§1](https://arxiv.org/html/2609.00494#S1.p2.1),[§5](https://arxiv.org/html/2609.00494#S5.p1.1)\.
- Zrnic and Candes \(2024\)T\. Zrnic and E\. CandesActive Statistical Inference\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 62993–63010\.External Links:[Link](https://proceedings.mlr.press/v235/zrnic24a.html)Cited by:[§1](https://arxiv.org/html/2609.00494#S1.p3.1),[§2\.2](https://arxiv.org/html/2609.00494#S2.SS2.p2.1),[§2\.3](https://arxiv.org/html/2609.00494#S2.SS3.p2.1),[§4\.4](https://arxiv.org/html/2609.00494#S4.SS4.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.00494#S5.p2.1)\.

## Appendix APolicy Learning Details

This appendix provides experimental details on implementing FSA\-policy and running simulations\.

#### Regression target\.

The policy learning objective is to predict the residual magnitude\|Δi\|=\|ϕih−ϕif\|\|\\Delta\_\{i\}\|=\|\\phi^\{h\}\_\{i\}\-\\phi^\{f\}\_\{i\}\|from pre\-labeling features\. On both datasets, this target is binary:\|Δ\|=1\|\\Delta\|=1when the judge and human disagree, and\|Δ\|=0\|\\Delta\|=0when they agree\. Figure[3](https://arxiv.org/html/2609.00494#A1.F3)shows the class distribution\. The disagreement rate is28\.2%28\.2\\%on AutoFA and34\.7%34\.7\\%on RAGTruth\.

![Refer to caption](https://arxiv.org/html/2609.00494v1/figures/appendix_target_distribution.png)Figure 3:Distribution of the policy learning target\|Δ\|=\|ϕh−ϕf\|\|\\Delta\|=\|\\phi^\{h\}\-\\phi^\{f\}\|\. Both datasets have an imbalanced binary target, with agreement \(green\) as the majority class\.
#### Score model\.

We train an XGBoost regressor\([Chen and Guestrin, 2016](https://arxiv.org/html/2609.00494#bib.bib16)\)withobjective=reg:squarederrorto predict\|Δ\|\|\\Delta\|from the selected features\. On a binary target, this is equivalent to learningP⁡\(Δ≠0∣X\)P\(\\Delta\\neq 0\\mid X\), producing a residual\-risk score in\[0,1\]\[0,1\]\. We use 200 boosting rounds, learning rate 0\.05, and maximum tree depth 3\. The model is trained on the 40% calibration split and its output serves as the raw scores⁡\(Xi\)s\(X\_\{i\}\)for policy construction\.

#### Score normalization and uniform mixing\.

Following the oracle ASI policy in Section 3, we first transform the predicted residual risk asr⁡\(Xi\)=s⁡\(Xi\)r\(X\_\{i\}\)=\\sqrt\{s\(X\_\{i\}\)\}, and then convert it into inclusion probabilities:

πiraw=r⁡\(Xi\)⋅nbgt∑jr⁡\(Xj\),\\pi\_\{i\}^\{\\text\{raw\}\}=r\(X\_\{i\}\)\\cdot\\frac\{n\_\{\\text\{bgt\}\}\}\{\\sum\_\{j\}r\(X\_\{j\}\)\},with water\-filling to ensureπi≤1\\pi\_\{i\}\\leq 1\. To prevent extreme weights that destabilize the ASI variance estimate, we also apply uniform mixing\([Li et al\., 2025](https://arxiv.org/html/2609.00494#bib.bib4)\):

πi=\(1−τ\)⋅πiraw\+τ⋅nbgtN,\\pi\_\{i\}=\(1\-\\tau\)\\cdot\\pi\_\{i\}^\{\\text\{raw\}\}\+\\tau\\cdot\\frac\{n\_\{\\text\{bgt\}\}\}\{N\},whereτ∈\[0,1\]\\tau\\in\[0,1\]interpolates between the learned policy \(τ=0\\tau=0\) and uniform sampling \(τ=1\\tau=1\)\. Largerτ\\tauimproves stability at the cost of policy concentration\.

#### Mixing coefficient selection\.

We selectτ\\tauby grid search over\{0\.05,0\.10,…,1\.0\}\\\{0\.05,0\.10,\\ldots,1\.0\\\}, evaluating the ASI variance on the calibration split for each candidate\. Figure[4](https://arxiv.org/html/2609.00494#A1.F4)shows that both datasets exhibit a concave ESS\-gain curve: too\-smallτ\\taucreates extreme inverse\-probability weights that inflate variance, while too\-largeτ\\taudilutes the policy signal\. The selected values areτ=0\.40\\tau=0\.40for AutoFA andτ=0\.55\\tau=0\.55for RAGTruth\. The higherτ\\tauon RAGTruth reflects its larger𝔼⁡\[Δ2\]/Var⁡\(ϕh\)\\mathbb\{E\}\[\\Delta^\{2\}\]/\\mathrm\{Var\}\(\\phi^\{h\}\)ratio, which requires a larger mixing coefficient for stability\.

![Refer to caption](https://arxiv.org/html/2609.00494v1/figures/appendix_tau_search.png)Figure 4:Uniform mixing coefficient \(τ\\tau\) search\. ESS gain is evaluated on the calibration split atnbgt=300n\_\{\\text\{bgt\}\}=300\. Both curves are concave: smallτ\\tauoverfits, largeτ\\tauunder\-concentrates\. Red dotted line marks the selectedτ∗\\tau^\{\*\}\.
#### Power tuning\.

The ASI estimator uses a power\-tuning coefficientλ∈\[0,1\]\\lambda\\in\[0,1\]that controls how strongly the judge contribution enters the correction\([Angelopoulos et al\., 2024](https://arxiv.org/html/2609.00494#bib.bib2)\)\. Atλ=1\\lambda=1, the estimator uses the full judge signal; atλ=0\\lambda=0, it ignores the judge entirely\. Power\-tuning selectsλ\\lambdato minimize the policy\-dependent variance term:

J⁡\(λ\)=𝔼⁡\[\(ϕh−λ​ϕf\)2⋅\(1π⁡\(X\)−1\)\]\.J\(\\lambda\)=\\mathbb\{E\}\\\!\\left\[\(\\phi^\{h\}\-\\lambda\\phi^\{f\}\)^\{2\}\\cdot\\left\(\\frac\{1\}\{\\pi\(X\)\}\-1\\right\)\\right\]\.SinceJ⁡\(λ\)J\(\\lambda\)is quadratic inλ\\lambda, the optimum has a closed\-form solution:

λ∗=𝔼⁡\[ϕh​ϕf⋅w\]𝔼⁡\[\(ϕf\)2⋅w\],\\lambda^\{\*\}=\\frac\{\\mathbb\{E\}\\\!\\left\[\\phi^\{h\}\\phi^\{f\}\\cdot w\\right\]\}\{\\mathbb\{E\}\\\!\\left\[\(\\phi^\{f\}\)^\{2\}\\cdot w\\right\]\},wherewi=1/π⁡\(Xi\)−1w\_\{i\}=1/\\pi\(X\_\{i\}\)\-1\. In practice, we estimateλ∗\\lambda^\{\*\}from the calibration split using sample averages\. BecauseJ⁡\(λ\)J\(\\lambda\)is convex, the tuned estimator always has variance no greater than the untuned \(λ=1\\lambda=1\) estimator\.

Figure[5](https://arxiv.org/html/2609.00494#A1.F5)illustrates the effect on AutoFA: without power\-tuning \(λ=1\\lambda=1\), the ASI variance exceeds the labeled baseline at small budgets because the judge residual is large\. Power\-tuning \(λ∗≈0\.2\\lambda^\{\*\}\\approx 0\.2\) shrinks the judge contribution to the level most appropriate for the residual structure, consistently reducing variance below the baseline\.

![Refer to caption](https://arxiv.org/html/2609.00494v1/figures/appendix_power_tuning.png)Figure 5:Effect of power\-tuning on AutoFA\. Left: variance comparison showing thatλ=1\\lambda=1\(untuned\) can be worse than baseline, whileλ∗\\lambda^\{\*\}\(tuned\) consistently improves\. Right: the optimalλ∗≈0\.2\\lambda^\{\*\}\\approx 0\.2is stable across budgets, indicating the judge should be used at∼\\sim20% strength\.

## Appendix BFeature Selection Details

This appendix describes the feature selection procedure and provides post\-hoc validation\.

#### Selection procedure\.

Feature candidates are derived from the qualitative FSA in Section[4\.2](https://arxiv.org/html/2609.00494#S4.SS2)\. We evaluate each candidate’s residual\-ranking quality via 5\-fold cross\-validated Spearman correlation on the calibration set\. We then select the final feature set using two criteria: \(1\) high CV Spearman, and \(2\) parsimony: we prefer a smaller feature set because, with onlyN≈1000N\\approx 1000calibration samples, using many features can introduce spurious correlations and cause the score model to overfit\.

#### Joint feature importance\.

Figure[6](https://arxiv.org/html/2609.00494#A2.F6)shows SHAP values\([Lundberg and Lee, 2017](https://arxiv.org/html/2609.00494#bib.bib9)\)for the score model trained on all candidate features, confirming which features drive the model’s predictions\. On AutoFA, ref\-support\-strength dominates\. On RAGTruth, self\-confidence and flag\-reliability jointly dominate\. Features from other families contribute marginally\.

![Refer to caption](https://arxiv.org/html/2609.00494v1/figures/shap_autofa.png)

![Refer to caption](https://arxiv.org/html/2609.00494v1/figures/shap_ragtruth.png)

Figure 6:SHAP feature importance for the residual\-risk score model trained on all candidate features\. Left: AutoFA\. Right: RAGTruth\.
#### Post\-hoc policy validation\.

To further evaluate how the selected features translate into effective policies, we compare held\-out ESS gain across feature configurations in Table[5](https://arxiv.org/html/2609.00494#A2.T5)\. On AutoFA, ref\-support\-strength \(\|ρ\|=0\.50\|\\rho\|=0\.50\) alone outperforms self\-confidence \(\|ρ\|=0\.12\|\\rho\|=0\.12\) alone by a large margin, validating that FSA identifies a stronger signal\. On RAGTruth, self\-confidence \(\|ρ\|=0\.63\|\\rho\|=0\.63\) is already strong; the addition of flag\-reliability provides a modest improvement \(\|ρ\|=0\.69\|\\rho\|=0\.69\)\. In both cases, adding more features beyond the selected set does not improve ESS, consistent with the observation of overfitting at limited calibration size\.

DatasetFeature configurationCVρ\\rhoESS gainAutoFASelf\-confidence only0\.12\+16\.6%\+16\.6\\%Ref\-support\-strength0\.50\+40\.5%\\mathbf\{\+40\.5\\%\}All candidates \(5\)0\.52\+24\.7%\+24\.7\\%RAGTruthSelf\-confidence only0\.63\+17\.8%\+17\.8\\%Self\-confidence\+ flag\-reliability0\.69\+19\.7%\\mathbf\{\+19\.7\\%\}All candidates \(6\)0\.71\+9\.6%\+9\.6\\%Table 5:Feature selection validation\. CVρ\\rho: 5\-fold cross\-validated Spearman on calibration set\. ESS gain: post\-hoc evaluation on held\-out split atnbgt=300n\_\{\\mathrm\{bgt\}\}=300\. Bold: selected configuration\. Adding more features improves CV correlation but can reduce ESS due to overfitting\.The key observation is that higher univariate or multivariate correlation does not guarantee better policy performance\. On AutoFA, the “all features” model achieves CVρ=0\.52\\rho=0\.52but only\+24\.7%\+24\.7\\%ESS, while the single\-feature model achieves\+40\.5%\+40\.5\\%ESS\. This occurs because the multi\-feature model requires more aggressive uniform mixing \(τ=0\.70\\tau=0\.70\) to stabilize extreme weights, which dilutes the policy concentration\. The single feature provides smoother, full\-population ranking that translates more effectively into sampling probabilities\.

## Appendix CHuman Annotation Details

Both datasets leverage human annotators to acquire reference labels and we summarize the details as follows: RAGTruth\([Niu et al\., 2024](https://arxiv.org/html/2609.00494#bib.bib5)\)reports two independent annotators per response and a third review for substantial disagreements, and consistency rates of91\.8%91\.8\\%at the response level and78\.8%78\.8\\%at the span level\. Annotators were recruited through a professional vendor and compensated at$25\\$25per hour\. For AutoFA, all records received a human annotation, and90%90\\%received a second, independent annotation\. For records where two annotators disagree with each other on the final response factual verdict, a third human moderator was involved to adjudicate and made the final judgment\. The human annotators are instructed to extract core, verifiable claims from the model responses and search the web to judge the factual correctness of the claims and final overall response level factual correctness\.

## Appendix DQualitative Analysis

We provide two qualitative examples from RAGTruth illustrating LLM judge errors and how FSA identifies them\.

In record 168, the response states that Walter Scott was “shot whilst running from Officer Michael Slager\.” The reference states that Scott was “fatally shot in the back by a police officer” and separately describes video showing Scott running from the officer while the officer fires eight shots\. The response therefore combines two closely connected facts into a concise, well\-supported statement, but the LLM judge flags “shot whilst running” as hallucinated, apparently requiring the temporal relation to be stated verbatim\. Importantly, the LLM judge assigns 0\.85 self\-confidence, so confidence\-based sampling would not identify this as a high\-risk record\. In contrast, the FSA feature flag\-reliability receives a score of 2 \(higher means LLM\-judge is more likely to be wrong\), indicating elevated over\-detection risk and allowing the residual\-risk policy to prioritize the record for human annotation\.

In record 5718, the structured reference lists the same hours, 12:00–17:00, for every day from Monday through Sunday\. The response accurately summarizes this as operating “seven days a week from 12:00 to 17:00,” yet the LLM judge flags the statement, apparently failing to aggregate the seven JSON entries\. The LLM judge reports a low self\-confidence score of 0\.30 while the FSA assigns the maximum unreliability \(flag\-reliability=3\)\.

## Appendix EFeature Extraction Prompts

This appendix lists the prompts used for self\-confidence scoring and FSA feature extraction\. All prompts are written as Jinja2 templates: variables enclosed by\{\{ \}\}are replaced with record\-specific content at inference time\. The prompts are used only to extract auxiliary features or confidence scores, not to determine the final human label\.

### E\.1Self\-Confidence Prompt

We use the prompt in Figure[7](https://arxiv.org/html/2609.00494#A5.F7)to elicit the judge’s confidence in its own previous verdict\. The same structure is used for both datasets, with minor wording changes depending on whether the underlying task is factuality verification or hallucination detection\.

Self\-Confidence Score: AutoFA VariantYou previously evaluated whether a model response is factually correct by comparing it against reference answers, and concluded that the response is\{factually correct / factually incorrect\}\.Inputs•User query:\{\{ query \}\}•Model response:\{\{ response \}\}•Reference answer 1:\{\{ ref\_1 \}\}•Reference answer 2:\{\{ ref\_2 \}\}•Reference answer 3:\{\{ ref\_3 \}\}•Previous verdict:\{The response is factually correct / incorrect\.\}InstructionHow likely is it that your previous verdict is correct?Output only a single number between 0 and 1 representing the probability\. Do not provide an explanation or reasoning\.OutputProbability:

Figure 7:Prompt used for generating self\-confidence score\.
### E\.2AutoFA Feature Extraction Prompt

The prompt in Figure[8](https://arxiv.org/html/2609.00494#A5.F8)extracts five FSA features for AutoFA\. The judge assesses the verification difficulty of a response given the query and three reference answers\. It is explicitly instructed not to make the final factuality decision\.

AutoFA: Evidence and Rubric FeaturesYou are an experienced evaluator in factuality assessment for voice assistant responses\.Your job is not to decide the final label\. Your job is to extract a compact set of interpretable features that characterize the difficulty of verifying whether the response is factually correct\.Use only the provided information\. Return valid JSON only\. Do not provide chain\-of\-thought\. Use integer scores in\{0,1,2,3\}, where higher values indicate more of the corresponding feature\.Feature definitions1\.reference\_temporal\_consistencyHow consistent the three references are on time\-dependent facts\.0= fully consistent;3= major conflicts on the core answer\.2\.reference\_support\_strengthHow strongly the references support the response’s core claims\.0= no support;3= all references confirm the core answer\.3\.inference\_requiredHow much inference is needed to verify the response against the references\.0= pure extractive matching;3= substantial reasoning required\.4\.claim\_importance\_varianceHow much the claims vary in importance to the core question\.0= all claims are equally important;3= core answer plus many peripheral claims\.5\.answer\_completeness\_vs\_correctnessWhether the response appears correct but potentially incomplete\.0= comprehensive;3= answers only a narrow slice\.Output format``` { "feature_name": { "score": int, "justification": str }, ... } ``` Inputs•User query:\{\{ query \}\}•Model response:\{\{ response \}\}•Reference answer 1:\{\{ ref\_1 \}\}•Reference answer 2:\{\{ ref\_2 \}\}•Reference answer 3:\{\{ ref\_3 \}\}

Figure 8:Prompt used for feature extraction for AutoFA\.
### E\.3RAGTruth Feature Extraction Prompt

The prompt in Figure[9](https://arxiv.org/html/2609.00494#A5.F9)extracts general verification\-difficulty features for RAGTruth records\. The general verification\-difficulty features characterize the response–reference relationship, while*flag\_reliability*additionally assesses the judge’s flagged spans\.

RAGTruth: Evidence and Rubric FeaturesYou are an experienced evaluator in groundedness assessment\.Your task is to extract features that characterize the difficulty of verifying whether the response is faithful to the reference\. If hallucination spans are provided, also assess whether those flagged spans are plausible\.Use only the provided information\. Return valid JSON only\. Do not provide chain\-of\-thought\. Use integer scores in\{0,1,2,3\}, where higher values indicate more of the corresponding feature\.Feature definitions1\.reference\_support\_strength: how strongly the reference supports the response’s claims\.0= very weak support;3= strong and direct support\.2\.inference\_required: how much inference beyond verbatim extraction is needed to verify the response\.0= purely extractive;3= substantial interpretation\.3\.qualitative\_language: degree of subjective or interpretive language in the response\.0= purely factual;3= heavily qualitative\.4\.flag\_reliability: if hallucination spans are provided, how likely the judge is correct about those flags\.0= very likely correct;3= likely incorrect due to over\-detection\. If no spans are provided, set this score to0and state that no flagged spans were provided\.Output formatReturn a JSON object of the following form:\{"feature\_name": \{"score": int, "justification": str\}, \.\.\.\}Inputs•Task type:\{\{ task\_type \}\}•Reference:\{\{ reference \}\}•Response:\{\{ response \}\}•Judge’s flagged spans:\{\{ spans \}\}

Figure 9:Prompt used for feature extraction for RAGTruth