B-CARE：基于类别相对证据的盲评估立场错误排序方法

# B-CARE: Blind Classwise Assessment of Relative Evidence for Stance Error Ranking
Source: [https://arxiv.org/html/2610.29350](https://arxiv.org/html/2610.29350)
Meysam Sabbaghan Doina CarageaAffiliation:Department of Computer ScienceAffiliation:Kansas State UniversityEmail:[\{meysamsabbaghan,dcaragea\}@ksu\.edu](mailto:)

###### Abstract

Stance classifiers make errors that can be costly to identify under limited human\-review budgets\. Prioritizing potentially incorrect predictions remains challenging, particularly across heterogeneous base models\. We introduce B\-CARE \(BlindClasswiseAssessment ofRelativeEvidence for Stance Error Ranking\), a prediction\-blind framework for identifying high\-risk stance predictions\. Given only a target and text, a fixed LLM verifier assigns ordinal evidence scores to all candidate stance labels without observing the base model’s prediction or confidence\. B\-CARE constructs a comparative risk score by contrasting the strongest alternative\-class evidence with the evidence supporting the predicted class\. The resulting classwise evidence profile can be reused across multiple base models without training a separate error detector or calibrating risk scores\. We evaluate B\-CARE on four stance benchmarks and three heterogeneous base LLMs, covering 12 dataset\-\-model settings and 5,206 unique test instances\. Evaluation includes error\-ranking discrimination \(AUROC and AUPRC\), selective prediction \(E\-AURC\), and review effectiveness \(ErrorCapture at fixed review budgets\)\. B\-CARE achieves a mean AUROC of 0\.735 and a mean AUPRC of 0\.524, capturing 41\.06% of errors within the highest\-risk 20% of predictions\. Although B\-CARE does not uniformly outperform strong confidence\-based baselines, it provides a reusable, complementary evidence signal for prioritizing potentially incorrect stance predictions across heterogeneous models\. The code is available in an anonymized repository\.111[https://anonymous\.4open\.science/r/B\-CARE\-Code\-EC84/](https://anonymous.4open.science/r/B-CARE-Code-EC84/)

## 1Introduction

Stance detection identifies whether a text favors, opposes, or takes no clear position toward a target[Mohammad et al\. \(2016\)](https://arxiv.org/html/2610.29350#bib.bib25)\. Despite advances in LLM\-based prompting[Akash et al\. \(2025\)](https://arxiv.org/html/2610.29350#bib.bib2)and reasoning[Zhang et al\. \(2026\)](https://arxiv.org/html/2610.29350#bib.bib41), as well as extensive study of external\-context integration[Nguyen and Kim \(2025\)](https://arxiv.org/html/2610.29350#bib.bib27), classification errors persist\. Under limited human\-review budgets, a key question is:*which predictions should be reviewed first?*

Existing reliability signals include maximum softmax probability \(MSP\)[Hendrycks and Gimpel \(2017\)](https://arxiv.org/html/2610.29350#bib.bib16), predictive entropy \(PE\)[Malinin and Gales \(2018\)](https://arxiv.org/html/2610.29350#bib.bib22), andp⁡\(True\)p\(\\mathrm\{True\}\)[Kadavath et al\. \(2022\)](https://arxiv.org/html/2610.29350#bib.bib17)\. Other approaches use semantic uncertainty or similarity\-based signals[Kuhn et al\. \(2023\)](https://arxiv.org/html/2610.29350#bib.bib18);[Farquhar et al\. \(2024\)](https://arxiv.org/html/2610.29350#bib.bib7);[Nguyen et al\. \(2025\)](https://arxiv.org/html/2610.29350#bib.bib26), evidential uncertainty[Kunitomo\-Jacquin et al\. \(2026\)](https://arxiv.org/html/2610.29350#bib.bib19), and consistency\- or verification\-based checks[Zhang et al\. \(2023a\)](https://arxiv.org/html/2610.29350#bib.bib36);[Xue et al\. \(2026\)](https://arxiv.org/html/2610.29350#bib.bib33);[Tan et al\. \(2025\)](https://arxiv.org/html/2610.29350#bib.bib29);[Goel et al\. \(2025\)](https://arxiv.org/html/2610.29350#bib.bib11);[Hamidieh et al\. \(2026\)](https://arxiv.org/html/2610.29350#bib.bib15)\. Closest to our setting, MPVStance[Zhang et al\. \(2025c\)](https://arxiv.org/html/2610.29350#bib.bib40)verifies and revises stance predictions, while MARCH[Li et al\. \(2026\)](https://arxiv.org/html/2610.29350#bib.bib21)checks claims without revealing the solver’s answer to its checker\. B\-CARE instead ranks errors in fixed predictions using prediction\-blind, graded evidence for competing classes\.

We introduce B\-CARE \(BlindClasswiseAssessment ofRelativeEvidence for Stance Error Ranking\)\. A fixed LLM verifier independently assesses ordinal evidenceEc​\(x\)E\_\{c\}\(x\)for each candidate stance class without observing the base prediction or confidence\. For a base classifierBBpredictingy^B\\hat\{y\}\_\{B\}, the error\-risk score is

RB​\(x\)=maxc≠y^B⁡Ec​\(x\)−Ey^B​\(x\)\.R\_\{B\}\(x\)=\\max\_\{c\\neq\\hat\{y\}\_\{B\}\}E\_\{c\}\(x\)\-E\_\{\\hat\{y\}\_\{B\}\}\(x\)\.\(1\)Unlike binary verifier disagreement, this score preserves the strength of predicted\-class support relative to competing evidence\. The resulting evidence profiles can be reused across classifiers without training an error detector or performing dataset\-specific risk calibration\.

We evaluate B\-CARE on SemEval\-2016 Task 6A[Mohammad et al\. \(2016\)](https://arxiv.org/html/2610.29350#bib.bib25), VaccineStance \(1,000 examples\)[Sabbaghan et al\. \(2026\)](https://arxiv.org/html/2610.29350#bib.bib43), COVID\-19\-Stance[Glandt et al\. \(2021\)](https://arxiv.org/html/2610.29350#bib.bib10), and P\-Stance[Li et al\. \(2021\)](https://arxiv.org/html/2610.29350#bib.bib20), with Phi\-4[Abdin and others \(2024\)](https://arxiv.org/html/2610.29350#bib.bib1), Mistral Small 24B Instruct \(2501\)[Mistral AI \(2025\)](https://arxiv.org/html/2610.29350#bib.bib24), and Gemma 3[Gemma Team \(2025\)](https://arxiv.org/html/2610.29350#bib.bib9)\. Across 12 dataset–model settings and 5,206 unique test instances, B\-CARE achieves mean AUROC 0\.735 and AUPRC 0\.524, capturing 41\.06% of existing errors at a 20% review budget\. Ablations support prediction blindness and graded evidence, although performance varies across datasets and does not consistently exceed strong confidence baselines\. Exploratory results also indicate complementarity with predictive entropy\. Our contributions are:

- •Comparative error ranking:Prediction\-blind classwise evidence and relative scoring without a trained error detector\.
- •Controlled analysis:Ablations of prediction exposure, evidence granularity, and score components\.
- •Evaluation and complementarity:Error ranking, selective prediction, and review effectiveness across four datasets and three base LLMs, including confidence complementarity\.

## 2Related Work

#### Stance Detection and Evidence\-Based Reasoning\.

Stance detection identifies a text’s position toward a target, including implicit stances[Mohammad et al\. \(2016\)](https://arxiv.org/html/2610.29350#bib.bib25)\. Prior work investigates target\-aware modeling, contextual knowledge, and cross\-target transfer[Garg and Caragea \(2024\)](https://arxiv.org/html/2610.29350#bib.bib8);[Beck et al\. \(2023\)](https://arxiv.org/html/2610.29350#bib.bib4);[Zhang et al\. \(2024\)](https://arxiv.org/html/2610.29350#bib.bib37);[Zhang et al\. \(2023b\)](https://arxiv.org/html/2610.29350#bib.bib35);[Allaway and McKeown \(2020\)](https://arxiv.org/html/2610.29350#bib.bib3)\. Benchmarks span political and public\-health discourse[Li et al\. \(2021\)](https://arxiv.org/html/2610.29350#bib.bib20);[Glandt et al\. \(2021\)](https://arxiv.org/html/2610.29350#bib.bib10), while recent work explores open\-target generalization, multi\-path reasoning, and retrieval\-augmented verification[Akash et al\. \(2025\)](https://arxiv.org/html/2610.29350#bib.bib2);[Zhang et al\. \(2025b\)](https://arxiv.org/html/2610.29350#bib.bib39);[Zhang et al\. \(2025c\)](https://arxiv.org/html/2610.29350#bib.bib40)\. MPVStance[Zhang et al\. \(2025c\)](https://arxiv.org/html/2610.29350#bib.bib40)uses multi\-perspective verification and retrieved evidence to assess and revise stance responses\. Unlike MPVStance, B\-CARE ranks errors in fixed predictions without revising them\. Joint stance and rumor\-veracity modeling has also been studied[Yang et al\. \(2024\)](https://arxiv.org/html/2610.29350#bib.bib34)\. External evidence does not consistently improve prediction[Nguyen and Kim \(2025\)](https://arxiv.org/html/2610.29350#bib.bib27), and zero\-shot biases complicate reliability assessment[Dubreuil et al\. \(2025\)](https://arxiv.org/html/2610.29350#bib.bib6)\.

#### Uncertainty, Consistency, and Error Detection\.

Uncertainty quantification for LLMs encompasses both white\-box and black\-box approaches, including probability\-based and consistency\-based methods[Shelmanov et al\. \(2025\)](https://arxiv.org/html/2610.29350#bib.bib28)\. Reliability assessment has been studied through fact verification[Guan et al\. \(2024\)](https://arxiv.org/html/2610.29350#bib.bib13), token\-level and semantic consistency[Wastl et al\. \(2025\)](https://arxiv.org/html/2610.29350#bib.bib32);[Gupta et al\. \(2025\)](https://arxiv.org/html/2610.29350#bib.bib14), and belief\-state\-based hallucination detection[Zhang et al\. \(2025a\)](https://arxiv.org/html/2610.29350#bib.bib38)\. Sampling\-based methods include SelfCheckGPT[Manakul et al\. \(2023\)](https://arxiv.org/html/2610.29350#bib.bib23)\. Chain\-of\-Verification uses independently answered questions to verify generated content[Dhuliawala et al\. \(2024\)](https://arxiv.org/html/2610.29350#bib.bib5)\. B\-CARE instead explicitly compares graded evidence for predicted and competing stance classes\.

#### Prediction\-Blind Verification and Error Ranking\.

Confidence\-based error detection and selective classification provide established approaches to prioritizing predictions for review[Hendrycks and Gimpel \(2017\)](https://arxiv.org/html/2610.29350#bib.bib16);[Geifman and El\-Yaniv \(2017\)](https://arxiv.org/html/2610.29350#bib.bib30);[Varshney et al\. \(2022\)](https://arxiv.org/html/2610.29350#bib.bib31)\. Independent verification and limitations of self\-consistency have also been investigated[Dhuliawala et al\. \(2024\)](https://arxiv.org/html/2610.29350#bib.bib5);[Tan et al\. \(2025\)](https://arxiv.org/html/2610.29350#bib.bib29)\. MARCH[Li et al\. \(2026\)](https://arxiv.org/html/2610.29350#bib.bib21)employs a solver–proposer–checker framework for hallucination mitigation\. Its checker verifies claims derived from the solver’s response against retrieved evidence without seeing the original answer\. Thus, answer\-blind verification is not itself unique to B\-CARE\.

B\-CARE obtains graded evidence for every stance class directly from the target and text, without observing the base prediction or deriving verification questions from it\. This evidence is contrasted with each base prediction to rank potential errors, rather than revise predictions or verify response\-derived factual claims\. We compare prediction\-blind and prediction\-aware verification, graded and binary evidence, and error prioritization under fixed review budgets\. The evidence profile is reusable across base classifiers without training a separate error detector\.

## 3Methodology

![Refer to caption](https://arxiv.org/html/2610.29350v1/figures/B-CARE-Evidence-RiskInfographic.png)Figure 1:Overview of B\-CARE\.An independent, prediction\-blind LLM verifier assigns ordinal evidence scores to all candidate stance labels without observing the base prediction\. B\-CARE contrasts evidence supporting the predicted label with its strongest alternative to produce an ordinal error\-risk score for review prioritization\. The evidence profile can be reused across multiple base classifiers without additional verifier generations\.Figure[1](https://arxiv.org/html/2610.29350#S3.F1)illustrates the B\-CARE framework\. A base classifier predicts the stance of a target–text pair, while an independent verifier assesses textual evidence for each candidate class without observing the prediction\. B\-CARE then compares the predicted class with its strongest alternative to rank potentially incorrect predictions for review\.

### 3\.1Error Ranking

Letx=\(T,X\)x=\(T,X\)denote an input comprising a targetTTand textXX\. A base classifierBBproduces a stance prediction

y^B=fB​\(x\),y^B∈𝒴,\\hat\{y\}\_\{B\}=f\_\{B\}\(x\),\\qquad\\hat\{y\}\_\{B\}\\in\\mathcal\{Y\},\(2\)where𝒴\\mathcal\{Y\}denotes the dataset\-specific label set\. SemEval\-2016, VaccineStance, and COVID\-19\-Stance use𝒴=\{favor,against,none\}\\mathcal\{Y\}=\\\{\\textsc\{favor\},\\textsc\{against\},\\textsc\{none\}\\\}, while P\-Stance uses𝒴=\{favor,against\}\\mathcal\{Y\}=\\\{\\textsc\{favor\},\\textsc\{against\}\\\}\. The COVID\-19\-Stance labelneutralis mapped tonone\.

Given the gold labelyy, we define the evaluation\-only error indicator

zB\(x\)=𝕀\[y^B≠y\]\.z\_\{B\}\(x\)=\\mathbb\{I\}\[\\hat\{y\}\_\{B\}\\neq y\]\.\(3\)Gold labels are unavailable to the risk estimator at inference time\. Our objective is to assign each existing prediction a scalar risk scoreRB​\(x\)R\_\{B\}\(x\), with larger values indicating higher priority for review\. Unlike stance classification, this task does not generate or correct labels\. The risk score is not interpreted as a calibrated probability of error\.

### 3\.2Prediction\-Blind Classwise Evidence

B\-CARE employs Llama\-3\.3\-70B\-Instruct through OpenRouter as a fixed external evidence verifierVV, with temperature00and a maximum of 80 output tokens per request\. One independent request produces a complete classwise evidence profile\. Frozen ternary and binary prompt variants are used for the corresponding label sets, with a common evidence scale and output schema\.

Given the target, text, and candidate labels, the verifier produces an ordinal evidence profile

E⁡\(x\)=\(Ec​\(x\)\)c∈𝒴,E\(x\)=\\bigl\(E\_\{c\}\(x\)\\bigr\)\_\{c\\in\\mathcal\{Y\}\},\(4\)whereEc​\(x\)∈\{0,1,2,3\}E\_\{c\}\(x\)\\in\\\{0,1,2,3\\\}\. The scale represents increasing textual support: no evidence \(0\), weak or ambiguous evidence \(1\), moderate evidence \(2\), and strong, direct evidence \(3\)\.

The verifier evaluates textual evidence for each candidate stance and is explicitly instructed not to output a final stance prediction\. It receives neither the base prediction nor its confidence, logits, or model identity\. Gold labels and correctness indicators are also withheld\. Thus, evidence generation depends on the input and label set rather than on the base prediction\.

For a base predictiony^B\\hat\{y\}\_\{B\}, we define the evidence supporting that prediction as

SB​\(x\)=Ey^B​\(x\)\.S\_\{B\}\(x\)=E\_\{\\hat\{y\}\_\{B\}\}\(x\)\.\(5\)The strongest competing evidence is

AB​\(x\)=maxc∈𝒴c≠y^B⁡Ec​\(x\)\.A\_\{B\}\(x\)=\\max\_\{\\begin\{subarray\}\{c\}c\\in\\mathcal\{Y\}\\\\ c\\neq\\hat\{y\}\_\{B\}\\end\{subarray\}\}E\_\{c\}\(x\)\.\(6\)B\-CARE then assigns the comparative evidence risk

RB​\(x\)=AB​\(x\)−SB​\(x\)\.R\_\{B\}\(x\)=A\_\{B\}\(x\)\-S\_\{B\}\(x\)\.\(7\)
Since evidence scores lie in\{0,1,2,3\}\\\{0,1,2,3\\\}, the risk belongs to\{−3,−2,…,3\}\\\{\-3,\-2,\\ldots,3\\\}\. Positive scores indicate stronger evidence for an alternative stance, whereas negative scores indicate stronger support for the predicted label\. A zero score indicates equal evidence for the predicted class and its strongest alternative, not necessarily equal support across all candidate classes\. Predictions are ranked in descending order ofRB​\(x\)R\_\{B\}\(x\)\. For review budgetqq, we reviewk=max⁡\(1,round⁡\(q​N\)\)k=\\max\(1,\\mathrm\{round\}\(qN\)\)predictions per dataset–model setting\.

### 3\.3Prediction Blindness and Evidence Reuse

Prediction blindness separates evidence assessment from base predictions and enables evidence reuse across classifiers\.

#### Prediction exposure\.

A prediction\-aware verifier may condition its assessment on the predicted label, potentially introducing confirmation bias\. B\-CARE instead generates evidence without observing that label\. We examine this distinction in Section[6\.3](https://arxiv.org/html/2610.29350#S6.SS3), comparing blind and prediction\-aware verification under the same verifier backbone, evidence scale, output schema, and risk formula\. The prediction\-aware variant additionally receives the base prediction\.

#### Evidence reuse\.

Because classwise evidence depends only on the target–text input and label set, one verifier generation can serve multiple base classifiers evaluated on the same example\. Their respective risk scores are computed offline using the shared evidence vector\.

Prediction\-aware verification can also cache responses when base classifiers agree; across 5,206 examples, B\-CARE requires 5,206 verifier generations versus 6,544 for a cached prediction\-aware alternative \(20\.4% fewer; retries, tokens, and latency excluded\)\.

## 4Experimental Setup

### 4\.1Datasets and Models

We evaluate B\-CARE on four stance benchmarks with 5,206 unique test examples across 13 targets: SemEval\-2016 Task 6A \(five targets, excluding Donald Trump\)[Mohammad et al\. \(2016\)](https://arxiv.org/html/2610.29350#bib.bib25), VaccineStance \(1,000 examples\)[Sabbaghan et al\. \(2026\)](https://arxiv.org/html/2610.29350#bib.bib43), COVID\-19\-Stance \(four targets, 200 each\)[Glandt et al\. \(2021\)](https://arxiv.org/html/2610.29350#bib.bib10), and P\-Stance \(2,157 examples, three targets\)[Li et al\. \(2021\)](https://arxiv.org/html/2610.29350#bib.bib20)\. The first three usefavor,against, andnone\(mapping neutral tonone\); P\-Stance is binary\. Target statistics appear in Appendix[A](https://arxiv.org/html/2610.29350#A1)\.

Phi\-4 \(14B\)[Abdin and others \(2024\)](https://arxiv.org/html/2610.29350#bib.bib1), Mistral Small 24B Instruct \(2501\)[Mistral AI \(2025\)](https://arxiv.org/html/2610.29350#bib.bib24), and Gemma 3 \(27B\)[Gemma Team \(2025\)](https://arxiv.org/html/2610.29350#bib.bib9)were run on a university HPC cluster with revision\-pinned Hugging Face checkpoints, local Transformers inference, and frozen stance prompts\.

Across the 12 dataset–model settings, B\-CARE uses Llama\-3\.3\-70B\-Instruct[Grattafiori and others \(2024\)](https://arxiv.org/html/2610.29350#bib.bib12);[Meta AI \(2024\)](https://arxiv.org/html/2610.29350#bib.bib42)via OpenRouter as a fixed prediction\-blind verifier \(temperature 0\)\. A separate set of 2,000 validation examples \(500 per dataset\) was used for method development and validation\. The verifier prompt, ordinal scale, output schema, and inference settings were finalized before test evaluation\. Exact model revisions and configurations are provided in the reproducibility materials\. The verifier prompts and prediction\-exposure controls are detailed in Appendix[B](https://arxiv.org/html/2610.29350#A2)\.

No supervised error\-risk training or calibration is required\. Training examples are used only for specified baselines, including 20 few\-shot examples forp⁡\(True\)p\(\\mathrm\{True\}\)\.

### 4\.2Baselines and Controlled Variants

We compare against uncertainty, self\-evaluation, sampling, and verification baselines and controlled variants\.

#### Base\-model uncertainty\.

Maximum Softmax Probability \(MSP\)[Hendrycks and Gimpel \(2017\)](https://arxiv.org/html/2610.29350#bib.bib16)uses one minus the maximum class probability; Predictive Entropy \(PE\)[Malinin and Gales \(2018\)](https://arxiv.org/html/2610.29350#bib.bib22)measures closed\-set stance uncertainty\.

#### Self\-evaluation\.

p⁡\(True\)p\(\\mathrm\{True\}\)[Kadavath et al\. \(2022\)](https://arxiv.org/html/2610.29350#bib.bib17);[Farquhar et al\. \(2024\)](https://arxiv.org/html/2610.29350#bib.bib7)estimates the correctness of the base prediction following the Semantic Uncertainty implementation\. Our Direct Self\-Verification \(DSV\) control uses a separate True/False correctness prompt\.

#### Sampling\-based uncertainty\.

Label Entropy \(LE\) measures entropy over sampled stance labels; Variation Ratio \(VR\) measures disagreement with the modal label\. Semantic Entropy \(SE\)[Kuhn et al\. \(2023\)](https://arxiv.org/html/2610.29350#bib.bib18);[Farquhar et al\. \(2024\)](https://arxiv.org/html/2610.29350#bib.bib7)groups semantically equivalent responses; Evidential Semantic Entropy \(EVSE\)[Kunitomo\-Jacquin et al\. \(2026\)](https://arxiv.org/html/2610.29350#bib.bib19)extends this through evidential modeling\.

#### Comparative evidence controls\.

Same\-Model Comparative Evidence applies B\-CARE’s classwise evidence scoring andAB−SBA\_\{B\}\-S\_\{B\}formula using the base model as verifier\. Evidence\-Label Disagreement \(LD\) uses the external verifier’s classwise evidence to check whether the base prediction uniquely attains the maximum evidence score; ties are counted as disagreement\. Same\-Model Label Disagreement applies the same unique\-maximum rule to evidence generated by the base model\.

#### Model\-disagreement controls\.

Cross\-Base Disagreement \(CBD\) measures disagreement among the three base models; Llama Label Disagreement compares the base prediction with an independently generated external Llama stance label\.

#### External verification controls\.

External Direct Correctness Verification \(EDCV\) exposes the base prediction for direct correctness assessment\. Prediction\-Aware Comparative Evidence \(PACE\) retains B\-CARE’s verifier, ordinal scale, output schema, andAB−SBA\_\{B\}\-S\_\{B\}formula but reveals the base prediction, isolating prediction exposure\. Random ranking provides a chance reference; standalone Llama stance classification is evaluated separately\.

#### Component and sensitivity analyses\.

We compare support\-only \(−SB\-S\_\{B\}\), alternative\-only \(ABA\_\{B\}\), and comparative \(AB−SBA\_\{B\}\-S\_\{B\}\) scores, plus lexicographic ranking, softmax\-based support, and normalized evidence share\. A random\-label exposure control tests the effect of revealing an unrelated candidate label\. These analyses are separate from primary baseline comparisons\. Full implementation details for all baselines and controlled variants are provided in Appendix[C](https://arxiv.org/html/2610.29350#A3)\.

### 4\.3Metrics and Statistical Analysis

Our primary metric is error\-ranking AUROC, with incorrect predictions as positives\. We also report AUPRC, error prevalence, area under the risk–coverage curve \(AURC\), excess AURC \(E\-AURC\), and ErrorCapture@qq\(EC@qq\), the fraction of existing errors captured by reviewing the highest\-riskqqfraction of predictions\.

BecauseRB​\(x\)∈\{−3,−2,…,3\}R\_\{B\}\(x\)\\in\\\{\-3,\-2,\\ldots,3\\\}, ties are common\. AUROC uses midranks, AUPRC uses average precision over score thresholds, and EC@qquses expected fractional allocation at budget\-boundary ties to avoid arbitrary ordering\.

Paired within\-setting bootstrap comparisons use 10,000 matched\-example resamples with predefined seeds\. Means and standard deviations across the 12 settings reflect between\-setting variation, not run\-to\-run uncertainty\. Win counts are descriptive because settings are not independent experimental replications\. Tie handling for AURC and E\-AURC is described in Appendix[C](https://arxiv.org/html/2610.29350#A3)\.

## 5Main Results

### 5\.1Overall Error\-Ranking Performance

![Refer to caption](https://arxiv.org/html/2610.29350v1/figures/Heatmap-of-Evidence-Label-Disagreement.png)Figure 2:AUROC comparison across 12 dataset–model settings\.B\-CARE and 13 competing error\-ranking methods are evaluated on four stance\-detection benchmarks and three base models, treating incorrect predictions as positive\. Cell values are shown to three decimal places; boldface identifies the highest unrounded AUROC within each setting\. The rightmost column reports the mean across all 12 settings\. B\-CARE achieves the highest mean AUROC \(0\.735\) among the 14 methods in this primary comparison and leads in five settings\.Figure[2](https://arxiv.org/html/2610.29350#S5.F2)compares B\-CARE with 13 competing error\-ranking methods across 12 dataset–model settings, covering four stance\-detection benchmarks and three base models\. AUROC measures how effectively each method ranks incorrect predictions above correct ones, independently of a review budget or decision threshold\.

B\-CARE achieves the highest AUROC in five of the 12 settings: SemEval–Phi, Vaccines–Gemma, and all three P\-Stance settings\. Notable AUROC values include 0\.797 on SemEval–Phi, 0\.849 on SemEval–Mistral, and 0\.766 on P\-Stance–Mistral\. Its performance across all three P\-Stance base models suggests that comparative classwise evidence can provide a useful error\-ranking signal across different model architectures\.

Performance nevertheless varies substantially across datasets and models\. On SemEval, B\-CARE performs strongly against most alternatives, althoughp⁡\(True\)p\(\\mathrm\{True\}\)achieves higher AUROC for Mistral and Gemma\. On Vaccines, B\-CARE leads with Gemma, while Same\-Model Evidence achieves the highest AUROC for Phi and confidence\-based methods remain competitive for Mistral\. On COVID, MSP and Predictive Entropy outperform B\-CARE across all three base models\. Thus, the benefits of comparative evidence are setting\-dependent rather than universal\.

B\-CARE also outperforms Direct Verification \(EDCV\) and Prediction\-Aware Comparative Evidence \(PACE\) in AUROC across the evaluated settings\. Unlike direct correctness judgments or evidence conditioned on the predicted label, B\-CARE compares prediction\-independent evidence for competing stance classes\.

A separate matched evaluation against verifier probabilities is reported in Section[6\.5](https://arxiv.org/html/2610.29350#S6.SS5)\. Additional label\-disagreement, random\-ranking, and normalized evidence\-share controls are summarized in Appendix[C\.1](https://arxiv.org/html/2610.29350#A3.SS1)\(Table[T3](https://arxiv.org/html/2610.29350#A3.T3)\)\.

### 5\.2Complementary Evaluation Metrics

Table 1:Complementary error\-ranking performance \(mean±\\pmsample standard deviation across 12 dataset–model settings\)\. Higher AUPRC and EC and lower E\-AURC are better\. EC is expressed as a percentage and uses expected fractional allocation at budget\-boundary ties\. Bold indicates the best mean in each column\.Beyond AUROC, we evaluate AUPRC, excess area under the risk–coverage curve \(E\-AURC\), and ErrorCapture \(EC\) at review budgets of 10%, 20%, and 30%\. Table[1](https://arxiv.org/html/2610.29350#S5.T1)reports mean performance across all 12 dataset–model settings\. B\-CARE achieves the highest mean AUPRC \(0\.524\) and EC at all three budgets \(23\.85%, 41\.06%, and 54\.35%, respectively\), indicating effective error prioritization under limited review budgets\. However, Predictive Entropy achieves a lower E\-AURC \(0\.119 versus 0\.125\)\. These findings demonstrate advantages in specific review metrics without implying uniform superiority\.

### 5\.3Statistical Significance

We compare AUROC using paired, example\-level bootstrap resampling with unadjusted percentile 95% confidence intervals across 156 comparisons\. B\-CARE shows positive AUROC differences with intervals excluding zero in all 12 settings against Direct Verification, Evidential SE, Label Entropy, Variation Ratio, and Evidence\-Label Disagreement, and in 11/12 settings against Same\-Model Evidence and Semantic Entropy\. The only nonsignificant comparisons for these two baselines occur on Vaccines with Phi and Vaccines with Mistral, respectively\. Against Prediction\-Aware Evidence, B\-CARE records seven significant wins, no significant losses, and five nonsignificant differences\. Comparisons with MSP and Predictive Entropy each yield four wins, four losses, and four nonsignificant differences\.

The Same\-Model Evidence comparison preserves the evidence rubric and ranking rule but changes the verifier and inference environment; the observed performance gap therefore cannot be attributed solely to verifier capacity\. These findings are based on unadjusted intervals and do not establish family\-wise significance across all comparisons\. Complete results appear in Appendix[D](https://arxiv.org/html/2610.29350#A4)\.

### 5\.4Complementarity with Predictive Entropy

We examine complementarity between B\-CARE and Predictive Entropy \(PE\)\. Across 12 dataset–model settings, the sets of actual errors captured within B\-CARE’s and PE’s independently selected top\-20% review sets have a mean Jaccard overlap of 0\.406, while their union captures 55\.5% of all errors \(Table[T6](https://arxiv.org/html/2610.29350#A5.T6); see also Figure[A1](https://arxiv.org/html/2610.29350#A5.F1)in Appendix[E](https://arxiv.org/html/2610.29350#A5)\)\. Importantly, this union exceeds a single 20% review budget\.

An exploratory, untuned 0\.5/0\.5 average of within\-setting normalized ranks achieves a mean AUROC of 0\.760, compared with 0\.735 for B\-CARE and 0\.719 for PE\. The hybrid outperforms B\-CARE in 9/12 settings but underperforms in all three SemEval settings\. These findings suggest complementary, dataset\-dependent signals rather than uniform improvements\. The hybrid is not part of the frozen B\-CARE method\. Additional results appear in Appendix[E](https://arxiv.org/html/2610.29350#A5)\.

## 6Ablation Studies

### 6\.1Component Ablation of the B\-CARE Score

We examine the contribution of each component of the B\-CARE score by comparing the full formulation \(A−SA\-S\) with support\-only \(−S\-S\) and alternative\-only \(AA\) variants\. As shown in Figure[3](https://arxiv.org/html/2610.29350#S6.F3), the full score achieves the highest mean AUROC \(0\.735\), compared with 0\.728 for−S\-Sand 0\.686 forAA\. It outperforms the two variants in 11/12 and 10/12 settings, respectively\. Paired bootstrap comparisons identify significant gains in 6/12 and 9/12 settings, with no significant losses\. These results indicate that predicted\-label support provides a strong ranking signal, while competing\-class evidence offers additional benefits\. Detailed results are provided in Appendix[F](https://arxiv.org/html/2610.29350#A6)\.

![Refer to caption](https://arxiv.org/html/2610.29350v1/figures/Academic-AUROC-Model-Comparison.png)Figure 3:Component ablation of the B\-CARE score across 12 dataset–model settings\. Higher AUROC indicates better error ranking\.
### 6\.2Ordinal versus Continuous Evidence Scoring

We test whether continuous evidence scores improve on ordinal grades\. The same fixed, prediction\-blind Llama verifier is prompted for real\-valued classwise scores on\[0,3\]\[0,3\], with the sameRB=AB−SBR\_\{B\}=A\_\{B\}\-S\_\{B\}formulation\. The two variants achieve nearly identical mean AUROC \(0\.7346 ordinal vs\. 0\.7345 continuous\)\. As shown in Figure[4](https://arxiv.org/html/2610.29350#S6.F4), ordinal scoring is better on SemEval for all three base models, continuous scoring is better on COVID \(intervals excluding zero in two of three settings\), and neither has a clear advantage on Vaccines or P\-Stance\. Finer granularity therefore yields no consistent gain, supporting ordinal scoring as a competitive and interpretable choice\.

![Refer to caption](https://arxiv.org/html/2610.29350v1/figures/AUROC-Forest-Plot-by-Dataset.png)Figure 4:Ordinal versus continuous evidence scoring across 12 dataset–model settings\. Dots show AUROC differences \(continuous minus ordinal\); horizontal bars indicate unadjusted 95% paired bootstrap confidence intervals \(10,000 resamples\)\. Positive values favor continuous scoring, negative values favor ordinal scoring, and intervals excluding zero indicate differences at the unadjusted 95% level\.
### 6\.3Effect of Prediction Blindness

We assess the role of prediction blindness by comparing B\-CARE’s prediction\-blind evidence \(PBCE\) with a prediction\-aware variant \(PACE\)\. Both use the same Llama\-3\.3\-70B verifier, evidence scale, andAB−SBA\_\{B\}\-S\_\{B\}score; PACE additionally receives the base prediction\. As shown in Figure[5](https://arxiv.org/html/2610.29350#S6.F5), PBCE achieves higher AUROC in all 12 dataset–model settings, with a mean improvement of 0\.0244\. Seven paired\-bootstrap 95% confidence intervals exclude zero in favor of PBCE, with none favoring PACE\. These results support prediction blindness as a beneficial design choice, although not every individual improvement is statistically significant\.

![Refer to caption](https://arxiv.org/html/2610.29350v1/figures/Scientific-Heatmap.png)Figure 5:Effect of prediction blindness on error\-ranking AUROC\. Cells report the AUROC difference \(PBCE minus PACE\) with unadjusted 95% paired\-bootstrap confidence intervals \(10,000 resamples\)\. Darker purple indicates larger gains; stars denote intervals excluding zero\.
### 6\.4Graded Evidence versus Evidence\-Label Disagreement

We assess whether graded relative evidence improves error ranking beyond Evidence\-Label Disagreement, using identical frozen verifier outputs\. As shown in Table[T8](https://arxiv.org/html/2610.29350#A7.T8)\(Appendix[G](https://arxiv.org/html/2610.29350#A7)\), B\-CARE achieves higher AUROC in all 12 settings \(meanΔ\\DeltaAUROC=\+0\.0803=\+0\.0803\), with all 12 unadjusted 95% paired\-bootstrap confidence intervals excluding zero\.

These results show that preserving graded relative evidence provides a stronger error\-ranking signal than binary Evidence\-Label Disagreement under the evaluated settings\. Detailed results and additional analyses are provided in Appendix[G](https://arxiv.org/html/2610.29350#A7)\.

### 6\.5Comparison with Verifier Probabilities

We also compare B\-CARE with LlamaProb \(LP\), which ranks errors by1−pV​\(y^B∣x,t\)1\-p\_\{V\}\(\\hat\{y\}\_\{B\}\\mid x,t\)from the same verifier\. Both were evaluated on a self\-hosted Llama\-3\.3\-70B\-Instruct with identical test instances and frozen base predictions; this matched evaluation is separate from the 14\-method comparison, which used a different inference environment\. B\-CARE achieves higher AUROC in 9 of 12 settings, with nearly identical mean AUROC \(0\.760 vs\. 0\.761\)\. B\-CARE achieves higher mean AUPRC \(0\.603 vs\. 0\.592\) and EC@20% \(0\.444 vs\. 0\.428\), whereas LP achieves lower mean E\-AURC \(0\.108 vs\. 0\.116\)\.

B\-CARE leads on VaccineStance and P\-Stance; LP leads on COVID\-19\-Stance\. B\-CARE thus has the highest mean AUROC among the 14 methods in the primary evaluation, but verifier\-probability ranking performs comparably on average\. Per\-setting results, inference\-environment provenance, and COVID\-19\-Stance diagnostics are in Appendix[H](https://arxiv.org/html/2610.29350#A8)\.

## 7Conclusion

We introduced B\-CARE, a training\-free framework for ranking errors in frozen stance predictions using prediction\-blind, classwise ordinal evidence from an external LLM verifier\. Across four datasets, three base models, and 12 evaluation settings, B\-CARE achieved mean AUROC of 0\.735 and AUPRC of 0\.524, capturing 41\.06% of existing errors within a 20% review budget\. Controlled ablations demonstrated the benefits of prediction blindness and graded comparative evidence over prediction\-aware and binary\-disagreement alternatives\. However, performance varied across datasets, and B\-CARE did not consistently outperform strong confidence baselines\. These findings position B\-CARE as a complementary, reusable signal for prioritizing human review rather than a universal replacement for model uncertainty\. Future work will investigate adaptive combinations with confidence\-based signals, finer\-grained evidence scoring, and generalization beyond stance detection\.

## Limitations

B\-CARE relies on the quality of classwise evidence generated by an external LLM verifier\. Although prediction blindness prevents direct exposure to the base prediction, it does not eliminate shared model biases or guarantee correct evidence\. The seven\-level ordinal risk score also produces ties that can limit ranking resolution, particularly within large groups of equally scored examples\.

Our evaluation spans four English\-language stance datasets, 13 targets, and three base LLMs, but uses a single external verifier family\. Generalization to other languages, domains, tasks, and verifiers remains untested\. Performance varies across datasets and does not consistently surpass strong confidence\-based baselines, suggesting that comparative evidence and model uncertainty capture complementary signals\.

External verification incurs additional inference cost relative to uncertainty scores derived directly from base\-model outputs\. Furthermore, the original OpenRouter\-hosted verifier lacked a pinned backend checkpoint revision, limiting exact output reproducibility\. Finally, B\-CARE prioritizes potential errors rather than correcting them; improvements from actual human review have not been directly evaluated\.

## References

- M\. Abdinet al\.Phi\-4 technical report\.External Links:2412\.08905,[Link](https://arxiv.org/abs/2412.08905)Cited by:[§1](https://arxiv.org/html/2610.29350#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.29350#S4.SS1.p2.1)\.
- Akashet al\.\(2025\)A\. U\. Akash, A\. Fahmy, and A\. TrabelsiCan large language models address open\-target stance detection?\.InFindings of ACL 2025,pp\. 971–985\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.54),[Link](https://aclanthology.org/2025.findings-acl.54/)Cited by:[§1](https://arxiv.org/html/2610.29350#S1.p1.1),[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px1.p1.1)\.
- Allaway and McKeown \(2020\)E\. Allaway and K\. McKeownZero\-shot stance detection: a dataset and model using generalized topic representations\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 8913–8931\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.717),[Link](https://aclanthology.org/2020.emnlp-main.717/)Cited by:[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px1.p1.1)\.
- Becket al\.\(2023\)T\. Beck, A\. Waldis, and I\. GurevychRobust integration of contextual information for cross\-target stance detection\.InProceedings of the 12th Joint Conference on Lexical and Computational Semantics \(\*SEM 2023\),pp\. 494–511\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.starsem-1.43),[Link](https://aclanthology.org/2023.starsem-1.43/)Cited by:[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px1.p1.1)\.
- Dhuliawalaet al\.\(2024\)S\. Dhuliawala, M\. Komeili, J\. Xu, R\. Raileanu, X\. Li, A\. Celikyilmaz, and J\. WestonChain\-of\-verification reduces hallucination in large language models\.InFindings of ACL 2024,pp\. 3563–3578\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.212),[Link](https://aclanthology.org/2024.findings-acl.212/)Cited by:[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px3.p1.1)\.
- Dubreuilet al\.\(2025\)A\. Dubreuil, A\. Gourru, C\. Largeron, and A\. TrabelsiAre stereotypes leading LLMs’ zero\-shot stance detection?\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 31517–31530\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1605),[Link](https://aclanthology.org/2025.emnlp-main.1605/)Cited by:[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px1.p1.1)\.
- Farquharet al\.\(2024\)S\. Farquhar, J\. Kossen, L\. Kuhn, and Y\. GalDetecting hallucinations in large language models using semantic entropy\.Nature630,pp\. 625–630\.External Links:[Document](https://dx.doi.org/10.1038/s41586-024-07421-0),[Link](https://doi.org/10.1038/s41586-024-07421-0)Cited by:[Appendix C](https://arxiv.org/html/2610.29350#A3.SS0.SSS0.Px3.p1.1),[Appendix C](https://arxiv.org/html/2610.29350#A3.SS0.SSS0.Px6.p1.1),[§1](https://arxiv.org/html/2610.29350#S1.p2.1),[§4\.2](https://arxiv.org/html/2610.29350#S4.SS2.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2610.29350#S4.SS2.SSS0.Px3.p1.1)\.
- Garg and Caragea \(2024\)K\. Garg and C\. CarageaStanceformer: target\-aware transformer for stance detection\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 4969–4984\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.286),[Link](https://aclanthology.org/2024.findings-emnlp.286/)Cited by:[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px1.p1.1)\.
- Geifman and El\-Yaniv \(2017\)Y\. Geifman and R\. El\-YanivSelective classification for deep neural networks\.InAdvances in Neural Information Processing Systems,Vol\.30,pp\. 4878–4887\.External Links:[Link](https://arxiv.org/abs/1705.08500)Cited by:[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px3.p1.1)\.
- Gemma Team \(2025\)Gemma TeamGemma 3 technical report\.External Links:2503\.19786,[Link](https://arxiv.org/abs/2503.19786)Cited by:[§1](https://arxiv.org/html/2610.29350#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.29350#S4.SS1.p2.1)\.
- Glandtet al\.\(2021\)K\. Glandt, S\. Khanal, Y\. Li, D\. Caragea, and C\. CarageaStance detection in COVID\-19 tweets\.InProceedings of ACL\-IJCNLP 2021,pp\. 1596–1611\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.127),[Link](https://aclanthology.org/2021.acl-long.127/)Cited by:[§1](https://arxiv.org/html/2610.29350#S1.p4.1),[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.29350#S4.SS1.p1.1)\.
- Goelet al\.\(2025\)A\. Goel, D\. Schwartz, and Y\. QiZero\-knowledge LLM hallucination detection and mitigation through fine\-grained cross\-model consistency\.InEMNLP 2025 Industry Track,pp\. 1982–1999\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-industry.139),[Link](https://aclanthology.org/2025.emnlp-industry.139/)Cited by:[§1](https://arxiv.org/html/2610.29350#S1.p2.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafioriet al\.The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.External Links:[Link](https://arxiv.org/abs/2407.21783)Cited by:[§4\.1](https://arxiv.org/html/2610.29350#S4.SS1.p3.1)\.
- Guanet al\.\(2024\)J\. Guan, J\. Dodge, D\. Wadden, M\. Huang, and H\. PengLanguage models hallucinate, but may excel at fact verification\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Mexico City, Mexico,pp\. 1090–1111\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.62),[Link](https://aclanthology.org/2024.naacl-long.62/)Cited by:[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px2.p1.1)\.
- Guptaet al\.\(2025\)R\. Gupta, P\. H\. Panicker, S\. Bhatia, and G\. RamakrishnanConsistency is the key: detecting hallucinations in LLM generated text by checking inconsistencies about key facts\.InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics,Mumbai, India,pp\. 2053–2068\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-ijcnlp.129),[Link](https://aclanthology.org/2025.findings-ijcnlp.129/)Cited by:[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px2.p1.1)\.
- Hamidiehet al\.\(2026\)K\. Hamidieh, V\. Thost, W\. Gerych, M\. Yurochkin, and M\. GhassemiComplementing self\-consistency with cross\-model disagreement for uncertainty quantification\.arXiv preprint arXiv:2604\.17112\.External Links:[Link](https://arxiv.org/abs/2604.17112)Cited by:[§1](https://arxiv.org/html/2610.29350#S1.p2.1)\.
- Hendrycks and Gimpel \(2017\)D\. Hendrycks and K\. GimpelA baseline for detecting misclassified and out\-of\-distribution examples in neural networks\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Hkg4TI9xl)Cited by:[Appendix C](https://arxiv.org/html/2610.29350#A3.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.29350#S1.p2.1),[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px3.p1.1),[§4\.2](https://arxiv.org/html/2610.29350#S4.SS2.SSS0.Px1.p1.1)\.
- Kadavathet al\.\(2022\)S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez,et al\.Language models \(mostly\) know what they know\.External Links:2207\.05221,[Link](https://arxiv.org/abs/2207.05221)Cited by:[Appendix C](https://arxiv.org/html/2610.29350#A3.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.29350#S1.p2.1),[§4\.2](https://arxiv.org/html/2610.29350#S4.SS2.SSS0.Px2.p1.1)\.
- Kuhnet al\.\(2023\)L\. Kuhn, Y\. Gal, and S\. FarquharSemantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VD-AYtP0dve)Cited by:[Appendix C](https://arxiv.org/html/2610.29350#A3.SS0.SSS0.Px6.p1.1),[§1](https://arxiv.org/html/2610.29350#S1.p2.1),[§4\.2](https://arxiv.org/html/2610.29350#S4.SS2.SSS0.Px3.p1.1)\.
- Kunitomo\-Jacquinet al\.\(2026\)L\. Kunitomo\-Jacquin, E\. Marrese\-Taylor, K\. Fukuda, and M\. HamasakiEvidential semantic entropy for LLM uncertainty quantification\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),Rabat, Morocco,pp\. 7107–7122\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.eacl-long.334),[Link](https://aclanthology.org/2026.eacl-long.334/)Cited by:[Appendix C](https://arxiv.org/html/2610.29350#A3.SS0.SSS0.Px7.p1.1),[§1](https://arxiv.org/html/2610.29350#S1.p2.1),[§4\.2](https://arxiv.org/html/2610.29350#S4.SS2.SSS0.Px3.p1.1)\.
- Liet al\.\(2021\)Y\. Li, T\. Sosea, A\. Sawant, A\. J\. Nair, D\. Inkpen, and C\. CarageaP\-stance: a large dataset for stance detection in political domain\.InFindings of ACL\-IJCNLP 2021,pp\. 2355–2365\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.findings-acl.208),[Link](https://aclanthology.org/2021.findings-acl.208/)Cited by:[§1](https://arxiv.org/html/2610.29350#S1.p4.1),[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.29350#S4.SS1.p1.1)\.
- Liet al\.\(2026\)Z\. Li, Y\. Zhang, P\. Cheng, J\. Song, M\. Zhou, H\. Li, S\. Hu, Y\. Qin, E\. Zhao, X\. Jiang, and G\. JiangMARCH: multi\-agent reinforced check for hallucination\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 39389–39415\.External Links:[Link](https://aclanthology.org/2026.acl-long.1828/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1828)Cited by:[§1](https://arxiv.org/html/2610.29350#S1.p2.1),[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px3.p1.1)\.
- Malinin and Gales \(2018\)A\. Malinin and M\. GalesPredictive uncertainty estimation via prior networks\.InAdvances in Neural Information Processing Systems,Vol\.31\.External Links:[Link](https://papers.nips.cc/paper_files/paper/2018/hash/3ea2db50e62ceefceaf70a9d9a56a6f4-Abstract.html)Cited by:[§1](https://arxiv.org/html/2610.29350#S1.p2.1),[§4\.2](https://arxiv.org/html/2610.29350#S4.SS2.SSS0.Px1.p1.1)\.
- Manakulet al\.\(2023\)P\. Manakul, A\. Liusie, and M\. GalesSelfCheckGPT: zero\-resource black\-box hallucination detection for generative large language models\.InProceedings of EMNLP 2023,pp\. 9004–9017\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.557),[Link](https://aclanthology.org/2023.emnlp-main.557/)Cited by:[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px2.p1.1)\.
- Meta AI \(2024\)Meta AILlama\-3\.3\-70B\-Instruct\.Note:Hugging Face Model CardReleased December 6, 2024External Links:[Link](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct)Cited by:[§4\.1](https://arxiv.org/html/2610.29350#S4.SS1.p3.1)\.
- Mistral AI \(2025\)Mistral AIMistral small 24b instruct 2501\.Note:Hugging Face Model CardExternal Links:[Link](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501)Cited by:[§1](https://arxiv.org/html/2610.29350#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.29350#S4.SS1.p2.1)\.
- Mohammadet al\.\(2016\)S\. Mohammad, S\. Kiritchenko, P\. Sobhani, X\. Zhu, and C\. CherrySemEval\-2016 Task 6: detecting stance in tweets\.InProceedings of SemEval\-2016,pp\. 31–41\.External Links:[Document](https://dx.doi.org/10.18653/v1/S16-1003),[Link](https://aclanthology.org/S16-1003/)Cited by:[§1](https://arxiv.org/html/2610.29350#S1.p1.1),[§1](https://arxiv.org/html/2610.29350#S1.p4.1),[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.29350#S4.SS1.p1.1)\.
- Nguyenet al\.\(2025\)D\. Nguyen, A\. Payani, and B\. MirzasoleimanBeyond semantic entropy: boosting LLM uncertainty quantification with pairwise semantic similarity\.InFindings of ACL 2025,pp\. 4530–4540\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.234),[Link](https://aclanthology.org/2025.findings-acl.234/)Cited by:[§1](https://arxiv.org/html/2610.29350#S1.p2.1)\.
- Nguyen and Kim \(2025\)Q\. M\. Nguyen and T\. KimIs external information useful for stance detection with LLMs?\.InFindings of ACL 2025,pp\. 14798–14807\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.764),[Link](https://aclanthology.org/2025.findings-acl.764/)Cited by:[§1](https://arxiv.org/html/2610.29350#S1.p1.1),[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px1.p1.1)\.
- Sabbaghanet al\.\(2026\)M\. Sabbaghan, N\. Gyawali, A\. Z\. Jahromi, S\. Loera\-Lozano, D\. Caragea, and C\. CarageaMoPrO\-sd: modular prompt optimization for stance detection\.External Links:2610\.13386,[Link](https://arxiv.org/abs/2610.13386)Cited by:[§1](https://arxiv.org/html/2610.29350#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.29350#S4.SS1.p1.1)\.
- Shelmanovet al\.\(2025\)A\. Shelmanov, M\. Panov, R\. Vashurin, A\. Vazhentsev, E\. Fadeeva, and T\. BaldwinUncertainty quantification for large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 5: Tutorial Abstracts\),pp\. 3–4\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-tutorials.3),[Link](https://aclanthology.org/2025.acl-tutorials.3/)Cited by:[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px2.p1.1)\.
- Tanet al\.\(2025\)H\. Tan, F\. Sun, S\. Liu, D\. Su, Q\. Cao, X\. Chen, J\. Wang, X\. Cai, Y\. Wang, H\. Shen, and X\. ChengToo consistent to detect: a study of self\-consistent errors in LLMs\.InProceedings of EMNLP 2025,pp\. 4755–4765\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.238),[Link](https://aclanthology.org/2025.emnlp-main.238/)Cited by:[§1](https://arxiv.org/html/2610.29350#S1.p2.1),[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px3.p1.1)\.
- Varshneyet al\.\(2022\)N\. Varshney, S\. Mishra, and C\. BaralTowards improving selective prediction ability of NLP systems\.InProceedings of the 7th Workshop on Representation Learning for NLP,pp\. 221–226\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.repl4nlp-1.23),[Link](https://aclanthology.org/2022.repl4nlp-1.23/)Cited by:[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px3.p1.1)\.
- Wastlet al\.\(2025\)M\. Wastl, J\. Vamvas, and R\. SennrichUZH at SemEval\-2025 task 3: token\-level self\-consistency for hallucination detection\.InProceedings of the 19th International Workshop on Semantic Evaluation \(SemEval\-2025\),pp\. 257–270\.External Links:[Link](https://aclanthology.org/2025.semeval-1.38/)Cited by:[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px2.p1.1)\.
- Xueet al\.\(2026\)Y\. Xue, K\. Greenewald, Y\. Mroueh, and B\. MirzasoleimanVerify when uncertain: beyond self\-consistency in black box hallucination detection\.Transactions on Machine Learning Research\.External Links:[Link](https://arxiv.org/abs/2502.15845)Cited by:[§1](https://arxiv.org/html/2610.29350#S1.p2.1)\.
- Yanget al\.\(2024\)R\. Yang, W\. Gao, J\. Ma, H\. Lin, and B\. WangReinforcement tuning for detecting stances and debunking rumors jointly with large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 13423–13439\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.796),[Link](https://aclanthology.org/2024.findings-acl.796/)Cited by:[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025a\)D\. Zhang, Q\. Hong, B\. Hou, J\. Lin, C\. Zhang, J\. Li, and J\. WangBold claims or self\-doubt? factuality hallucination type detection via belief state\.InFindings of the Association for Computational Linguistics: EMNLP 2025,Suzhou, China,pp\. 9946–9959\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.527),[Link](https://aclanthology.org/2025.findings-emnlp.527/)Cited by:[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2023a\)J\. Zhang, Z\. Li, K\. Das, B\. Malin, and S\. KumarSAC3\{\}^\{3\}: reliable hallucination detection in black\-box language models via semantic\-aware cross\-check consistency\.InFindings of EMNLP 2023,pp\. 15445–15458\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.1032),[Link](https://aclanthology.org/2023.findings-emnlp.1032/)Cited by:[§1](https://arxiv.org/html/2610.29350#S1.p2.1)\.
- Zhanget al\.\(2023b\)R\. Zhang, H\. Yang, and W\. MaoCross\-lingual cross\-target stance detection with dual knowledge distillation framework\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 10804–10819\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.666),[Link](https://aclanthology.org/2023.emnlp-main.666/)Cited by:[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2024\)Z\. Zhang, Y\. Li, J\. Zhang, and H\. XuLLM\-driven knowledge injection advances zero\-shot and cross\-target stance detection\.InProceedings of NAACL 2024 \(Short Papers\),pp\. 371–378\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-short.32),[Link](https://aclanthology.org/2024.naacl-short.32/)Cited by:[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026\)Z\. Zhang, J\. Zhang, J\. Guo, and X\. ChengModeling human\-like cognition for stance detection: integrating intuitive judgment and analytical reasoning\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),San Diego, California, United States,pp\. 5852–5867\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.264),[Link](https://aclanthology.org/2026.acl-long.264/)Cited by:[§1](https://arxiv.org/html/2610.29350#S1.p1.1)\.
- Zhanget al\.\(2025b\)Z\. Zhang, J\. Zhang, H\. Xu, J\. Guo, and X\. ChengMPRF: interpretable stance detection through multi\-path reasoning framework\.InProceedings of EMNLP 2025,pp\. 454–470\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.24),[Link](https://aclanthology.org/2025.emnlp-main.24/)Cited by:[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025c\)Z\. Zhang, Z\. Zhang, J\. Zhang, H\. Xu, and X\. ChengMPVStance: mitigating hallucinations in stance detection with multi\-perspective verification\.InProceedings of ACL 2025,pp\. 1053–1067\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.53),[Link](https://aclanthology.org/2025.acl-long.53/)Cited by:[§1](https://arxiv.org/html/2610.29350#S1.p2.1),[§2](https://arxiv.org/html/2610.29350#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix ADataset Statistics

This appendix summarizes the evaluation data used in B\-CARE\. Tables[T1](https://arxiv.org/html/2610.29350#A1.T1)and[T2](https://arxiv.org/html/2610.29350#A1.T2)report dataset\-level and target\-wise statistics, respectively\. The evaluation includes 5,206 examples across 13 targets and four benchmarks, comprising three ternary stance datasets and one binary dataset\.

Table T1:Evaluation\-set statistics\.NNdenotes the number of unique test examples, and Tgt\. denotes the number of targets\. F, A, and N represent FAVOR, AGAINST, and NONE, respectively\. P\-Stance does not contain the NONE class\.Table T2:Target\-wise distribution of test examples across the four evaluation benchmarks\. VaccineStance contains a single target \(Vaccines\), whereas SemEval\-2016, COVID\-19\-Stance, and P\-Stance contain five, four, and three targets, respectively\. Counts correspond to the evaluation sets used in B\-CARE\.
## Appendix BVerifier Prompt Templates

We provide the exact verifier prompts used in the B\-CARE experiments and the principal verification controls\. B\-CARE evaluates classwise textual evidence without access to the base model’s prediction, confidence, or gold label\. The three\-class template is used for SemEval\-2016, VaccineStance, and COVID\-19\-Stance, while P\-Stance uses a binary variant\. We additionally document the prediction\-aware PACE modification and the direct\-correctness EDCV control\.

### B\.1B\-CARE Prediction\-Blind Evidence Prompts

A1\. B\-CARE Three\-Class Evidence PromptInputTARGET:\{target\}TEXT:\{text\}InstructionsEvaluate the textual evidence for each possible stance of the TEXT toward the TARGET\.Do not decide which stance label is correct overall\.Instead, evaluate each hypothesis independently based only on evidence contained or clearly implied in the TEXT\.FAVORIs there explicit or clearly implied evidence that the author supports, endorses, agrees with, or expresses a favorable position toward the TARGET?AGAINSTIs there explicit or clearly implied evidence that the author opposes, rejects, criticizes, disagrees with, or expresses an unfavorable position toward the TARGET?NONEIs there explicit or clearly implied evidence that the author does not express a discernible stance of support or opposition toward the TARGET, such as presenting factual information, asking a neutral question, or discussing the topic without taking a position?Evidence\-Strength ScaleFor each hypothesis return an evidence\-strength score:Output FormatReturn JSON only:\{"FAVOR": 0,"AGAINST": 0,"NONE": 0\}Do not output a final stance label\.Do not explain your answer\.

A2\. B\-CARE Binary Evidence Prompt \(P\-Stance\)InputTARGET:\{target\}TEXT:\{text\}InstructionsEvaluate the textual evidence for each possible stance of the TEXT toward the TARGET\.Do not decide which stance label is correct overall\.Instead, evaluate each hypothesis independently based only on evidence contained or clearly implied in the TEXT\.FAVORIs there explicit or clearly implied evidence that the author supports, endorses, agrees with, or expresses a favorable position toward the TARGET?AGAINSTIs there explicit or clearly implied evidence that the author opposes, rejects, criticizes, disagrees with, or expresses an unfavorable position toward the TARGET?Evidence\-Strength ScaleFor each hypothesis return an evidence\-strength score:Output FormatReturn JSON only:\{"FAVOR": 0,"AGAINST": 0\}Do not output a final stance label\.Do not explain your answer\.

### B\.2Prediction\-Aware Control

A3\. PACE Prediction\-Aware ModificationPACE uses the corresponding B\-CARE evidence template, with the following additional field inserted immediately after the TEXT input:BASE PREDICTION:\{base\_prediction\}All remaining instructions, evidence\-strength definitions, and output fields are unchanged\.For three\-class datasets, PACE uses the three\-class B\-CARE template \(A1\)\.For P\-Stance, PACE uses the binary B\-CARE template \(A2\)\.

### B\.3Direct Correctness Verification Control

A4\. EDCV Three\-Class PromptSystem MessageRespond with exactly one integer character: 0, 1, 2, or 3\. Do not write any other text\.User MessageTask: Evaluate whether a stance prediction is correct\.Target:\{target\}Text:\{text\}Predicted stance:\{base\_prediction\}The possible stance labels are FAVOR, AGAINST, and NONE\.Determine how strongly the text supports the claim that the predicted stance is correct\.Return an integer correctness score from 0 to 3:0 = the predicted stance is clearly incorrect1 = the predicted stance is probably incorrect2 = the predicted stance is probably correct3 = the predicted stance is clearly correctOutput exactly one integer: 0, 1, 2, or 3\.

A5\. EDCV Binary Prompt \(P\-Stance\)System MessageRespond with exactly one integer character: 0, 1, 2, or 3\. Do not write any other text\.User MessageTask: Evaluate whether a stance prediction is correct\.Target:\{target\}Text:\{text\}Predicted stance:\{base\_prediction\}The possible stance labels are FAVOR and AGAINST\.Determine how strongly the text supports the claim that the predicted stance is correct\.Return an integer correctness score from 0 to 3:0 = the predicted stance is clearly incorrect1 = the predicted stance is probably incorrect2 = the predicted stance is probably correct3 = the predicted stance is clearly correctOutput exactly one integer: 0, 1, 2, or 3\.

## Appendix CBaseline and Control Implementation Details

All methods assign higher risk scores to instances more likely to contain incorrect base\-model predictions\. Unless stated otherwise,y^\\hat\{y\}denotes the frozen base\-model prediction\.

The three base models \(Phi, Mistral, and Gemma\) use greedy decoding \(do\_sample=false,max\_new\_tokens=16\), without an explicit temperature passed to generation\. The fixed external Llama verifier uses temperature0\.00\.0andmax\_tokens=80\.

The sampling\-based uncertainty baselines \(LE, VR, SE, and EVSE\) useM=5M=5generations, temperature1\.01\.0, top\-p=0\.9p=0\.9, andmax\_new\_tokens=8\. MSP, PE, and DSV use token\-probability scoring rather than sampling\. Thep⁡\(True\)p\(\\mathrm\{True\}\)baseline also uses token\-probability scoring for its final risk score, with temperature1\.01\.0used to generate brainstorm responses\.

#### Maximum Softmax Probability and Predictive Entropy\.

For each stance classcc, we compute the sum of teacher\-forced token log\-probabilities of its label verbalizer, without length normalization\. Applying softmax to these scores givesp⁡\(c∣x\)p\(c\\mid x\)\. Following[Hendrycks and Gimpel \(2017\)](https://arxiv.org/html/2610.29350#bib.bib16), the error\-risk scores are

RMSP\\displaystyle R\_\{\\mathrm\{MSP\}\}=1−maxc⁡p⁡\(c∣x\),\\displaystyle=1\-\\max\_\{c\}p\(c\\mid x\),\(8\)RPE\\displaystyle R\_\{\\mathrm\{PE\}\}=−∑cp\(c∣x\)logp\(c∣x\)\.\\displaystyle=\-\\sum\_\{c\}p\(c\\mid x\)\\log p\(c\\mid x\)\.\(9\)Both scores are computed from the frozen base model\.

#### Prediction Alignment for MSP and PE\.

MSP and Predictive Entropy use the teacher\-forced class\-verbalizer distribution rather than the frozen greedy\-decoded label directly\. To assess alignment, we compared the greedy\-decoded prediction with the argmax of the verbalizer probabilities\. Across all 12 dataset–model settings, 20 mismatches were observed among 15,618 test examples \(0\.128%\)\. Thus, the two prediction procedures agree for nearly all evaluated examples, although MSP and PE quantify uncertainty under the verbalizer distribution rather than confidence in the frozen greedy prediction\.

#### p⁡\(True\)p\(\\mathrm\{True\}\)\.

We adapt Kadavath\-style self\-evaluation using the open Semantic Uncertainty implementation[Kadavath et al\. \(2022\)](https://arxiv.org/html/2610.29350#bib.bib17);[Farquhar et al\. \(2024\)](https://arxiv.org/html/2610.29350#bib.bib7)\. The candidate answer is the frozen base predictiony^\\hat\{y\}\. The procedure uses 20 TRAIN\-only few\-shot demonstrations \(seed 2026092817\) and ten brainstormed stance responses\. No TEST gold labels are used in prompt construction\.

Following the reference implementation’sp\_false\_fixedconvention, error risk is

Rp⁡\(True\)=1−P⁡\(A∣q\),R\_\{p\(\\mathrm\{True\}\)\}=1\-P\(\\texttt\{ A\}\\mid q\),\(10\)whereqqis the complete A/B self\-evaluation prompt\. We extract the raw\-completion token probability without chat\-template wrapping or renormalization over A/B alternatives\. All reported results use the corrected implementation\.

#### Direct Self\-Verification\.

DSV is a separate direct\-correctness control\. The base model assesses whether its predicted stance is correct using a True/False prompt\. Unlikep⁡\(True\)p\(\\mathrm\{True\}\), DSV normalizes the two answer probabilities:

RDSV\\displaystyle R\_\{\\mathrm\{DSV\}\}=1−P⁡\(True\),\\displaystyle=1\-P\(\\mathrm\{True\}\),\(11\)P⁡\(True\)\\displaystyle P\(\\mathrm\{True\}\)=eℓTrueeℓTrue\+eℓFalse\.\\displaystyle=\\frac\{e^\{\\ell\_\{\\mathrm\{True\}\}\}\}\{e^\{\\ell\_\{\\mathrm\{True\}\}\}\+e^\{\\ell\_\{\\mathrm\{False\}\}\}\}\.Here,ℓTrue\\ell\_\{\\mathrm\{True\}\}andℓFalse\\ell\_\{\\mathrm\{False\}\}are the corresponding token log\-probabilities\.

#### Label Entropy and Variation Ratio\.

Letncn\_\{c\}be the number of valid sampled responses assigned to classcc, and letnvalid=∑cncn\_\{\\mathrm\{valid\}\}=\\sum\_\{c\}n\_\{c\}\. Label Entropy \(LE\) and Variation Ratio \(VR\) are

RLE\\displaystyle R\_\{\\mathrm\{LE\}\}=−∑cncnvalidlogncnvalid,\\displaystyle=\-\\sum\_\{c\}\\frac\{n\_\{c\}\}\{n\_\{\\mathrm\{valid\}\}\}\\log\\frac\{n\_\{c\}\}\{n\_\{\\mathrm\{valid\}\}\},\(12\)RVR\\displaystyle R\_\{\\mathrm\{VR\}\}=1−maxc⁡ncM\.\\displaystyle=1\-\\frac\{\\max\_\{c\}n\_\{c\}\}\{M\}\.\(13\)LE uses valid parsed labels, whereas VR retains the planned sample countMMin its denominator\.

#### Semantic Entropy\.

Semantic Entropy \(SE\)[Kuhn et al\. \(2023\)](https://arxiv.org/html/2610.29350#bib.bib18);[Farquhar et al\. \(2024\)](https://arxiv.org/html/2610.29350#bib.bib7)usesM=5M=5stochastic generations from the base stance prompt\. Generated responses are mapped to canonical stance labels when parsing succeeds; otherwise, the original decoded response is retained, with empty responses represented explicitly\. No response is excluded from SE clustering solely because it cannot be parsed as a stance label\.

Semantic relationships are evaluated using DeBERTa\-v2\-xlarge\-MNLI \([https://huggingface\.co/microsoft/deberta\-v2\-xlarge\-mnli](https://huggingface.co/microsoft/deberta-v2-xlarge-mnli)\), with the input text and target included in the NLI context\. Bidirectional NLI judgments group semantically equivalent responses\. Sample sequence likelihoods are computed from token log\-probabilities, normalized across the sampled responses, and summed within semantic clusters\. The SE risk score is

RSE=−∑keℓkℓk,R\_\{\\mathrm\{SE\}\}=\-\\sum\_\{k\}e^\{\\ell\_\{k\}\}\\ell\_\{k\},\(14\)whereℓk\\ell\_\{k\}denotes the normalized log\-probability mass of semantic clusterkk\.

#### Evidential Semantic Entropy\.

Evidential Semantic Entropy \(EVSE\)[Kunitomo\-Jacquin et al\. \(2026\)](https://arxiv.org/html/2610.29350#bib.bib19)uses the same sampled and parsed response set and NLI model, but organizes pairwise entailment relationships into a hierarchy\. Entailment cycles are merged, and the resulting structure includes nodes representing unresolved or uncommitted evidence\. Probability masses are assigned from the sequence likelihoods of unique observed responses; residual mass is allocated to the ignorance set\. The resulting basic belief assignment is transformed into a pignistic probability distribution, whose entropy provides the reported uncertainty score\.

#### Same\-Model Comparative Evidence\.

This control replaces the fixed external verifier with the corresponding base model\. The base model generates prediction\-blind classwise ordinal evidence using the same evidence scale, prompt, and output schema as B\-CARE\. Local generation uses greedy decoding \(do\_sample=false,max\_new\_tokens=96\); the OpenRouter same\-model path uses temperature0\.00\.0andmax\_tokens=80\. The risk remains

Rsame=AB−SB\.R\_\{\\mathrm\{same\}\}=A\_\{B\}\-S\_\{B\}\.\(15\)This comparison preserves the evidence rubric, prediction\-blind prompting, and relative\-evidence ranking rule while changing the verifier model\. However, differences in model architecture, inference backend, generation settings, and chat formatting prevent attributing the observed performance differences exclusively to verifier capacity\. The controlled ablations of prediction blindness and graded versus binary evidence provide complementary evidence for the contributions of B\-CARE’s design\.

#### Evidence\-Label Disagreement\.

Using the frozen external\-verifier evidence vector𝐄\\mathbf\{E\}, we define

RLD=\{0,Ey^\>maxc≠y^⁡Ec,1,otherwise\.R\_\{\\mathrm\{LD\}\}=\\begin\{cases\}0,&E\_\{\\hat\{y\}\}\>\\max\_\{c\\neq\\hat\{y\}\}E\_\{c\},\\\\ 1,&\\text\{otherwise\}\.\\end\{cases\}\(16\)Ties at the maximum evidence level count as disagreement\. Unlike B\-CARE, this binary control discards the magnitude of comparative evidence\.

#### Cross\-Base Disagreement\.

Lety^\\hat\{y\}be the evaluated base prediction, and leto1,o2o\_\{1\},o\_\{2\}be the predictions of the other two base models\. Cross\-Base Disagreement \(CBD\) assigns the following ordinal risk:

RCBD=\{0,o1=o2=y^,1,∑i=12𝕀\[oi=y^\]=1,2,o1=o2≠y^,3,\|\{y^,o1,o2\}\|=3\.R\_\{\\mathrm\{CBD\}\}=\\begin\{cases\}0,&o\_\{1\}=o\_\{2\}=\\hat\{y\},\\\\ 1,&\\sum\_\{i=1\}^\{2\}\\mathbb\{I\}\[o\_\{i\}=\\hat\{y\}\]=1,\\\\ 2,&o\_\{1\}=o\_\{2\}\\neq\\hat\{y\},\\\\ 3,&\|\\\{\\hat\{y\},o\_\{1\},o\_\{2\}\\\}\|=3\.\\end\{cases\}\(17\)Here,𝕀⁡\[⋅\]\\mathbb\{I\}\[\\cdot\]is the indicator function\. CBD uses only hard predictions, without probability or evidence scores\.

#### External Direct Correctness Verification\.

EDCV provides the external verifier with the target, input text, and base predictiony^\\hat\{y\}\. The verifier returns a graded correctness assessmentC∈\{0,1,2,3\}C\\in\\\{0,1,2,3\\\}, yielding

REDCV=3−C\.R\_\{\\mathrm\{EDCV\}\}=3\-C\.\(18\)EDCV uses the external Llama verifier with temperature0\.00\.0andmax\_tokens=16\. Unlike B\-CARE, EDCV explicitly conditions on the base prediction and assesses its correctness directly\.

#### Prediction\-Aware Comparative Evidence\.

PACE retains the external verifier, classwise evidence scale, output schema, and comparative riskAB−SBA\_\{B\}\-S\_\{B\}, but additionally reveals the base predictiony^\\hat\{y\}in the verifier prompt\. It uses temperature0\.00\.0andmax\_tokens=80\. Thus, the intended difference from prediction\-blind B\-CARE is exposure ofy^\\hat\{y\}before evidence generation\.

#### Evidence Components and Alternative Rankings\.

For frozen evidence𝐄\\mathbf\{E\}, letSB=Ey^S\_\{B\}=E\_\{\\hat\{y\}\}andAB=maxc≠y^⁡EcA\_\{B\}=\\max\_\{c\\neq\\hat\{y\}\}E\_\{c\}\. We compare the primary scoreAB−SBA\_\{B\}\-S\_\{B\}with support\-only−SB\-S\_\{B\}and alternative\-onlyABA\_\{B\}\.

As an additional ranking sensitivity analysis, we consider normalized evidence share:

Rshare=1−Ey^∑cEc\.R\_\{\\mathrm\{share\}\}=1\-\\frac\{E\_\{\\hat\{y\}\}\}\{\\sum\_\{c\}E\_\{c\}\}\.\(19\)When∑cEc=0\\sum\_\{c\}E\_\{c\}=0, normalized evidence share uses the uniform class share\.

#### Shared Evaluation Conventions\.

AUROC calculations use midranks for tied risk scores\. ErrorCapture uses fractional allocation within tied risk groups at the selection boundary\. Paired bootstrap comparisons use 10,000 resamples unless a method\-specific protocol specifies otherwise\.

#### Tie Handling in AURC and E\-AURC\.

For risk–coverage evaluation, instances are ranked in ascending order of estimated error risk\. When multiple instances receive identical risk scores, ties are handled using expected fractional allocation\. Specifically, for a group ofggtied instances containingeeerrors, each successive coverage step receives an expected error contribution ofe/ge/g\. AURC is computed by averaging the selective error rate across all coverage levels\. E\-AURC is obtained by subtracting the oracle AURC, which ranks all correct predictions before incorrect predictions\. This treatment avoids dependence on arbitrary ordering within tied groups\.

### C\.1Additional Error\-Ranking Controls

We evaluate four complementary controls to further examine the contributions of classwise evidence, graded scoring, and chance\-level ranking\. All comparisons cover the complete set of 12 dataset–model settings across four benchmarks and three base models\.

#### Same\-Model Label Disagreement\.

This control uses prediction\-blind ordinal classwise evidence generated by the corresponding base model\. An instance is assigned risk zero only when its frozen base prediction uniquely maximizes the classwise evidence\. All other cases, including evidence ties, receive risk one\. Unlike Same\-Model Comparative Evidence, which retains the graded relative\-evidence score, this control discards evidence magnitude and uses binary disagreement\.

#### Llama Label Disagreement\.

This control compares the frozen base\-model stance prediction with a separately generated stance label from the external Llama verifier\. Unlike Evidence\-Label Disagreement, which uses the ordinal classwise evidence vector and a unique\-maximum rule, this control compares two discrete stance labels directly\.

#### Random Ranking\.

We assign independent risk scores sampled from a uniform distribution and calculate the mean error\-ranking AUROC across 50 fixed random seeds\. This provides a chance\-level reference under the same frozen test predictions and error labels\.

#### Normalized Evidence Share\.

This control uses the same prediction\-blind classwise evidence as B\-CARE but replaces the original comparative score with

Rshare​\(B,x\)=1−Ey^B​\(x\)∑c∈𝒴Ec​\(x\)\.R\_\{\\mathrm\{share\}\}\(B,x\)=1\-\\frac\{E\_\{\\hat\{y\}\_\{B\}\}\(x\)\}\{\\sum\_\{c\\in\\mathcal\{Y\}\}E\_\{c\}\(x\)\}\.\(20\)When the total evidence is zero, the predicted\-class evidence share is defined as1/\|𝒴\|1/\|\\mathcal\{Y\}\|\. This variant tests whether normalized support for the predicted class provides an effective alternative to the originalAB−SBA\_\{B\}\-S\_\{B\}ranking score\.

Table T3:Additional error\-ranking controls across 12 dataset–model settings\. AUROC values are macro\-averaged equally across the 12 settings\.Δ\\Deltadenotes the difference between the overall B\-CARE mean AUROC \(0\.7346\) and the corresponding control mean AUROC\. Random Ranking averages 50 fixed\-seed runs\.B\-CARE achieves higher mean AUROC than all four controls across the 12 evaluated dataset–model settings\.

Compared with Same\-Model LD, B\-CARE achieves a mean AUROC improvement of 0\.1490 \(0\.7346 versus 0\.5856\)\. This result indicates that B\-CARE’s graded external comparative evidence provides a stronger error\-ranking signal than binary disagreement derived from the corresponding base model under the evaluated conditions\. However, this comparison also changes verifier identity and inference environment and therefore does not isolate the contribution of evidence granularity alone\.

Compared with Llama LD, B\-CARE achieves a mean AUROC improvement of 0\.0642 \(0\.7346 versus 0\.6704\)\. Llama LD uses a separately generated external Llama stance label rather than graded classwise evidence\. Consequently, this comparison changes the form of the verifier output and does not isolate evidence\-score granularity alone\.

Random Ranking achieves approximately chance\-level performance, with a mean AUROC of 0\.5008\. B\-CARE improves upon this control by 0\.2338\.

The normalized evidence\-share variant achieves a mean AUROC of 0\.6940, compared with 0\.7346 for B\-CARE, corresponding to an improvement of 0\.0406\. Although B\-CARE performs better on average, normalized evidence\-share ranking can perform better in individual settings, indicating that the advantage of the original comparative score is setting\-dependent\.

## Appendix DStatistical Significance Analysis

We assess the statistical significance of AUROC differences using an example\-level paired nonparametric bootstrap within each dataset–model setting\. For each baseline, we computeΔ​AUROC=AUROC⁡\(B\-CARE\)−AUROC⁡\(baseline\)\\Delta\\mathrm\{AUROC\}=\\mathrm\{AUROC\}\(\\textsc\{B\-CARE\}\)\-\\mathrm\{AUROC\}\(\\mathrm\{baseline\}\)and obtain percentile 95% confidence intervals from bootstrap resampling \(primarilyB=10,000B=10\{,\}000\)\. A significant win or loss is recorded when the confidence interval lies entirely above or below zero, respectively; otherwise, the difference is classified as nonsignificant\. No correction for multiple comparisons is applied\.

Table[T4](https://arxiv.org/html/2610.29350#A4.T4)summarizes the significance outcomes across all 156 comparisons\. B\-CARE achieves 122 significant wins, 11 significant losses, and 23 nonsignificant differences\. All significant losses occur against confidence\-based or self\-verification baselines: MSP, Predictive Entropy,p⁡\(True\)p\(\\mathrm\{True\}\), and Self\-Verification\. In contrast, no significant losses are observed against the remaining nine baselines\. These results indicate broad improvements over evidence\-based alternatives, while highlighting the complementary strengths of confidence\-based error ranking\. Nonsignificant differences should not be interpreted as evidence of equivalence\.

BaselineWinLossNSMSP444Predictive Entropy444p⁡\(True\)p\(\\mathrm\{True\}\)822Self\-Verification912Direct Verification1200Prediction\-Aware Evidence705Same\-Model Evidence1101Evidential SE1200Semantic Entropy1101Label Entropy1200Variation Ratio1200Cross\-Model Disagreement903Evidence\-Label Disagreement1101Total1221123Table T4:Summary of 156 paired bootstrap AUROC comparisons across 12 dataset–model settings\. Win and Loss indicate significant positive and negative differences, respectively; NS indicates a nonsignificant difference under unadjusted 95% confidence intervals\.
## Appendix EComplementarity Analysis

![Refer to caption](https://arxiv.org/html/2610.29350v1/figures/BCARE_complementarity_crisp_labels1.png)Figure A1:Complementarity between B\-CARE and Predictive Entropy \(PE\) across 12 dataset–model settings\.Top:AUROC of B\-CARE, PE, and the exploratory equal\-weight hybrid\.Bottom:Errors captured exclusively by B\-CARE, exclusively by PE, or by both within separate top\-20% review sets\. Segment heights represent percentages of all errors; labels indicate error counts\.We provide the per\-setting results underlying the complementarity analysis in Section[5\.4](https://arxiv.org/html/2610.29350#S5.SS4)\. For each dataset–model setting, B\-CARE and Predictive Entropy \(PE\) independently rank test instances by estimated error risk\. At a 20% review budget, we compare the sets of actual errors identified by their respective top\-ranked instances\.

We partition captured errors into three mutually exclusive groups: errors identified only by B\-CARE, only by PE, and by both methods\. The union capture rate is the number of distinct errors identified by either method divided by the total number of actual errors\. Jaccard overlap is computed over the two sets of captured errors\. Because the union combines two independently selected review sets, it does not represent performance under a single 20% review budget\.

To examine whether the two ranking signals can be combined, we also evaluate an exploratory fixed hybrid formed by averaging their within\-setting normalized ranks with equal weights\. No fusion weights are tuned\. This hybrid is a post\-hoc diagnostic and is not part of the frozen B\-CARE method\.

DatasetModelB\-CAREPEHybrid𝚫𝐁\\boldsymbol\{\\Delta\_\{\\mathrm\{B\}\}\}𝚫𝐏𝐄\\boldsymbol\{\\Delta\_\{\\mathrm\{PE\}\}\}SemEvalPhi0\.7970\.6120\.739−0\.058\-0\.058\+0\.127\+0\.127Mistral0\.8490\.6630\.804−0\.045\-0\.045\+0\.141\+0\.141Gemma0\.7380\.6690\.733−0\.005\-0\.005\+0\.064\+0\.064VaccinesPhi0\.6980\.7260\.742\+0\.044\+0\.044\+0\.017\+0\.017Mistral0\.6720\.7180\.723\+0\.051\+0\.051\+0\.005\+0\.005Gemma0\.7240\.7100\.750\+0\.026\+0\.026\+0\.040\+0\.040COVIDPhi0\.6900\.7730\.775\+0\.085\+0\.085\+0\.002\+0\.002Mistral0\.6840\.7520\.754\+0\.070\+0\.070\+0\.002\+0\.002Gemma0\.7070\.7980\.793\+0\.086\+0\.086−0\.005\-0\.005P\-StancePhi0\.7620\.7600\.789\+0\.027\+0\.027\+0\.028\+0\.028Mistral0\.7660\.7360\.781\+0\.016\+0\.016\+0\.045\+0\.045Gemma0\.7280\.7100\.740\+0\.012\+0\.012\+0\.030\+0\.030Macro mean–0\.7350\.7190\.760\+0\.026\+0\.026\+0\.041\+0\.041Table T5:Per\-setting AUROC of B\-CARE, Predictive Entropy \(PE\), and the exploratory fixed equal\-weight hybrid\.ΔB\\Delta\_\{\\mathrm\{B\}\}andΔPE\\Delta\_\{\\mathrm\{PE\}\}denote hybrid AUROC minus B\-CARE and PE AUROC, respectively\. Bold indicates the highest AUROC in each setting\. Values are rounded to three decimals\.DatasetModel𝒌\\boldsymbol\{k\}ErrorsB\-onlyPE\-onlyBothUnion \(%\)JaccardSemEvalPhi250460124538757\.40\.330Mistral250550135419248\.70\.343Gemma25035887477859\.20\.368VaccinesPhi20033043556248\.50\.388Mistral20030133585949\.80\.393Gemma20037763596549\.60\.348COVIDPhi16027049534153\.00\.287Mistral16025144484855\.80\.343Gemma16028141535653\.40\.373P\-StancePhi431385604514765\.50\.583Mistral431391754213765\.00\.539Gemma431328503411460\.40\.576Macro mean55\.50\.406Table T6:Overlap of actual errors captured by B\-CARE and Predictive Entropy \(PE\) within separately selected top\-20% review sets\.kkis the number of reviewed instances per method, and Errors is the total number of incorrect base\-model predictions\. B\-only, PE\-only, and Both are mutually exclusive error counts\. Union is the percentage of all errors captured by either method; Jaccard measures overlap between the two captured\-error sets\. Macro means are unweighted across 12 settings\.Table[T5](https://arxiv.org/html/2610.29350#A5.T5)and Figure[A1](https://arxiv.org/html/2610.29350#A5.F1)\(top\) show that the hybrid outperforms B\-CARE in nine settings, with the three exceptions occurring on SemEval\. The hybrid achieves the highest AUROC among the three approaches in eight settings\. Table[T6](https://arxiv.org/html/2610.29350#A5.T6)and Figure[A1](https://arxiv.org/html/2610.29350#A5.F1)\(bottom\) show a mean captured\-error Jaccard overlap of 0\.406 and a mean union capture rate of 55\.5%\. Together, these findings suggest that the two methods identify partially distinct errors, although their combination does not consistently improve ranking performance\.

## Appendix FComponent Ablation: Detailed Results

Table[T7](https://arxiv.org/html/2610.29350#A6.T7)reports the AUROC of the full B\-CARE score \(A−SA\-S\), support\-only \(−S\-S\), and alternative\-only \(AA\) across all 12 dataset–model settings\. We also report paired AUROC differences and 95% confidence intervals based on 10,000 paired bootstrap resamples\. Intervals are unadjusted for multiple comparisons\.

DatasetModelB\-CARE−S\-SAAΔ−S\\Delta\_\{\-S\}ΔA\\Delta\_\{A\}95% CI \(Δ−S\\Delta\_\{\-S\}\)95% CI \(ΔA\\Delta\_\{A\}\)SemEvalPhi0\.79690\.78230\.7933\+0\.0146\+0\.0036\[0\.0093, 0\.0203\]\[\-0\.0058, 0\.0131\]Mistral0\.84900\.83720\.8521\+0\.0119\-0\.0031\[0\.0081, 0\.0160\]\[\-0\.0096, 0\.0036\]Gemma0\.73780\.71910\.7419\+0\.0187\-0\.0041\[0\.0119, 0\.0262\]\[\-0\.0143, 0\.0065\]VaccinesPhi0\.69840\.70110\.6162\-0\.0026\+0\.0822\[\-0\.0103, 0\.0052\]\[0\.0637, 0\.1014\]Mistral0\.67230\.67110\.6093\+0\.0013\+0\.0630\[\-0\.0059, 0\.0088\]\[0\.0452, 0\.0809\]Gemma0\.72430\.72280\.6427\+0\.0015\+0\.0815\[\-0\.0066, 0\.0095\]\[0\.0632, 0\.0999\]COVIDPhi0\.69030\.67720\.6692\+0\.0131\+0\.0211\[0\.0045, 0\.0225\]\[0\.0051, 0\.0376\]Mistral0\.68400\.68000\.6635\+0\.0039\+0\.0205\[\-0\.0030, 0\.0115\]\[0\.0060, 0\.0358\]Gemma0\.70680\.69710\.6795\+0\.0097\+0\.0273\[0\.0020, 0\.0181\]\[0\.0118, 0\.0438\]P\-StancePhi0\.76160\.76100\.6591\+0\.0007\+0\.1026\[\-0\.0027, 0\.0042\]\[0\.0800, 0\.1254\]Mistral0\.76550\.76060\.6780\+0\.0049\+0\.0876\[0\.0010, 0\.0090\]\[0\.0656, 0\.1097\]Gemma0\.72820\.72580\.6290\+0\.0024\+0\.0992\[\-0\.0018, 0\.0068\]\[0\.0736, 0\.1246\]Macro mean0\.73460\.72790\.6861\+0\.0067\+0\.0484——Table T7:Detailed component ablation across 12 dataset–model settings \(AUROC\)\.Δ−S\\Delta\_\{\-S\}andΔA\\Delta\_\{A\}denote the AUROC differences between B\-CARE and the support\-only and alternative\-only variants, respectively\. Confidence intervals are based on 10,000 paired bootstrap resamples and are unadjusted for multiple comparisons\. Bold indicates the highest AUROC per setting\.
## Appendix GGraded Evidence versus Evidence\-Label Disagreement

We compare B\-CARE with Evidence\-Label Disagreement \(LD\), a binary baseline using identical frozen, prediction\-blind verifier outputs, without additional LLM calls\. For base predictiony^B\\hat\{y\}\_\{B\}and ordinal classwise evidenceEcE\_\{c\}, the two risk scores are

RB\\displaystyle R\_\{B\}=maxc≠y^B⁡Ec−Ey^B,\\displaystyle=\\max\_\{c\\neq\\hat\{y\}\_\{B\}\}E\_\{c\}\-E\_\{\\hat\{y\}\_\{B\}\},\(21\)RLD\\displaystyle R\_\{\\mathrm\{LD\}\}=𝕀\[Ey^B≤maxc≠y^BEc\]\.\\displaystyle=\\mathbb\{I\}\\\!\\left\[E\_\{\\hat\{y\}\_\{B\}\}\\leq\\max\_\{c\\neq\\hat\{y\}\_\{B\}\}E\_\{c\}\\right\]\.Thus, LD flags disagreement unless the base prediction has uniquely maximal evidence; ties are treated as disagreement\.

#### AUROC comparison\.

Table[T8](https://arxiv.org/html/2610.29350#A7.T8)reports results across four datasets and three base models\. B\-CARE outperforms LD in all 12 settings, with all unadjusted 95% paired\-bootstrap confidence intervals excluding zero\. The macro\-average AUROC increases from approximately 0\.6543 to 0\.7346, corresponding to a mean improvement of approximately 0\.0803\.

Table T8:AUROC comparison of graded B\-CARE and Evidence\-Label Disagreement across 12 dataset–model settings\.Δ\\Deltadenotes B\-CARE minus LD\. Confidence intervals are unadjusted 95% paired\-bootstrap intervals \(10,000 resamples\)\.
#### Precision–recall and review budgets\.

The advantage extends beyond AUROC \(Table[T9](https://arxiv.org/html/2610.29350#A7.T9); Figure[A2](https://arxiv.org/html/2610.29350#A7.F2)\)\. B\-CARE improves AUPRC in all 12 settings and achieves higher mean expected error recall at 10%, 20%, and 30% review budgets\. Expected recall accounts for random tie\-breaking within equal\-risk groups\.

Table T9:Macro\-average gains of B\-CARE over LD\. Wins count settings with positive point\-estimate differences; CI\>0\>0counts unadjusted 95% paired\-bootstrap intervals excluding zero in favor of B\-CARE\.![Refer to caption](https://arxiv.org/html/2610.29350v1/figures/BCARE_binary_appendix_final.png)Figure A2:Macro\-average improvement in expected error recall \(percentage points\) for B\-CARE over Evidence\-Label Disagreement at fixed review budgets\. Results are averaged equally across 12 settings: SemEval, Vaccines, COVID\-19, and P\-Stance, each evaluated with Phi, Mistral, and Gemma\. At a 50% review budget, mean recall is 0\.7426 for B\-CARE versus 0\.6354 for LD; this comparison is descriptive\.
#### Within\-group ranking and tie sensitivity\.

B\-CARE retains ranking information even among instances assigned the same LD score: its mean within\-group AUROC is 0\.6469 for LD=0=0and 0\.5764 for LD=1=1\. Furthermore, changing the tie policy to count a prediction as agreement whenever it belongs to the maximal\-evidence set does not alter the conclusion: B\-CARE outperforms LD in all 12 settings under both tie conventions\.

## Appendix HMatched Self\-Hosted Verifier Evaluation

#### Motivation and experimental setting\.

We conducted a supplementary matched comparison between B\-CARE and LlamaProb, a probability\-based verifier comparator, using a self\-hosted Llama\-3\.3\-70B\-Instruct model on RunPod\. Dedicated GPU resources were used because availability and scheduling contention on the university’s shared computing infrastructure constrained timely execution\. This decision was operational rather than motivated by benchmark performance\.

Both scoring methods were evaluated using the same local verifier deployment, the same test instances, and identical frozen base\-model predictions\. The local Hugging Face checkpoint was Llama\-3\.3\-70B\-Instruct, with a revision identifier beginning6f6073b4\. The frozen base models were Phi\-4, Mistral\-Small\-24B\-Instruct\-2501, and Gemma\-3\-27B\.

This matched evaluation is distinct from the primary OpenRouter\-based experiments\. In particular, the RunPod verifier evidence and the original OpenRouter evidence should not be treated as interchangeable\. The comparison is intended to examine alternative error\-ranking signals under a common local inference environment, rather than differences between inference providers\.

#### Error\-risk definitions\.

For an input textxx, stance targettt, and frozen base\-model predictiony^B\\hat\{y\}\_\{B\}, B\-CARE obtains prediction\-blind classwise evidence scoresEc​\(x,t\)∈\{0,1,2,3\}E\_\{c\}\(x,t\)\\in\\\{0,1,2,3\\\}\.

Its error\-risk score is

RB​\(x,t,y^B\)=maxc≠y^B⁡Ec​\(x,t\)−Ey^B​\(x,t\)\.R\_\{\\mathrm\{B\}\}\(x,t;\\hat\{y\}\_\{B\}\)=\\max\_\{c\\neq\\hat\{y\}\_\{B\}\}E\_\{c\}\(x,t\)\-E\_\{\\hat\{y\}\_\{B\}\}\(x,t\)\.\(22\)
The resulting margin takes seven possible integer values, from−3\-3to\+3\+3\.

LlamaProb instead computes

RLP​\(x,t,y^B\)=1−pV​\(y^B∣x,t\),R\_\{\\mathrm\{LP\}\}\(x,t;\\hat\{y\}\_\{B\}\)=1\-p\_\{V\}\(\\hat\{y\}\_\{B\}\\mid x,t\),\(23\)
wherepVp\_\{V\}denotes the verifier’s closed\-set probability assigned to the frozen base prediction\.

In the audited RunPod implementation, LlamaProb probabilities were obtained using teacher\-forced label log\-probabilities and FP32 softmax normalization\. Gold labels were not used to construct either risk score\.

Higher scores indicate greater estimated error risk for both methods\.

Although both methods use the same verifier, they differ in their elicitation and scoring procedures\. Accordingly, this comparison evaluates two complete error\-ranking approaches rather than isolating the effect of score discretization alone\.

#### Matched AUROC comparison\.

Table[T10](https://arxiv.org/html/2610.29350#A8.T10)and Figure[A3](https://arxiv.org/html/2610.29350#A8.F3)presents the AUROC results across four datasets and three frozen base models\.

B\-CARE achieves higher AUROC in nine of the twelve settings, covering all three base models on SemEval, Vaccines, and P\-Stance\. The improvements range from0\.0050\.005to0\.0080\.008on SemEval,0\.0270\.027to0\.0390\.039on Vaccines, and0\.0490\.049to0\.0520\.052on P\-Stance\. COVID exhibits the opposite pattern: LlamaProb achieves higher AUROC across all three base models\.

Consequently, despite nine setting\-level wins for B\-CARE, the unweighted mean AUROC across the twelve settings is approximately0\.7600\.760for B\-CARE and0\.7610\.761for LlamaProb\. The number of setting\-level wins is descriptive and does not itself constitute a statistical significance test\.

Table T10:Matched RunPod AUROC comparison between B\-CARE and LlamaProb \(LP\)\.Δ=B​\-​CARE−LP\\Delta=\\mathrm\{B\\text\{\-\}CARE\}\-\\mathrm\{LP\}; positive values favor B\-CARE\. Best AUROC values are shown in bold\. Values are displayed to three decimal places\.![Refer to caption](https://arxiv.org/html/2610.29350v1/figures/AUROC-Difference-Heatmap-Summary.png)Figure A3:AUROC differences between B\-CARE and LlamaProb across four datasets and three frozen base models\. Positive differences favor B\-CARE; negative differences favor LlamaProb\. Blue intensity represents the absolute magnitude of the difference, not statistical significance\.
#### Diagnosing the COVID performance reversal\.

To investigate LlamaProb’s consistent advantage on COVID, we analyzed matched per\-instance outputs for 800 test examples under each of the three frozen base models\. Table[T11](https://arxiv.org/html/2610.29350#A8.T11)reports the AUROC differences, paired bootstrap confidence intervals, and the contribution of B\-CARE score ties\.

LlamaProb outperforms B\-CARE by approximately 0\.094–0\.097 AUROC across the three models\. The 95% confidence intervals, estimated using 10,000 paired bootstrap resamples, exclude zero in all three settings\. An implementation audit independently recomputed the AUROCs and found no discrepancies in the error labels, evaluation procedure, or LlamaProb probability extraction\.

Table T11:COVID diagnostic results from the matched RunPod evaluation \(n=800n=800\)\.Δ\\DeltaAUROC is B\-CARE minus LlamaProb\. Confidence intervals are estimated from 10,000 paired bootstrap resamples\. Tie share is the percentage of the observed AUROC gap attributable to error–correct pairs tied by B\-CARE\. The Gemma difference is calculated from unrounded scores and may differ from subtraction of displayed three\-decimal AUROCs\.
#### Ordinal compression and tied rankings\.

A substantial portion of the observed COVID performance gap is associated with the limited ranking resolution of B\-CARE’s ordinal evidence margin\. Because classwise evidence scores belong to\{0,1,2,3\}\\\{0,1,2,3\\\}, the resulting risk margin has only seven possible values\.

A substantial proportion of COVID instances receives the minimum risk score,RB=−3R\_\{\\mathrm\{B\}\}=\-3: 51\.4% for Phi and Mistral and 48\.3% for Gemma\. In contrast, LlamaProb produces approximately 790 distinct scores among the 800 COVID examples\.

Consequently, approximately 21–24% of error–correct instance pairs are tied under B\-CARE\. These pairs cannot be distinguished by its final risk margin, even when their underlying texts or verifier evidence profiles differ\.

#### Pairwise AUROC decomposition\.

AUROC can be expressed as the fraction of error–correct pairs correctly ordered by risk, with tied pairs receiving half credit:

AUROC=Nwin\+12​NtieN\+​N−,\\mathrm\{AUROC\}=\\frac\{N\_\{\\mathrm\{win\}\}\+\\frac\{1\}\{2\}N\_\{\\mathrm\{tie\}\}\}\{N\_\{\+\}N\_\{\-\}\},\(24\)
whereN\+N\_\{\+\}andN−N\_\{\-\}denote the numbers of incorrect and correct base\-model predictions, respectively\.

For pairs tied under B\-CARE, its pairwise AUROC contribution is necessarily0\.50\.5\. LlamaProb, however, can distinguish these pairs, achieving a conditional pairwise AUROC of approximately 0\.78–0\.80\.

LetNB,tieN\_\{\\mathrm\{B,tie\}\}denote the number of error–correct pairs tied by B\-CARE, and letALP\|B,tieA\_\{\\mathrm\{LP\}\\mid\\mathrm\{B,tie\}\}denote LlamaProb’s conditional pairwise AUROC on those pairs\. Their contribution to the B\-CARE\-minus\-LlamaProb AUROC difference is

Δtie=\(12−ALP\|B,tie\)​NB,tieN\+​N−\.\\Delta\_\{\\mathrm\{tie\}\}=\\left\(\\frac\{1\}\{2\}\-A\_\{\\mathrm\{LP\}\\mid\\mathrm\{B,tie\}\}\\right\)\\frac\{N\_\{\\mathrm\{B,tie\}\}\}\{N\_\{\+\}N\_\{\-\}\}\.\(25\)
This decomposition isolates the observed ranking difference among pairs that B\-CARE assigns equal risk scores\. Such pairs account for 72\.5%, 68\.3%, and 66\.5% of the total AUROC gap for Phi, Mistral, and Gemma, respectively\.

Thus, approximately two\-thirds to three\-quarters of LlamaProb’s COVID advantage arises from comparisons that B\-CARE cannot resolve\.

As an additional diagnostic, preserving B\-CARE’s ordering between distinct risk scores while using LlamaProb to break ties increases AUROC to approximately 0\.77–0\.79\. This hybrid ranking is used solely to quantify within\-bin information loss; it is not proposed as a new prediction\-blind method\.

#### Class\-conditional and target\-specific effects\.

The remaining COVID differences are not explained primarily by the NONE class\. Removing instances with gold or predicted NONE labels increases LlamaProb’s relative advantage, whereas B\-CARE performs slightly better on the gold\-NONE subset\.

A substantial portion of the residual disagreement occurs among instances predicted as AGAINST\. Within this predicted class, AUROC differences frequently favor LlamaProb by approximately 0\.14–0\.16\.

One recurring pattern involves incorrect AGAINST predictions for which the verifier nevertheless assigns strong ordinal evidence to AGAINST\. The resulting B\-CARE margin can be−2\-2or−3\-3, indicating low estimated error risk\. For some of these instances, LlamaProb assigns near\-zero probability to the predicted label, producing a substantially higher error\-risk score\.

The disagreement is especially pronounced for the Face Masks target, where LlamaProb achieves AUROC of approximately 0\.93–0\.94 and its advantage ranges from approximately 0\.11 to 0\.19\.

However, this pattern is not uniform across COVID targets\. School Closures provides a counterexample: both methods exhibit relatively weak discrimination, and B\-CARE achieves a higher AUROC point estimate\.

These findings indicate that the aggregate COVID reversal reflects both ordinal score compression and target\-specific ranking disagreements rather than uniform failure across all targets or stance classes\.

#### Interpretation of the COVID diagnostics\.

The pairwise decomposition identifies score ties as the largest measured contributor to the COVID AUROC gap\. Nevertheless, this accounting is descriptive: it does not establish that discretization alone causes the entire difference between the methods\.

Likewise, the observed AGAINST disagreement does not establish why the verifier assigns particular evidence scores without additional instance\-level annotation\.

The confidence intervals and tie decomposition are specific to the matched RunPod diagnostic evaluation\. Their numerical values should not be assumed to transfer directly to the original OpenRouter\-generated evidence\.

#### Error capture under limited review budgets\.

AUROC measures discrimination across all error–correct pairs, whereas practical error auditing often involves reviewing only a limited number of high\-risk instances\.

Figure[A4](https://arxiv.org/html/2610.29350#A8.F4)compares the mean fraction of errors retrieved at review budgets of 10%, 20%, and 30% across the twelve matched dataset–model settings\.

B\-CARE captures 26\.1%, 44\.4%, and 58\.5% of errors at these budgets, compared with 25\.6%, 42\.8%, and 56\.0% for LlamaProb\. These correspond to absolute improvements of 0\.5, 1\.6, and 2\.5 percentage points, respectively\.

These aggregate differences do not imply that B\-CARE performs better at every budget or within every dataset–model setting\.

![Refer to caption](https://arxiv.org/html/2610.29350v1/figures/RunPod_EC_Clean_No_Shading.png)Figure A4:Mean error capture across twelve matched dataset–model settings at review budgets of 10%, 20%, and 30%\. B\-CARE is shown in purple and LlamaProb in orange\. Higher values indicate better error capture\.Table T12:Aggregate matched RunPod metrics averaged across twelve dataset–model settings\. Arrows indicate the preferred direction\. EC denotes error capture at the specified review budget\.
#### Comparison across ranking criteria\.

Table[T12](https://arxiv.org/html/2610.29350#A8.T12)summarizes the aggregate performance of the two methods across complementary ranking metrics\.

B\-CARE achieves higher mean AUPRC \(0\.603 versus 0\.592\) and higher mean error capture at all three evaluated budgets\. LlamaProb achieves a marginally higher mean AUROC \(0\.761 versus 0\.760\) and lower mean E\-AURC \(0\.108 versus 0\.116\)\.

These results are compatible because the metrics emphasize different ranking properties\. AUROC measures global error–correct pairwise discrimination, whereas AUPRC is sensitive to precision–recall behavior and error prevalence\. Error capture evaluates retrieval within a specified review budget, while E\-AURC summarizes performance across the risk–coverage curve\.

Accordingly, improved error capture at selected budgets does not necessarily imply superior global AUROC\.

The matched results show that B\-CARE’s prediction\-blind ordinal evidence can support competitive error ranking and early error retrieval, while LlamaProb’s continuous scores provide additional discrimination within some B\-CARE risk bins\. Neither method consistently dominates across all reported criteria\.