当评判者意见一致却判断错误时：一致性与准确性的分离是无标签评估的盲点

# When Judges Agree and Are Wrong: Consistency-Accuracy Dissociation as a Blind Spot for Label-Free Evaluation
Source: [https://arxiv.org/html/2610.21128](https://arxiv.org/html/2610.21128)
## When Judges Agree and Are Wrong: Consistency\-Accuracy Dissociation as a Blind Spot for Label\-Free EvaluationThanks:© 2026 The Vanguard Group, Inc\. All rights reserved\. This material is provided for informational purposes only and is not intended to be investment advice or a recommendation to take any particular investment action\.

###### Abstract

Label\-free evaluation methods, such as self\-consistency filtering, majority\-vote aggregation, and agreement\-based reliability scoring, rely on an implicit assumption that consistency can serve as a proxy for correctness when ground\-truth labels are unavailable or expensive to obtain\. In this paper, we test this assumption directly\. Using four financial QA datasets \(multiple\-choice and free\-form\), ten consistency dimensions, and two frontier model families, we measure the joint distribution of consistency and accuracy at the level of individual examples rather than in aggregate\. We find that 60% of all incorrect answers fall in the high\-consistency region \(consistency≥0\.8\\geq 0\.8\), and that 77\.7% of these show complete answer collapse, i\.e\. identical wrong answers across all perturbation variants\. This high\-consistency/incorrect quadrant cannot be detected by any evaluation signal that is built on agreement alone, as the model appears maximally reliable while being wrong\. We show that the phenomenon is robust to the choice of consistency thresholdtt\(it persists fromt=0\.50t=0\.50tot=1\.00t=1\.00\), is shared across model families, and is concentrated in specific intersections of perturbation dimension and domain\. We map where this quadrant occurs, analyze its stability properties, and discuss the implications for evaluation pipelines that use agreement\-based signals in place of human labels\.

## 1Introduction

A growing family of evaluation methods treats consistency as a load\-bearing proxy for correctness\[[22](https://arxiv.org/html/2610.21128#bib.bib26),[8](https://arxiv.org/html/2610.21128#bib.bib21),[23](https://arxiv.org/html/2610.21128#bib.bib25)\]\. Self\-consistency sampling, majority\-vote filtering, and agreement\-based reliability scores are deployed precisely in those settings where ground\-truth labels are unavailable, and the underlying assumption is that a model which agrees with itself across meaning\-preserving perturbations is more likely to be right\. This assumption is rarely tested against ground truth\[[4](https://arxiv.org/html/2610.21128#bib.bib23)\]at the resolution where it matters most: not on average across a benchmark, but conditionally, within the population of wrong answers\. In this paper, we test it at that resolution and find that the assumption fails in the least desirable direction, as the majority of model errors are concentrated inside the high\-consistency region that agreement\-based methods would certify as reliable\.

We situate this test in financial question answering, a domain in which the construct\-validity question is both answerable and consequential\. It is answerable because professionally validated exam items \(CFA111https://www\.cfainstitute\.org/programs/cfa\-program, CMT222https://cmtassociation\.org/\) provide unambiguous gold labels against which the dissociation can be measured\. It is consequential because financial AI deployments are precisely the high\-stakes, regulated settings\[[12](https://arxiv.org/html/2610.21128#bib.bib18)\]in which consistency\-based evaluation carries real operational weight: proprietary data constraints and compliance boundaries frequently prevent ground\-truth labels from being shared\[[11](https://arxiv.org/html/2610.21128#bib.bib16),[13](https://arxiv.org/html/2610.21128#bib.bib17)\], which makes agreement\-based proxies load\-bearing rather than supplementary\. The financial domain additionally provides internal variation that allows us to isolate what drives the dissociation: a difficulty gradient \(CFA Level 1333https://www\.cfainstitute\.org/programs/cfa\-program/candidate\-resources/level\-i\-examvs\.CFA Level 2444https://www\.cfainstitute\.org/programs/cfa\-program/candidate\-resources/level\-ii\-exam\), format differences \(multiple\-choice vs\. free\-form numerical reasoning\), and distinct subject areas within a single professional knowledge domain\. As such, we are able to attribute the concentration of the phenomenon to question complexity and consistency dimension rather than to confounded differences across unrelated benchmarks\.

![Refer to caption](https://arxiv.org/html/2610.21128v1/figures/quadrant1.png)Figure 1:The joint consistency–accuracy space forms four quadrants: Q1 \(high consistency, correct\) corresponds to reliable behavior; Q2 \(low consistency, correct\) corresponds to accurate but inconsistent behavior; Q3 \(low consistency, incorrect\) corresponds to behavior that is unreliable but detectable; and Q4 \(high consistency, incorrect\) corresponds to CBI \(Consistent But Inaccurate\)We conduct our experiments across 4 financial QA datasets \(CFA Level 1andCFA Level 2\[[20](https://arxiv.org/html/2610.21128#bib.bib19)\],CMT\[[20](https://arxiv.org/html/2610.21128#bib.bib19)\], andFinQA\[[2](https://arxiv.org/html/2610.21128#bib.bib20)\]\), 10 consistency dimensions, 2 tested model families \(GPT 5\.5andClaude Opus 4\.5\), and 6 distinct generator\-validator pipeline configurations, measuring the joint consistency–accuracy distribution at the level of individual examples\. We treat the high\-consistency/incorrect quadrant \(Figure[1](https://arxiv.org/html/2610.21128#S1.F1)\) of this distribution as the primary object of study\.

The contributions of this work are as follows\. \(1\) We provide the comprehensive example\-levelcharacterization of the dissociation between consistency and accuracy in financial QA, and show that it is large and asymmetric: almost two\-thirds \(60%\) of all incorrect answers meet or exceed a consistency threshold of 0\.80, which is the standard operating threshold for agreement\-based filters, and 77\.7% of these Consistent But Inaccurate \(CBI\) examples show complete answer collapse, producing identical wrong responses across all perturbation variants\. \(2\) We demonstrate thatthe phenomenon is structuralrather than an artifact of experimental choices: it is threshold\-invariant, with CBI rates moving only from 7\.2% to 5\.05% as the threshold increases from 0\.50 to 1\.00; it is shared across model families, which converge on the same wrong answers with consistency of 1\.0 for specific questions and thereby defeat ensemble\-based and cross\-model agreement strategies; and it is stable across all 6 generator–validator configurations\. \(3\) Wemap where the phenomenon concentrates, showing that CBI rates increase with question complexity \(5\.22% forCFA Level 2compared to 3\.80% forCFA Level 1\) and with answer format \(7\.49% for free\-form numerical reasoning compared to 5\.42% for multiple choice\), while remaining diffuse across perturbation types, with rates between 0\.2% and 9\.7% in nine of ten dimensions and no single dimension dominating\. \(4\) Wequantify the implications for evaluation pipelinesthat substitute agreement\-based signals for human labels, and derive from our analysis two practical recommendations: a low\-cost pre\-deployment diagnostic for estimating quadrant mass, and a structured disclosure format for reporting the failure modes of agreement\-based evaluators\. \(5\) Finally, we establish thedirect practical consequenceof the above findings: consistency\-based filtering measurably improves the retained set, but any pipeline that uses it to certify data quality, to gate deployment decisions, or to substitute for human evaluation retains a majority of its errors inside the certified set, and those retained errors are disproportionately the deterministic ones that further sampling or ensembling cannot remove\. The residual is therefore not incidental noise around a working signal but a stable and largely irreducible component of it\.

## 2Related Work

[Wang et al\. \[23\]](https://arxiv.org/html/2610.21128#bib.bib25)introduce self\-consistency, replacing greedy decoding with sampling of multiple reasoning paths and returning the plurality answer, on the premise that complex problems admit several routes to a single correct answer while errors diverge\.[Taubenfeld et al\. \[22\]](https://arxiv.org/html/2610.21128#bib.bib26)propose CISC, which weights sampled reasoning paths by self\-assessed confidence rather than counting votes equally, matching or beating standard self\-consistency while reducing the required number of paths\. A parallel line of work measures consistency not over samples but over inputs:[Jang et al\. \[7\]](https://arxiv.org/html/2610.21128#bib.bib28)decompose behavioral consistency into negational, symmetric, additive, and semantic sub\-categories, providing the typology our perturbation dimensions follow\.[Tan et al\. \[21\]](https://arxiv.org/html/2610.21128#bib.bib22)identify self\-consistent errors, where a model repeatedly returns the same wrong answer, and show that consistency\-based detectors collapse to worse than chance on them\.[Ding \[4\]](https://arxiv.org/html/2610.21128#bib.bib23)audit self\-consistency and cross\-model agreement as confidence signals and conclude that agreement is a conditional proxy for correctness, with confident errors recurring across providers\.[Norman et al\. \[14\]](https://arxiv.org/html/2610.21128#bib.bib24)conduct a large judge meta\-evaluation and report kappa deflation: exact\-match agreement overstates chance\-corrected discrimination\.[Krumdick et al\. \[10\]](https://arxiv.org/html/2610.21128#bib.bib27)introduce BFF\-Bench, 160 business and finance questions authored by financial professionals, and VERDICTS, expert correctness annotations over 1,200 model responses\. Their central result is that without a reference answer, judges agree with human experts primarily on questions the judges could answer correctly themselves\. This work establishes that consistent errors exist and that agreement overstates validity; what remains unmeasured is the fraction of the*error population*lying inside the certified region, its stability under threshold choice, and its concentration by perturbation type and difficulty\.

## 3Method

Datasets\.We scope our experiments to financial QA for three reasons\. First, it is a domain in which consistency\-based evaluation is frequently used, since regulatory and data\-sharing constraints frequently preclude access to ground\-truth labels and make agreement\-based proxies necessary\. Second, professionally validated exam items \(CFA, CMT\) provide unambiguous gold labels against which the dissociation can be measured; this is a requirement our design depends on, and one that many specialized domains cannot satisfy\. Third, the available datasets vary in difficulty \(Level 1 vs\. Level 2\) and format \(multiple\-choice vs\. free\-form numerical reasoning\), which allows us to analyze the results through these lenses\. We use four financial QA datasets:CFA Level 1andCFA Level 2contain multiple\-choice questions from the Chartered Financial Analyst curriculum, with Level 2 requiring multi\-step quantitative reasoning;CMTcontains Chartered Market Technician exam items covering technical analysis and market structure; andFinQAcontains free\-form numerical reasoning questions that require extraction and calculation over financial tables\. BothCFA Level 1,CFA Level 2andCMTdatasets are accessed through the FINESSE\-Bench benchmark\[[20](https://arxiv.org/html/2610.21128#bib.bib19)\]\. We use subsets of randomly selected 200 examples from each dataset, resulting in 23,170 individual example\-level observations after perturbation expansion, which we explain next\.

Consistency dimensions\.Following the broader taxonomy of consistency phenomena in language models\[[15](https://arxiv.org/html/2610.21128#bib.bib1)\], each base item is perturbed along up to 10 consistency dimensions, each producing 3 variants:option order\(random permutation of answer labels\)\[[27](https://arxiv.org/html/2610.21128#bib.bib15),[16](https://arxiv.org/html/2610.21128#bib.bib14)\],option relabel\(substitution of label identifiers, e\.g\. A/B/C→\\toX/Y/Z\)\[[25](https://arxiv.org/html/2610.21128#bib.bib12),[28](https://arxiv.org/html/2610.21128#bib.bib13)\],paraphrase\(meaning\-preserving restatement of the stem\)\[[5](https://arxiv.org/html/2610.21128#bib.bib2),[3](https://arxiv.org/html/2610.21128#bib.bib3)\],factual\(anchoring with additional true context\)\[[24](https://arxiv.org/html/2610.21128#bib.bib4)\],additive\(insertion of irrelevant but plausible distractor text\)\[[19](https://arxiv.org/html/2610.21128#bib.bib5)\],crosslingual\(translation to Spanish, French, or German\)\[[17](https://arxiv.org/html/2610.21128#bib.bib6)\],symmetric\(swapping symmetric conditions in the problem\)\[[1](https://arxiv.org/html/2610.21128#bib.bib7)\],negation\(semantic inversion of the question, requiring a different correct answer\)\[[9](https://arxiv.org/html/2610.21128#bib.bib8)\],distractor\(rewording of incorrect options\)\[[26](https://arxiv.org/html/2610.21128#bib.bib9),[6](https://arxiv.org/html/2610.21128#bib.bib10)\], andself\-consistency\(repeated presentation with varied reasoning prompts\)\[[23](https://arxiv.org/html/2610.21128#bib.bib25),[8](https://arxiv.org/html/2610.21128#bib.bib21)\]\. Perturbations are generated by a dedicatedgeneratormodel and validated by avalidatormodel before being presented to the tested model, in order to ensure variant quality \(98\.8% validator pass rate across configurations\)\.

Models\.We test six distinct configurations \(Table[A\.1](https://arxiv.org/html/2610.21128#A1.T1)\) that combine twotested models\(GPT 5\.5,Claude Opus 4\.5\), twogenerators\(GPT\-5\.4, Claude Sonnet 4\.5\), and twovalidators\(GPT\-5\.2, Claude Haiku 4\.5\)\. We assign the stronger model from each family to the generator role and the smaller model to the validator role, motivated by evidence that verification is a comparatively easier capability than generation for language models: models frequently produce correct outputs that weaker models can still validate reliably, even when those same weaker models could not have generated the output themselves\[[18](https://arxiv.org/html/2610.21128#bib.bib11)\]\. The factorial structure allows us to assess whether CBI patterns depend on the tested model alone or whether they interact with the perturbation\-generation and validation pipeline\.

Metrics and definitions\.For each example \(a base item×\\timesdimension pair\), we record the following\.Consistencyis an entropy\-based measure of agreement across the baseline response and all variant responses, with range\[0,1\]\[0,1\], where 1 indicates identical answers across all variants\.Accuracyis a binary indicator of whether the model answer matches the gold label\. Combining both, we define an example as CBI as

CBI\(x\)=\[consistency\(x\)≥t\]∧\[model\_answer\(x\)≠gold\(x\)\],\\text\{CBI\}\(x\)=\[\\text\{consistency\}\(x\)\\geq t\]\\wedge\[\\text\{model\\\_answer\}\(x\)\\neq\\text\{gold\}\(x\)\],
with a default threshold oft=0\.80t=0\.80\. In addition, we measureanswer collapse, i\.e\. whether all variant responses produce the identical \(wrong\) label\. This is a stricter condition that indicates deterministic rather than stochastic error\.

The joint consistency–accuracy space forms four quadrants \(Figure[1](https://arxiv.org/html/2610.21128#S1.F1)\): Q1 \(high consistency, correct\) corresponds to reliable behavior; Q2 \(low consistency, correct\) corresponds to accurate but inconsistent behavior; Q3 \(low consistency, incorrect\) corresponds to behavior that is unreliable but detectable; and Q4 \(high consistency, incorrect\) corresponds to CBI, which is not detectable by agreement\-based signals\.

## 4Results and Discussion

We organize our findings around five questions, each answered directly from the experimental setup it required\.

Table 1:Aggregated quadrants summary \(consistency split t = 0\.8\)\.![Refer to caption](https://arxiv.org/html/2610.21128v1/figures/cbi_sensitivity_publication.png)Figure 2:CBI sensitivity, both overall \(left\) and across models \(right\)\.How large is the dissociation between consistency and accuracy, and where does the error mass concentrate?At the standard operating threshold for agreement\-based filters \(t = 0\.80\), 5\.24% of all 23,170 observations fall in the high\-consistency/incorrect quadrant \(Q4, Consistent But Inaccurate, Table[1](https://arxiv.org/html/2610.21128#S4.T1)\)\. This rate understates the practical problem: conditioned on being incorrect, 60% of all errors \(1,213 of 2,014\) fall in Q4, so an error is about 2\.03x more likely to appear in Q4 than in the low\-consistency/incorrect quadrant Q3 \(1,213 vs\. 801\)\. The association between consistency stratum and correctness is highly non\-random \(χ2​\(1,N=23,170\)=2085\.38,p<0\.001\\chi^\{2\}\(1,N=23,170\)=2085\.38,p<0\.001\)\. Using odds ratio as effect size for the 2x2 table, the odds of being incorrect in high\-consistency vs\. low\-consistency strata areO​R=0\.122OR=0\.122, equivalently the low\-consistency stratum has about 8\.20x higher odds of error\. Among CBI examples, 77\.7% exhibit complete answer collapse — identical wrong answers across all three perturbation variants — indicating that the dissociation is not a marginal disagreement a majority vote could resolve, but a deterministic convergence on a single incorrect answer\.

Is CBI an artifact of the consistency threshold, or a structural property of the model’s error distribution?We recompute CBI rates across thresholds from t = 0\.50 to t = 1\.00\. The overall rate moves from 7\.2% to 5\.05%, a compression of only 2\.15% across the full range, and the curve flattens at t = 0\.6 \(Figure[2](https://arxiv.org/html/2610.21128#S4.F2)\)\. This indicates that CBI examples are not borderline cases pushed just over an arbitrary cutoff; the bulk sit at or near perfect consistency regardless of where the threshold is drawn\. The pattern holds separately for each model family, with Claude Opus 4\.5 showing a consistently higher CBI rate \(5\.54–8\.15%\) than GPT\-5\.5 \(4\.64\-6\.4%\) at every threshold tested\.

Does CBI depend on the tested model, or on the perturbation\-generation pipeline?Across the 6 generator\-validator configurations, the overall CBI rate ranges narrowly \(from4\.2%4\.2\\%to6\.3%6\.3\\%\), and switching validator family \(GPT\-5\.2 vs\. Claude Haiku 4\.5\) shifts CBI rates by 0\.57% forClaude Opus 4\.5and 0\.7% forGPT 5\.5, which indicates the phenomenon is a property of the tested model’s learned preferences rather than an artifact of how perturbations were generated or validated\. To test whether cross\-model overlap could be explained by independent failures, we evaluated the common item set \(N=4,264N=4\{,\}264\), where the independence baselinePindep​\(shared error\)=0\.01P\_\{\\text\{indep\}\}\(\\text\{shared error\}\)=0\.01implies an expected shared\-error count ofEindep=44\.92E\_\{\\text\{indep\}\}=44\.92\. Empirically we observe159159shared CBI items \(and282282shared wrong items overall\), i\.e\.3\.54×3\.54\\timesthe independence expectation, and a goodness\-of\-fit chi\-square test strongly rejects the independence account,χ2​\(1\)=292\.76\\chi^\{2\}\(1\)=292\.76,p=1\.24×10−65p=1\.24\\times 10^\{\-65\}\. Because these failures are not independent, cross\-model ensembling — the strongest available mitigation against single\-model unreliability — does not detect them\.

Table 2:Dataset\-level CBI rates and counts\.![Refer to caption](https://arxiv.org/html/2610.21128v1/figures/cbi_heatmap_publication.png)Figure 3:CBI rate \(%\) by model and consistency dimension\.Where does CBI concentrate across domain difficulty, question format, and perturbation type?CBI rate increases with question complexity: 5\.22% forCFA Level 2\(multi\-step quantitative reasoning\) versus 3\.80% forCFA Level 1\(Table[2](https://arxiv.org/html/2610.21128#S4.T2)\)\. Comparing formats,CMT\(multiple\-choice\) shows a lower rate \(5\.42%\) thanFinQA\(free\-form numerical reasoning, 7\.49%\), and this gap does not depend on the model\. Breaking down by perturbation dimension \(Table[A\.2](https://arxiv.org/html/2610.21128#A2.T2)\), CBI is broadly distributed rather than concentrated in any single dimension: rates fall between roughly 3\.5% and 9\.7% for nine of the ten dimensions and for both models, with additive perturbation the highest for each \(9\.65% forGPT 5\.5, 7\.83% forClaude Opus 4\.5\) and factual perturbation the next highest forClaude Opus 4\.5\(9\.48%\)\. Aggregate CBI rates across dimensions are nearly identical for the two models \(5\.42% and 5\.20%\), but the dimension\-level profiles differ\.

Is CBI stochastic noise, or evidence of a stable, mechanistic failure?The 77\.7% answer\-collapse rate among CBI examples \(RQ1\) is the central piece of evidence here: a model producing the identical wrong answer across meaning\-preserving perturbation is not exhibiting noisy disagreement that would wash out under resampling, but a stable, reproducible preference\. The dimension\-level breakdown \(Table[A\.3](https://arxiv.org/html/2610.21128#A2.T3)\) shows that this property is not driven by any single dimension\. Collapse is complete \(100%\) for option order, option relabel, and distractor rewording, remains high for symmetric \(93\.0%\), negation \(85\.7%\), and crosslingual \(79\.5%\) perturbation, and even at the low end retains roughly two\-thirds for factual \(74\.8%\), self\-consistency \(69\.7%\), and paraphrase \(68\.5%\), so that for most CBI examples the model does not merely agree with itself at the level of a consistency score but returns the same specific wrong answer\. That collapse is strongest precisely on the surface\-form dimensions agreement\-based filters are designed to probe is where a consistency signal would be most confidently and most wrongly reassuring\. Negation warrants separate interpretation, since its underlying failure may be an inability to detect semantic inversion rather than the trained\-in wrong\-preference mechanism driving the other dimensions, but the two are operationally indistinguishable to any consistency\-based filter, which is the property that matters for downstream pipeline design\.

### 4\.1Limitations

Our findings are demonstrated within financial QA\. Whether the same quadrant mass and the same concentration patterns hold in other specialized domains, such as legal, biomedical, or engineering QA, remains an empirical question, although the cross\-model convergence we observe suggests that the mechanism is not specific to finance\.

### 4\.2Implications for Evaluation Pipelines

Our results do not show that consistency is uninformative; the concern is rather with what a consistency\-based filter leaves behind\. Such a filter certifies 89\.7% of all items, and because almost all items fall in the high\-consistency stratum, 60% of the total error mass ends up inside the certified set, so that gating buys approximately a one\-third reduction in error rate while leaving most errors in the region it has labeled reliable\. What is consequential about this residual is its character rather than its size: among CBI examples, 77\.7% show complete answer collapse, returning the same specific wrong answer across every perturbation variant, so these are not borderline cases sitting just above the threshold but deterministic preferences against which the two standard responses to unreliable output, raising the threshold and cross\-model ensembling, are both ineffective\.

#### No single dimension to exclude\.

CBI is distributed across dimensions rather than concentrated in any one of them, which has a direct practical consequence: if a single perturbation type drove the dissociation, a pipeline could exclude that dimension and recover a usable signal, but our results indicate that no such dimension exists\. Answer collapse is moreover highest on precisely those surface\-form dimensions that agreement\-based filters are designed to probe, reaching 100% for option order, option relabel, and distractor rewording, which is where a consistency signal is at once most reassuring and most misleading\.

#### Bearing on the inter\-judge agreement question\.

Agreement and correctness are related, but the relationship is conditional on domain and dimension, so the quantity that determines deployment risk is not a single transportable threshold but the error mass that remains above it\. Because our perturbations preserve meaning by construction, the failures we observe are categorically different from formatting sensitivity: the model is not reacting to surface form, but is stably wrong regardless of it, which is a harder case for any surface\-sensitivity mitigation to catch\.

#### Practical recommendations\.

We identify two concrete uses of quadrant analysis in the design of evaluation pipelines\. First, as adiagnostic before deployment, the mass of Q4 should be estimated on a small labeled sample before a consistency\-based filter is trusted in a new domain, rather than assuming that general\-domain calibration transfers\. Second, as a form ofstructured disclosure, the error mass remaining in the certified set should be reported alongside aggregate consistency, since it is this quantity that determines whether an agreement\-based signal is adequate for a given deployment\. We propose three fields for evaluation disclosure templates: the certified\-set error rate, the CBI\-within\-inaccurate rate, and the answer\-collapse rate, stratified by perturbation type\.

## 5Conclusion

Consistency carries real signal about correctness in financial QA, but it is not a substitute for it\. Because almost all items fall in the high\-consistency region, a standard agreement\-based filter certifies 90% of our data while keeping 60% of its errors, and 77\.7% of those retained errors are identical wrong answers repeated across every perturbation variant\. Neither a stricter threshold nor cross\-model ensembling removes them, since the rate is nearly flat across thresholds and the failures are shared across model families\. Consistency\-based gating should therefore be treated as an error\-rate reduction with a largely irreducible residual rather than a stand\-in for ground truth, and we suggest that this residual mass become a standard part of evaluation pipeline disclosure\.

## References

- \[1\]A\. Asai and H\. Hajishirzi\(2020\)Logic\-guided data augmentation and regularization for consistent question answering\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 5642–5650\.Cited by:[§3](https://arxiv.org/html/2610.21128#S3.p2.1)\.
- \[2\]Z\. Chen, W\. Chen, C\. Smiley, S\. Shah, I\. Borova, D\. Langdon, R\. Moussa, M\. Beane, T\. Huang, B\. R\. Routledge,et al\.\(2021\)Finqa: a dataset of numerical reasoning over financial data\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 3697–3711\.Cited by:[§1](https://arxiv.org/html/2610.21128#S1.p3.1)\.
- \[3\]M\. Choi\(2025\)RoParQ: paraphrase\-aware alignment of large language models towards robustness to paraphrased questions\.arXiv preprint arXiv:2511\.21568\.Cited by:[§3](https://arxiv.org/html/2610.21128#S3.p2.1)\.
- \[4\]K\. Ding\(2026\)When llms agree, are they right? auditing self\-consistency and cross\-model agreement as confidence signals\.arXiv preprint arXiv:2607\.08065\.Cited by:[§1](https://arxiv.org/html/2610.21128#S1.p1.1),[§2](https://arxiv.org/html/2610.21128#S2.p1.1)\.
- \[5\]Y\. Elazar, N\. Kassner, S\. Ravfogel, A\. Ravichander, E\. Hovy, H\. Schütze, and Y\. Goldberg\(2021\)Measuring and improving consistency in pretrained language models\.Transactions of the Association for Computational Linguistics9,pp\. 1012–1031\.External Links:[Link](https://aclanthology.org/2021.tacl-1.60/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00410)Cited by:[§3](https://arxiv.org/html/2610.21128#S3.p2.1)\.
- \[6\]A\. Elhady, E\. Agirre, and M\. Artetxe\(2025\)WiCkeD: a simple method to make multiple choice benchmarks more challenging\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 1183–1192\.Cited by:[§3](https://arxiv.org/html/2610.21128#S3.p2.1)\.
- \[7\]M\. Jang, D\. S\. Kwon, and T\. Lukasiewicz\(2022\)BECEL: benchmark for consistency evaluation of language models\.InProceedings of the 29th International Conference on Computational Linguistics,N\. Calzolari, C\. Huang, H\. Kim, J\. Pustejovsky, L\. Wanner, K\. Choi, P\. Ryu, H\. Chen, L\. Donatelli, H\. Ji, S\. Kurohashi, P\. Paggio, N\. Xue, S\. Kim, Y\. Hahm, Z\. He, T\. K\. Lee, E\. Santus, F\. Bond, and S\. Na \(Eds\.\),Gyeongju, Republic of Korea,pp\. 3680–3696\.External Links:[Link](https://aclanthology.org/2022.coling-1.324/)Cited by:[§2](https://arxiv.org/html/2610.21128#S2.p1.1)\.
- \[8\]S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DasSarma, E\. Tran\-Johnson,et al\.\(2022\)Language models \(mostly\) know what they know\.arXiv preprint arXiv:2207\.05221\.Cited by:[§1](https://arxiv.org/html/2610.21128#S1.p1.1),[§3](https://arxiv.org/html/2610.21128#S3.p2.1)\.
- \[9\]N\. Kassner and H\. Schütze\(2020\)Negated and misprimed probes for pretrained language models: birds can talk, but cannot fly\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 7811–7818\.Cited by:[§3](https://arxiv.org/html/2610.21128#S3.p2.1)\.
- \[10\]M\. Krumdick, C\. Lovering, V\. Reddy, S\. Ebner, and C\. Tanner\(2026\)No free labels: limitations of llm\-as\-a\-judge without human grounding\.External Links:2503\.05061,[Link](https://arxiv.org/abs/2503.05061)Cited by:[§2](https://arxiv.org/html/2610.21128#S2.p1.1)\.
- \[11\]G\. K\. Kulatilleke\(2022\)Challenges and complexities in machine learning based credit card fraud detection\.arXiv preprint arXiv:2208\.10943\.Cited by:[§1](https://arxiv.org/html/2610.21128#S1.p2.1)\.
- \[12\]U\. Nedunuri, A\. Das Gupta, and D\. Guha\(2026\)The paradox of explainability vs\. performance in high\-stakes autonomous ai systems: a systematic review of trade\-offs, regulatory gaps, and emerging solutions\.AI Perspectives & Advances8\(1\),pp\. 1\.Cited by:[§1](https://arxiv.org/html/2610.21128#S1.p2.1)\.
- \[13\]C\. Nie, Y\. Liu, and C\. Wang\(2026\)AI application in anti\-money laundering for sustainable and transparent financial systems\.InProceedings of the 2025 International Conference on Artificial Intelligence and Sustainable Development,ICAISD ’25,New York, NY, USA,pp\. 239–251\.External Links:ISBN 9798400721267,[Link](https://doi.org/10.1145/3786484.3786522),[Document](https://dx.doi.org/10.1145/3786484.3786522)Cited by:[§1](https://arxiv.org/html/2610.21128#S1.p2.1)\.
- \[14\]J\. D\. Norman, M\. U\. Rivera, and D\. A\. Hughes\(2026\)Reliability without validity: a systematic, large\-scale evaluation of llm\-as\-a\-judge models across agreement, consistency, and bias\.arXiv preprint arXiv:2606\.19544\.Cited by:[§2](https://arxiv.org/html/2610.21128#S2.p1.1)\.
- \[15\]J\. Novikova, C\. Anderson, B\. Blili\-Hamelin, D\. Rosati, and S\. Majumdar\(2025\)Consistency in language models: current landscape, challenges, and future directions\.InICML 2025 Workshop on Reliable and Responsible Foundation Models,Cited by:[§3](https://arxiv.org/html/2610.21128#S3.p2.1)\.
- \[16\]P\. Pezeshkpour and E\. Hruschka\(2024\)Large language models sensitivity to the order of options in multiple\-choice questions\.InFindings of the Association for Computational Linguistics: NAACL 2024,K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 2006–2017\.External Links:[Link](https://aclanthology.org/2024.findings-naacl.130/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.130)Cited by:[§3](https://arxiv.org/html/2610.21128#S3.p2.1)\.
- \[17\]J\. Qi, R\. Fernández, and A\. Bisazza\(2023\)Cross\-lingual consistency of factual knowledge in multilingual language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 10650–10666\.Cited by:[§3](https://arxiv.org/html/2610.21128#S3.p2.1)\.
- \[18\]J\. Saad\-Falcon, E\. K\. Buchanan, M\. F\. Chen, T\. Huang, B\. McLaughlin, T\. Bhathal, S\. Zhu, B\. Athiwaratkun, F\. Sala, S\. Linderman, A\. Mirhoseini, and C\. Re\(2025\)Weaver: shrinking the generation\-verification gap by scaling compute for verification\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=dRjt4vlYVQ)Cited by:[§3](https://arxiv.org/html/2610.21128#S3.p3.1)\.
- \[19\]F\. Shi, X\. Chen, K\. Misra, N\. Scales, D\. Dohan, E\. H\. Chi, N\. Schärli, and D\. Zhou\(2023\)Large language models can be easily distracted by irrelevant context\.InInternational conference on machine learning,pp\. 31210–31227\.Cited by:[§3](https://arxiv.org/html/2610.21128#S3.p2.1)\.
- \[20\]D\. Stanishevskii, N\. Kamkia, A\. Khoroshilov, D\. Zmitrovich, D\. Kokosinskii, Z\. Hayrapetyan, and A\. Kalmykov\(2026\)FINESSE\-bench: a hierarchical benchmark suite for financial domain knowledge and technical analysis in large language models\.External Links:[Link](https://github.com/LimexAILab/FINESSE-Bench)Cited by:[§1](https://arxiv.org/html/2610.21128#S1.p3.1),[§3](https://arxiv.org/html/2610.21128#S3.p1.1)\.
- \[21\]H\. Tan, F\. Sun, S\. Liu, D\. Su, Q\. Cao, X\. Chen, J\. Wang, X\. Cai, Y\. Wang, H\. Shen, and X\. Cheng\(2025\)Too consistent to detect: a study of self\-consistent errors in LLMs\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 4755–4765\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.238/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.238),ISBN 979\-8\-89176\-332\-6Cited by:[§2](https://arxiv.org/html/2610.21128#S2.p1.1)\.
- \[22\]A\. Taubenfeld, T\. Sheffer, E\. Ofek, A\. Feder, A\. Goldstein, Z\. Gekhman, and G\. Yona\(2025\)Confidence improves self\-consistency in llms\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 20090–20111\.External Links:[Link](http://dx.doi.org/10.18653/v1/2025.findings-acl.1030),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1030)Cited by:[§1](https://arxiv.org/html/2610.21128#S1.p1.1),[§2](https://arxiv.org/html/2610.21128#S2.p1.1)\.
- \[23\]X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou\(2023\)Self\-consistency improves chain of thought reasoning in language models\.External Links:2203\.11171,[Link](https://arxiv.org/abs/2203.11171)Cited by:[§1](https://arxiv.org/html/2610.21128#S1.p1.1),[§2](https://arxiv.org/html/2610.21128#S2.p1.1),[§3](https://arxiv.org/html/2610.21128#S3.p2.1)\.
- \[24\]J\. Xie, K\. Zhang, J\. Chen, R\. Lou, and Y\. Su\(2024\)Adaptive chameleon or stubborn sloth: revealing the behavior of large language models in knowledge conflicts\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 35623–35646\.Cited by:[§3](https://arxiv.org/html/2610.21128#S3.p2.1)\.
- \[25\]Z\. Yang, P\. Jian, and C\. Li\(2025\)Option symbol matters: investigating and mitigating multiple\-choice option symbol bias of large language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 1902–1917\.External Links:[Link](https://aclanthology.org/2025.naacl-long.95/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.95),ISBN 979\-8\-89176\-189\-6Cited by:[§3](https://arxiv.org/html/2610.21128#S3.p2.1)\.
- \[26\]Z\. Zhang, Z\. Jiang, L\. Xu, H\. Hao, and R\. Wang\(2024\)Multiple\-choice questions are efficient and robust llm evaluators\.arXiv preprint arXiv:2405\.11966\.Cited by:[§3](https://arxiv.org/html/2610.21128#S3.p2.1)\.
- \[27\]C\. Zheng, H\. Zhou, F\. Meng, J\. Zhou, and M\. Huang\(2024\)Large language models are not robust multiple choice selectors\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=shr9PXz7T0)Cited by:[§3](https://arxiv.org/html/2610.21128#S3.p2.1)\.
- \[28\]W\. Zhou, Q\. Wang, M\. Xu, M\. Chen, and X\. Duan\(2024\)Revisiting the self\-consistency challenges in multi\-choice question formats for large language model evaluation\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),N\. Calzolari, M\. Kan, V\. Hoste, A\. Lenci, S\. Sakti, and N\. Xue \(Eds\.\),Torino, Italia,pp\. 14103–14110\.External Links:[Link](https://aclanthology.org/2024.lrec-main.1229/)Cited by:[§3](https://arxiv.org/html/2610.21128#S3.p2.1)\.

## Appendix

## Appendix ADetails on Experimental Setup

Table A\.1:Configurations of model setup### A\.1Scoring

Consistency score\.Within\-set agreement of the answer set \(baseline \+ variants\), computed in the canonical frame\. The default MCQ formula is normalized Shannon entropy:1−H/l​o​g​\(n\)1\-H/log\(n\)where n is the set size \(all\-agree→\\to1\.0, all\-distinct→\\to0\.0\)\. Unparsed answers count against consistency; they never agree with anything\.

## Appendix BExtended CBI Breakdown by Dimensions and Models

Table A\.2:CBI rate by model and consistency dimensionTable A\.3:Share of CBI rows with identical samples answers by consistency dimension