When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals

# When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals
Source: [https://arxiv.org/html/2607.08065](https://arxiv.org/html/2607.08065)
Kaihua Ding University of Pennsylvania dkaihua@upenn\.eduThe author’s prior work spans output\-based error estimation\(Ding,[2018](https://arxiv.org/html/2607.08065#bib.bib3)\)and AI\-system evaluation and assessment design\(Ding,[2025a](https://arxiv.org/html/2607.08065#bib.bib4)\)\.

###### Abstract

LLM\-as\-judge\(Zheng et al\.,[2023](https://arxiv.org/html/2607.08065#bib.bib28)\)is increasingly the default for evaluating AI systems in enterprise pipelines, often scaled to ensembles\(Verga et al\.,[2024](https://arxiv.org/html/2607.08065#bib.bib22)\)or ‘mixture\-of\-experts’\(Shazeer et al\.,[2017](https://arxiv.org/html/2607.08065#bib.bib20)\)panels of judges\. These systems share a key assumption: that*consistency*—agreement among judges, or among a model’s own samples—indicates correctness\. We show this assumption is unreliable\. Agreement is not accuracy: a model can agree with itself, and different models can agree with each other, out of shared bias, a memorized heuristic, or an option\-position prior rather than truth\. We ask*when*agreement is nonetheless a usable proxy, in a large\-scale cross\-runner study:5353runners drewK=50K\{=\}50samples for assigned overlapping cases across comparisons of model tier, prompting, and scale on GPQA Diamond and AIME—265,000265\{,\}000samples\. Using majority\-correctness as the deployment label and a hierarchical runner\-clustered bootstrap, agreement is a positive but weak predictor \(ρ\\rho0\.200\.20–0\.590\.59, all positive under item\-clustered resampling\) whose usefulness is*regime\-dependent*: best for unsaturated mid\-tier models and for allocating compute, and worst—over\-confident yet no more accurate—for the most consistent frontier model \(agreement≥0\.8\\geq 0\.8on77%77\\%of GPQA case\-result entries,48%48\\%of those wrong\)\. An exploratory cross\-family check on three Claude tiers shows the same frontier over\-confidence, with confident errors recurring across providers above a marginal\-preserving null\. Self\-consistency is thus a*conditional*proxy for correctness, not a standalone confidence score\. We publicly release the de\-identified per\-run rows and answer distributions\.

When LLMs Agree, Are They Right? Auditing Self\-Consistency and Cross\-Model Agreement as Confidence Signals

Kaihua Ding††thanks:The author’s prior work spans output\-based error estimation\(Ding,[2018](https://arxiv.org/html/2607.08065#bib.bib3)\)and AI\-system evaluation and assessment design\(Ding,[2025a](https://arxiv.org/html/2607.08065#bib.bib4)\)\.University of Pennsylvaniadkaihua@upenn\.edu

## 1Introduction

Large language models are increasingly used to*evaluate*AI systems—as LLM\-as\-judge, and in ensemble or multi\-judge \(‘mixture\-of\-experts’\) panels now common in enterprise evaluation pipelines\(Zheng et al\.,[2023](https://arxiv.org/html/2607.08065#bib.bib28)\)\. These pipelines rest on a pervasive intuition: that*agreement*signals correctness—if several judges concur, or if a model returns the same answer across many stochastic samples, the answer is trusted, and if they disagree it is treated as a guess\. But agreement\-with\-itself is not the same as being right: a model can repeat an answer because it is genuinely certain, or because every sample passes through the same memorized heuristic, shared misconception, option\-position prior, or systematic hallucination\. This is the self\-referential analogue of the*self\-preference*bias documented when models judge their own outputs\(Zheng et al\.,[2023](https://arxiv.org/html/2607.08065#bib.bib28); Panickssery et al\.,[2024](https://arxiv.org/html/2607.08065#bib.bib17)\)—agreement with oneself can encode shared bias rather than correctness\. Self\-consistency decoding operationalizes this by returning the majority answer overKKsamples\(Wang et al\.,[2023](https://arxiv.org/html/2607.08065#bib.bib23)\), building on chain\-of\-thought prompting\(Wei et al\.,[2022](https://arxiv.org/html/2607.08065#bib.bib24)\)\. The same agreement signal is now reused well beyond decoding—as a confidence estimate for cost\-based routing and cascades\(Chen et al\.,[2023](https://arxiv.org/html/2607.08065#bib.bib2)\), for adaptive sample\-budget allocation\(Aggarwal et al\.,[2023](https://arxiv.org/html/2607.08065#bib.bib1)\), and for selective prediction and abstention\(Geifman and El\-Yaniv,[2017](https://arxiv.org/html/2607.08065#bib.bib9); Kamath et al\.,[2020](https://arxiv.org/html/2607.08065#bib.bib12)\)\.

That self\-consistency raises*accuracy*via majority voting is well established, as is the broader finding that elicited LLM confidence is often miscalibrated and can even worsen after post\-training\(Kadavath et al\.,[2022](https://arxiv.org/html/2607.08065#bib.bib11); Lin et al\.,[2022](https://arxiv.org/html/2607.08065#bib.bib14); Tian et al\.,[2023](https://arxiv.org/html/2607.08065#bib.bib21); OpenAI,[2023](https://arxiv.org/html/2607.08065#bib.bib16)\)\. Less systematically examined is how reliably the*agreement*signal itself functions as the confidence proxy that routing and abstention systems already assume it to be—and*when*it fails—under a controlled, cross\-replicated audit on hard reasoning benchmarks\. We provide that audit, using a cross\-runner design to separate confident errors that recur across different runners and prompts from those that are sampling noise\. Our thesis is thatself\-consistency is not accuracy, but a regime\-dependent empirical proxy for it: a positive yet weak signal whose usefulness depends on measurable conditions—model tier and agreement regime, answer space, and intended use—which we map\. It is usable where agreement is unsaturated \(mid\-tier models\) and for allocating compute, but unreliable as a standalone confidence score, worst for the over\-confident frontier model\.

### Contributions\.

\(1\) A large\-scale \(265,000265\{,\}000\-sample\), cross\-runner audit of self\-consistency*as a confidence/abstention/routing signal*on GPQA Diamond and AIME, with a hierarchical runner\-clustered bootstrap and multiplicity control\. \(2\) Evidence that the agreement signal*degrades*on the most consistent frontier model—over\-confident but no more accurate—consistent with the known post\-RLHF calibration regression, making confidence\-routing toward it counterproductive\. \(3\) A recurrence\-based separation of recurring from stochastic confident\-wrongness, plus an option\-shuffle control showing part of GPQA “confidence” is positional\. \(4\) Honest null/marginal results: chain\-of\-thought improves accuracy but only marginally improves the agreement–correctness signal, and a confidence\-routed cascade is dominated by always using the mid\-tier model\. \(5\) A released de\-identified, schema\-validated dataset and analysis pipeline\. \(6\) An*exploratory*, agent\-mediated cross\-family check on three Claude tiers, consistent with the frontier over\-confidence and suggesting confident errors are partly*shared across providers*\(GPT and Claude pick the same wrong answers above a marginal\-preserving null for the two smaller tiers\)—so high agreement can reflect shared bias, not correctness\.

Table 1:Spearmanρ\\rhoof self\-consistencyCwith sample accuracyAand with majority\-correctnessM, per cell, with hierarchical runner\-clustered\-bootstrap 95% CIs \(B=2000B\{=\}2000\)\.C¯\\overline\{\\textit\{C\}\},M¯\\overline\{\\textit\{M\}\}are cell means\. All twelveρ​\(C,M\)\\rho\(\\textit\{C\},\\textit\{M\}\)are positive and survive Holm correction \(adj\.p≤\.0024p\\leq\.0024\)\.†\\daggerOnly the nano–GPQAρ​\(C,A\)\\rho\(\\textit\{C\},\\textit\{A\}\)CI crosses zero\. The frontier gpt\-4\.1 has the highest agreement \(C¯=\.89\\overline\{\\textit\{C\}\}\{=\}\.89\) yet the lowestρ\\rhoand no accuracy advantage over mini\.Table 2:*Post hoc*focal contrasts \(both GPQA\) on the deployment labelM\(E1: Wilcoxon on pairedΔ​M\\Delta\\textit\{M\}; E2: permutation onΔ​ρ\\Delta\\rho\); AIME secondary\. These contrasts were chosen*after*earlier analyses and prior review, so thepp\-values aredescriptive, not confirmatory; Holm is applied only to the twelve per\-cellρ​\(C,M\)\\rho\(\\textit\{C\},\\textit\{M\}\)tests of Table[1](https://arxiv.org/html/2607.08065#S1.T1)\. E2 is the stronger contrast—it survives a Bonferroni\-for\-two \(padj=\.05p\_\{\\text\{adj\}\}\{=\}\.05\) and a case\-clustered bootstrap, and recurs on AIME \(Δ​ρ=−\.26\\Delta\\rho\{=\}\-\.26,p=6×10−4p\{=\}6\{\\times\}10^\{\-4\}\); E1 is*borderline*\(itsΔ​M\\Delta\\textit\{M\}CI crosses zero\), though its accuracy effect is strong \(Δ​A=\+\.067\\Delta\\textit\{A\}\{=\}\+\.067,p=2×10−5p\{=\}2\{\\times\}10^\{\-5\}\)\.

## 2Setup

### Data provenance\.

The runs were produced by5353runners completing an assignment in a graduate course \(participants shared a course context and may have exchanged code or discussion, so runs are independently submitted but not guaranteed statistically independent\); we perform a*secondary, de\-identified*meta\-analysis of their submitted run\-level outputs\. No runner is identifiable: all personal identifiers are removed and replaced with anonymous indices\. We release de\-identified per\-runner–per\-case rows \(an anonymous runner index, case id, the answer\-count distribution, andC/A/M\\textit\{C\}/\\textit\{A\}/\\textit\{M\}\)*without*GPQA question text—sufficient to reproduce the clustered analysis \(§[Ethics Statement](https://arxiv.org/html/2607.08065#Sx2)\)\. Because runners were assigned overlapping cases \(and each runner ran both of its axis’s conditions on the same cases, so Axis\-B and Axis\-C contrasts are paired within runner\), each \(axis, condition, case\) cell is independently re\-sampled by a mean of2\.52\.5runners \(up to99\); in total the data span5,3005\{,\}300case\-result rows over394394unique cases\. This overlap is the basis for the replication analysis in §[7](https://arxiv.org/html/2607.08065#S7)\.

### Benchmarks\.

GPQA Diamond\(Rein et al\.,[2024](https://arxiv.org/html/2607.08065#bib.bib19)\): graduate\-level four\-option multiple choice \(\{A,B,C,D\}\\\{A,B,C,D\\\}\)\.AIME: integer\-answer competition mathematics \(\{0,…,999\}\\\{0,\\dots,999\\\}\)\. They bracket opposite ends of an answer\-space spectrum: GPQA’s four options floorCnear1/41/4even under random sampling, so highCis the discriminating regime; AIME’s large space floorsCfor any model that cannot reason\.

### Design\.

Each runner was assigned one of three controlled comparisons, each with two conditions \(aavs\.bb\), attemperature=1\.0=1\.0,K=50K\{=\}50, on the gpt\-4\.1 family:Axis Amodel tier \(nano vs\. mini, zero\-shot\);Axis Bprompt strategy \(mini zero\-shot vs\. mini chain\-of\-thought\);Axis Cmid vs\. frontier \(mini vs\. gpt\-4\.1, zero\-shot\)\. Runners implemented their own prompts, so prompt wording varies—a source of variance we treat as part of the measurement \(§[Limitations](https://arxiv.org/html/2607.08065#Sx1)\)\. Each runner ran*both*conditions of its axis on the*same*cases, so all Axis\-B and Axis\-C contrasts are paired within runner \(100%100\\%pairing\); Table[3](https://arxiv.org/html/2607.08065#S2.T3)reports the counts\. Exact model snapshots and run timestamps were not logged and are unrecoverable, so our replication is across runners and prompts, not across time \(§[Limitations](https://arxiv.org/html/2607.08065#Sx1)\)\.

Table 3:Design and counts per axis\. “rows” are case\-result rows over both conditions; “rep” is the mean number of runners per \(condition, case\) cell \(max66–99\)\. Each runner ran both conditions on the same cases, so Axis\-B/C contrasts are paired within runner\. Totals:5353runners,394394unique cases,5,3005\{,\}300rows\.
### Metrics\.

For each \(case, condition\) we compute self\-consistencyC=nmaj/K\\textit\{C\}=n\_\{\\text\{maj\}\}/K, sample accuracyA=ncorrect/K\\textit\{A\}=n\_\{\\text\{correct\}\}/K, and the majority\-correct indicatorM=𝟙​\[majority answer=ground truth\]\\textit\{M\}=\\mathbb\{1\}\[\\text\{majority answer\}=\\text\{ground truth\}\]\. Because deployed majority\-vote systems return the majority answer,Mis ourprimaryoutcome;Ais secondary\. We treatCas the deployed confidence*score under audit*, not as a calibrated probability:Chas benchmark\-dependent floors \(near1/41/4for four\-option GPQA, far lower for AIME\), so “over\-confident” means highCco\-occurring with errors, not a miscalibrated probability in the formal sense; ECE/Brier ofCagainstMare operational diagnostics of that deployment use \(reliability diagrams in the appendix\)\.*Unit of analysis:*cell means, ECE/Brier, and err∣C≥\.8\\mid\\textit\{C\}\{\\geq\}\.8arerow\-weightedover case\-result entries; “always\-CW” and the recurrence rates of §[7](https://arxiv.org/html/2607.08065#S7)arecase\-weightedover unique cases\.

### Inference\.

We report Spearmanρ\\rho\. Entries are not i\.i\.d\. \(each runner contributes many rows; each case is seen by many runners\), so all CIs use ahierarchical runner\-clustered bootstrap\(resample runners, then cases within runner;B=2000B\{=\}2000; Appendix[A](https://arxiv.org/html/2607.08065#A1)\)\. Because this corrects runner dependence but not the globally shared\-case factor, we also report acase\-clusteredbootstrap and a leave\-one\-runner check for the headline frontier result \(§[4](https://arxiv.org/html/2607.08065#S4)\)\. Before the revised analysis we designatedtwo endpoints\(both on GPQA, declared*post hoc*for this revision, not preregistered\)—the paired Axis\-B chain\-of\-thought effect \(E1; Wilcoxon onΔ​M\\Delta\\textit\{M\}\) and the Axis\-C frontier degradation \(E2; permutation onΔ​ρ​\(C,M\)\\Delta\\rho\(\\textit\{C\},\\textit\{M\}\)\); theirpp\-values are reported raw, with a Bonferroni\-for\-two check in Table[2](https://arxiv.org/html/2607.08065#S1.T2)\. The remaining cells are exploratory; these endpoints were declared for the revised analysis \(prompted by prior review\) and are*not*a preregistration\. Holm correction is applied only to the family of twelve per\-cellρ​\(C,M\)\\rho\(\\textit\{C\},\\textit\{M\}\)tests \(Table[1](https://arxiv.org/html/2607.08065#S1.T1)\); correlation*differences*use permutation tests\. No inferential claim in this paper is preregistered; the endpoint tests in Table[2](https://arxiv.org/html/2607.08065#S1.T2)are descriptive robustness summaries, not confirmatory\.

## 3Does self\-consistency predict correctness?

Table[1](https://arxiv.org/html/2607.08065#S1.T1)reportsρ​\(C,⋅\)\\rho\(\\textit\{C\},\\cdot\)for all twelve cells\. The picture is consistent with a*conditional proxy*: agreement tracks correctness everywhere, but weakly and with a strength that varies systematically across the regimes we vary next\.\(1\) Positive but weak\.No cell exceedsρ​\(C,M\)=0\.6\\rho\(\\textit\{C\},\\textit\{M\}\)=0\.6; most variance in correctness is unexplained by agreement, so a “high\-Cauto\-accept” rule still admits many wrong answers\.\(2\) The signal pivots on the label\.For nano on GPQA,ρ​\(C,A\)=0\.12\\rho\(\\textit\{C\},\\textit\{A\}\)=0\.12has a CI crossing zero whereasρ​\(C,M\)=0\.21\\rho\(\\textit\{C\},\\textit\{M\}\)=0\.21is positive—reinforcingMas the right target\. All twelveρ​\(C,M\)\\rho\(\\textit\{C\},\\textit\{M\}\)are positive and survive Holm correction \(adj\.p≤0\.0024p\\leq 0\.0024\); they also all remain positive and exclude zero under a*case*\-clustered bootstrap that resamples items rather than runners \(Table[4](https://arxiv.org/html/2607.08065#S3.T4)\), so the link is not an artifact of the clustering choice\.\(3\) Non\-monotonic in scale\.The frontier gpt\-4\.1 has the*lowest*GPQAρ​\(C,M\)\\rho\(\\textit\{C\},\\textit\{M\}\)\(0\.200\.20\) despite the highest meanC\(0\.890\.89\); we examine this next\.

Table 4:Item\-clustered robustness:*case*\-clustered bootstrap 95% CIs for all twelveρ​\(C,M\)\\rho\(\\textit\{C\},\\textit\{M\}\)\(resampling cases, not runners;B=1000B\{=\}1000\)\. All twelve are positive and exclude zero, so the agreement–correctness link is not an artifact of the runner\-clustered choice\.
## 4The most consistent model is the worst\-calibrated voter \(in this audit\)

In this audit \(one OpenAI model family, with an exploratory Claude check in §[8](https://arxiv.org/html/2607.08065#S8)\), on the same Axis\-C cases gpt\-4\.1 has a significantly lower agreement–correctness correlation than mini:Δ​ρ​\(C,M\)=−0\.18​\[−0\.31,−0\.03\]\\Delta\\rho\(\\textit\{C\},\\textit\{M\}\)=\-0\.18\\,\[\-0\.31,\-0\.03\]on GPQA \(p=0\.025p\{=\}0\.025, permutation\) and−0\.26​\[−0\.40,−0\.12\]\-0\.26\\,\[\-0\.40,\-0\.12\]on AIME \(p=0\.0006p\{=\}0\.0006\)\. This is not because gpt\-4\.1 is stronger here—its GPQA majority accuracy is slightly*lower*than mini’s \(M¯=0\.48\\overline\{\\textit\{M\}\}\{=\}0\.48vs\.0\.520\.52\)—but because it is over\-confident: meanC=0\.89\\textit\{C\}\{=\}0\.89, reachingC≥0\.8\\textit\{C\}\\geq 0\.8on77%77\\%of GPQA cases\. With agreement piled at the ceiling,Closes discriminative power\. gpt\-4\.1 also has the worst GPQA expected calibration error of all twelve cells \(ECE=0\.41\\mathrm\{ECE\}=0\.41, both fixed\-width and equal\-mass; Brier0\.420\.42\) versus0\.220\.22for CoT\-mini \(Table[5](https://arxiv.org/html/2607.08065#S4.T5); Figures[1](https://arxiv.org/html/2607.08065#S4.F1),[2](https://arxiv.org/html/2607.08065#S4.F2)\)\. The miscalibration is not GPQA\-specific: on AIME gpt\-4\.1’s ECE is0\.300\.30at majority accuracy only0\.170\.17\.

Table 5:GPQA calibration and confident\-wrongness\. ECE is fixed\-width \(equal\-mass agrees within±\.01\\pm\.01; Appendix\); Brier is the mean squared error ofCagainstM\. “err∣C≥\.8\\mid\\textit\{C\}\{\\geq\}\.8” is the row\-weighted majority\-error rate \(M=0\\textit\{M\}\{=\}0\) among high\-agreement entries; “always\-CW” is the \(case\-weighted\) fraction of unique cases confidently wrong \(C≥\.8\\textit\{C\}\{\\geq\}\.8*and*A≤\.3\\textit\{A\}\{\\leq\}\.3—a stricter sample\-level notion than majority\-error\) for*every*runner\. gpt\-4\.1 is worst on ECE, Brier,Pr⁡\[C≥\.8\]\\Pr\[\\textit\{C\}\{\\geq\}\.8\], and always\-CW; its conditional high\-confidence error \(\.48\.48\) is high but marginally below nano’s \(\.50\.50\)\. CoT\-mini is best on ECE, Brier, and always\-CW\.![Refer to caption](https://arxiv.org/html/2607.08065v1/x1.png)Figure 1:GPQA reliability: binned self\-consistencyCvs\. empirical majority\-correctness\. The frontier gpt\-4\.1 sits well*below*the diagonal at highC\(majority correct only52%52\\%whenC≥0\.8\\textit\{C\}\\geq 0\.8\), i\.e\. over\-confident; CoT\-mini \(72%72\\%\) tracks the diagonal more closely\.The deployment consequence is concrete:a high\-agreement \(C≥0\.8\\textit\{C\}\\geq 0\.8\) gpt\-4\.1 answer on GPQA is wrong48%48\\%of the time\(95%95\\%CI\[\.40,\.56\]\[\.40,\.56\], case\-clustered, row\-weighted over high\-agreement entries\), so a router that auto\-trusts such answers inherits roughly that error rate\. This direction is consistent with documented post\-training calibration regressions\(OpenAI,[2023](https://arxiv.org/html/2607.08065#bib.bib16)\), in which heavy post\-training can sharpen the output distribution at the cost of calibration; we do not identify the cause here\. Our contribution is not that this happens, but a quantification of how it degrades self\-consistency*as a deployable signal*, and \(§[7](https://arxiv.org/html/2607.08065#S7)\) that the resulting errors frequently recur across different runners and prompts\. These conclusions are not artifacts of the runner\-clustered bootstrap: under a*case*\-clustered bootstrap \(resampling items\) the frontierΔ​ρ​\(C,M\)\\Delta\\rho\(\\textit\{C\},\\textit\{M\}\)remains negative with a CI excluding zero \(GPQA−\.15​\[−\.27,−\.03\]\-\.15\\,\[\-\.27,\-\.03\], AIME−\.21​\[−\.32,−\.12\]\-\.21\\,\[\-\.32,\-\.12\]\) and is stable to leaving out any single runner\.

### A logged\-snapshot confirmation\.

Because the course runs lack recorded model snapshots \(§[Limitations](https://arxiv.org/html/2607.08065#Sx1)\), we re\-ran the Axis\-C comparison ourselves under a*single logged snapshot*\(gpt\-4\.1\-2025\-04\-14vs\.gpt\-4\.1\-mini\-2025\-04\-14,K=20K\{=\}20, fixed canonical prompt, the same4848GPQA cases\)\. The pattern reproduces: gpt\-4\.1 is more self\-consistent than mini \(meanC0\.890\.89vs\.0\.820\.82;C≥0\.8\\textit\{C\}\\geq 0\.8on81%81\\%vs\.63%63\\%\) at*identical*majority accuracy \(0\.480\.48each\), with worse calibration \(ECE=0\.41\\mathrm\{ECE\}=0\.41vs\.0\.340\.34\) and lowerρ​\(C,M\)\\rho\(\\textit\{C\},\\textit\{M\}\)\(0\.350\.35vs\.0\.410\.41; logged at20262026\-0505\-2424T1111:5353Z\)\. This makes a pure unlogged\-snapshot explanation less likely—though the re\-run is small \(4848GPQA,K=20K\{=\}20\) and theρ\\rho/ECE*gaps*are not individually significant at this scale \(Δ​ρ=−\.06​\[−\.37,\.26\]\\Delta\\rho\{=\}\-\.06\\,\[\-\.37,\.26\],Δ​ECE=\+\.07​\[−\.07,\.22\]\\Delta\\mathrm\{ECE\}\{=\}\+\.07\\,\[\-\.07,\.22\]\); the over\-confidence itself \(higherCat equal accuracy\) reproduces clearly\.

![Refer to caption](https://arxiv.org/html/2607.08065v1/x2.png)Figure 2:GPQA risk–coverage by self\-consistency for the three key cells\. The frontier gpt\-4\.1 curve is highest \(worst\) and CoT\-mini lowest, despite gpt\-4\.1’s higher mean agreement—self\-consistency orders correctness least well exactly where agreement is highest\.

## 5Chain\-of\-thought: accuracy yes, signal marginal

Axis B isolates chain\-of\-thought on a fixed model \(mini\) with a within\-runner paired design\. CoT robustly improves*accuracy*: pairedΔ​A=\+0\.067\\Delta\\textit\{A\}=\+0\.067\(GPQA, Wilcoxonp=2\.1×10−5p\{=\}2\.1\{\\times\}10^\{\-5\}\) and\+0\.069\+0\.069\(AIME,p=2\.9×10−4p\{=\}2\.9\{\\times\}10^\{\-4\}\), with majority flips22–5×5\{\\times\}more often toward correct than away\. Its effect on the deployment label is weaker:Δ​M=\+0\.066\\Delta\\textit\{M\}=\+0\.066\(GPQA; bootstrap CI\[−0\.005,0\.132\]\[\-0\.005,0\.132\], Wilcoxonp=0\.02p\{=\}0\.02\) and\+0\.080\+0\.080\(AIME,p=0\.003p\{=\}0\.003\)\. The effect on the*signal*is mixed: CoT raisesρ​\(C,A\)\\rho\(\\textit\{C\},\\textit\{A\}\)significantly on GPQA \(\+0\.199​\[0\.049,0\.367\]\+0\.199\\,\[0\.049,0\.367\],p=0\.0002p\{=\}0\.0002\) but its effect on the primaryρ​\(C,M\)\\rho\(\\textit\{C\},\\textit\{M\}\)is only borderline \(\+0\.123​\[−0\.030,0\.265\]\+0\.123\\,\[\-0\.030,0\.265\],p=0\.025p\{=\}0\.025GPQA;\+0\.031\+0\.031,p=0\.21p\{=\}0\.21AIME\)\. Under*case*\-clustered resampling the GPQAΔ​ρ​\(C,M\)\\Delta\\rho\(\\textit\{C\},\\textit\{M\}\)CI excludes zero \(\+\.123​\[\.016,\.238\]\+\.123\\,\[\.016,\.238\]\) while AIME stays null \(\+\.031​\[−\.023,\.080\]\+\.031\\,\[\-\.023,\.080\]\)\. CoT is thus a reliable accuracy intervention but at most a marginal one for the agreement–correctness signal\.

## 6Self\-consistency vs\. other confidence signals

We compare self\-consistency against two signals from prior calibration work, on the same gpt\-4\.1\-mini cases \(5050per benchmark\)\.*Verbalized confidence*\(Lin et al\.,[2022](https://arxiv.org/html/2607.08065#bib.bib14); Tian et al\.,[2023](https://arxiv.org/html/2607.08065#bib.bib21)\)is the mean of ten elicited0–100100self\-ratings;*P\(True\)*\(Kadavath et al\.,[2022](https://arxiv.org/html/2607.08065#bib.bib11)\)is the True/False next\-token logprob mass on the model’s own modal answer\. AURC \(area under the risk–coverage curve\) and ECE targetM; CIs are case\-level \(we ran these cases ourselves, so runner clustering does not apply\)\. Table[6](https://arxiv.org/html/2607.08065#S6.T6)reports GPQA\.

Table 6:Confidence signals on GPQA \(n=50n\{=\}50, gpt\-4\.1\-mini\)\. No signal dominates; differences sit within wide CIs\. Verbalized confidence parsed on only39/5039/50GPQA cases, so itsρ\\rho/AURC/ECE are computed over those parsed cases and are*not*directly comparable toC/P\(True\), which use all5050\(“cov\.” column\);∗\\astits AURC is computed on those3939cases, a favourable subset, so the verbalized\-confidence AURC advantage should be read with caution\.On GPQA, self\-consistency has the highest rank correlation with correctness \(ρ=0\.31\\rho\{=\}0\.31\), edging verbalized confidence \(0\.210\.21\) and P\(True\) \(0\.190\.19\); yet verbalized confidence yields the best risk–coverage \(AURC0\.300\.30vs\.0\.440\.44\) and a rank\-average combination is best\-calibrated \(ECE0\.130\.13\), with all differences within wide CIs \(n=50n\{=\}50\)\. On AIME every signal is weak—majority accuracy is near the floor—and verbalized confidence is essentially uninformative: it parsed on only3/503/50AIME cases and was nearly constant \(mean0\.920\.92\), while P\(True\) is only weakly predictive \(ρ=0\.12\\rho\{=\}0\.12\)\. Self\-consistency is thus*competitive with, not dominated by*, dedicated confidence signals on these hard benchmarks, and none is a reliable standalone abstention score \(Figure[3](https://arxiv.org/html/2607.08065#S6.F3)\)\.

![Refer to caption](https://arxiv.org/html/2607.08065v1/x3.png)Figure 3:Risk–coverage curves by confidence signal \(gpt\-4\.1\-mini\)\. Lower is better\. No signal is a reliable standalone deferral score on these hard benchmarks\.
## 7Confident errors recur—but are partly positional

Are confidently\-wrong outcomes mere sampling artifacts? The replication design says not entirely: several GPQA questions yield the same wrong majority answer atC=1\.0\\textit\{C\}\{=\}1\.0from*every*runner who ran them, across different model conditions\. For example,gpqa\_d\_076andgpqa\_d\_106\(both ground truth B\) each elicited a wrong “C” from all runners across multiple axes\. Quantitatively, for gpt\-4\.1 on GPQA,28%28\\%of unique cases are confidently wrong for*every*runner and50%50\\%for*at least one*—so a nontrivial subset of confident errors recur across different runners and prompts rather than being sampling noise\. We avoid the stronger claim that they are wholly intrinsic: AIME confident errors are often dispersed \(no single stable wrong integer, frequently unparseable\), and a minority of GPQA items appear ambiguous or mislabeled \(e\.g\. one item answered consistently against its listed key\)—a caveat for treating every such case as a model error\.

### An option\-shuffle control\.

Because GPQA is multiple choice, high agreement on a letter can reflect position priors rather than semantic confidence\(Zheng et al\.,[2024](https://arxiv.org/html/2607.08065#bib.bib27); Wei et al\.,[2024](https://arxiv.org/html/2607.08065#bib.bib25)\)\. On4848GPQA cases \(1212per ground\-truth letter\), for gpt\-4\.1\-mini and gpt\-4\.1, we ranK=50K\{=\}50under the original order and two random option permutations, remapping the ground\-truth letter under each \(Figure[4](https://arxiv.org/html/2607.08065#S7.F4)\)\. Two effects emerge\. First, answer content is*position\-sensitive*: mini’s point\-estimate majority accuracy falls from0\.50​\[0\.35,0\.65\]0\.50\\,\[0\.35,0\.65\]to0\.29​\[0\.19,0\.41\]0\.29\\,\[0\.19,0\.41\]under shuffling \(gpt\-4\.1 falls less,0\.46→0\.390\.46\{\\to\}0\.39\)—suggestive, though the CIs overlap atn=48n\{=\}48—and the majority*content*is stable across permutations for only35%35\\%of mini cases \(56%56\\%for gpt\-4\.1\)\. Second, “D” is*under\-selected in both*the original and shuffled conditions \(≈\\approx16%16\\%for mini,≈\\approx15%15\\%for gpt\-4\.1, vs\. a25%25\\%uniform baseline\), so the D\-avoidance is a position bias that survives content randomization\. A meaningful share of GPQA “confidence” is therefore positional, not semantic\.

![Refer to caption](https://arxiv.org/html/2607.08065v1/x4.png)Figure 4:Majority\-answer letter distribution under original vs\. shuffled options\. The model under\-selects “D” regardless of which content sits there, and GPQA accuracy drops under shuffling\.

## 8An exploratory cross\-family check: are confident errors shared?

The audit so far is within one provider\. As an*exploratory*robustness check—*not*a like\-for\-like reproduction—we re\-ran the GPQA and AIME measurements on three tiers of a second family, Anthropic’s Claude \(haiku, sonnet, opus\), on the same studied cases \(4848GPQA,1212per ground\-truth letter;2424AIME\)\. The caveats are substantial and we state them first: lacking API access, we drew samples through separate Claude*agent sessions*, which we treat as approximately independent samples but cannot control for sampling temperature or verify backend/session independence;K=10K\{=\}10per tier is far below the main study’sK=50K\{=\}50; and these models reason internally even when asked for a direct answer \(so AIME is effectively chain\-of\-thought—we therefore center the comparison on GPQA\)\. Absolute values are internal, not comparable to the gpt\-4\.1 runs; CIs are case\-level bootstraps\. With those caveats, the qualitative pattern recurs\.

### Frontier over\-confidence recurs\.

Table[7](https://arxiv.org/html/2607.08065#S8.T7)shows the same non\-monotonic pattern within Claude\. The frontier tier \(opus\) is the*most*self\-consistent \(meanC=0\.94\\textit\{C\}\{=\}0\.94,C≥0\.8\\textit\{C\}\\geq 0\.8on88%88\\%of GPQA cases\) but*not*the most accurate—its majority accuracy \(0\.580\.58\) trails the mid tier \(sonnet,0\.630\.63\)—and it has the worst calibration \(ECE=0\.35\\mathrm\{ECE\}=0\.35\) and a lowerρ​\(C,M\)\\rho\(\\textit\{C\},\\textit\{M\}\)\(0\.440\.44\) than sonnet \(0\.740\.74\)\. As in the gpt\-4\.1 family, the most\-consistent tier discriminates correctness*worse*than the mid tier\.

Table 7:Exploratory cross\-family check on Claude tiers \(GPQA,K=10K\{=\}10agent sessions per case,4848cases\)\. The frontier tier \(opus\) is the most self\-consistent and most over\-confident \(C≥0\.8\\textit\{C\}\\geq 0\.8on88%88\\%\) yet not the most accurate and the worst\-calibrated—reproducing the gpt\-4\.1 pattern \(Table[5](https://arxiv.org/html/2607.08065#S4.T5)\)\.
### Confident errors are partly shared across families\.

On the4646GPQA items the gpt\-4\.1 runs also cover, gpt\-4\.1 and the Claude tiers agree on the majority answer for4848–63%63\\%of items\. Because option\-position priors \(§[7](https://arxiv.org/html/2607.08065#S7)\) make a flat1/31/3null inappropriate, we test the same\-wrong\-answer rate \(among items*both*get wrong\) against a label\-permutation null that preserves each model’s empirical wrong\-answer distribution \(Table[8](https://arxiv.org/html/2607.08065#S8.T8)\)\. The observed rate exceeds the null for all three tiers and significantly so for haiku \(p=\.003p\{=\}\.003\) and sonnet \(p=\.005p\{=\}\.005\), though not opus \(p=\.07p\{=\}\.07; only1212shared\-wrong items\); and these shared wrong answers are*high\-confidence*for both providers \(meanC≈0\.85\\textit\{C\}\\approx 0\.85GPT,≈0\.78\\approx 0\.78Claude\), so they are confident errors, not low\-agreement coincidences\. Six of the4646items are answered wrongly by*all four*models \(e\.g\. one whose listed key is B that every model answers D\)\. Confident errors thus appear partly*shared*across providers rather than idiosyncratic—consistent with shared pretraining biases, shared misconceptions, or item flaws—which both strengthens the recurrence finding of §[7](https://arxiv.org/html/2607.08065#S7)and cautions that cross\-family agreement is itself not evidence of correctness \(§[Limitations](https://arxiv.org/html/2607.08065#Sx1)\)\.

Table 8:Shared confident errors \(GPQA,4646shared items, exploratory\)\. Among items*both*gpt\-4\.1 and the Claude tier answer wrongly \(“both\-wrong”\), how often they pick the*same*wrong option \(“same”; obs==same/both\-wrong\), vs\. a label\-permutation null preserving each model’s wrong\-answer marginal \(ppis the permutation tail\)\. Shared\-wrong items are high\-confidence for both \(meanC0\.840\.84–0\.930\.93gpt\-4\.1,0\.700\.70–0\.870\.87Claude\)\.

## 9Implications for systems

### Adaptive sampling\.

Simulating stop\-when\-consensus on the existing sample banks, an adaptive rule uses60%60\\%fewer samples on average \(19\.819\.8vs\.5050\) at*equal*majority error \(0\.6030\.603\); on CoT\-mini the saving is82%82\\%\. Agreement is thus useful for*allocating*compute even where it is weak for*trusting*an answer, consistent withAggarwal et al\. \([2023](https://arxiv.org/html/2607.08065#bib.bib1)\)\.

### Confidence\-routed cascade\.

Routing cheap→\\toexpensive byC\(Chen et al\.,[2023](https://arxiv.org/html/2607.08065#bib.bib2)\)beats random ordering on the cost–accuracy frontier \(peak majority accuracy0\.360\.36vs\.0\.340\.34\) but is*dominated by simply always using the mid\-tier model*\(0\.400\.40\); always\-frontier reaches only0\.330\.33, and an oracle router0\.440\.44\. Because escalation sends “hard” \(low\-C\) cases to the over\-confident, no\-more\-accurate frontier model \(§[4](https://arxiv.org/html/2607.08065#S4)\), routing toward it is counterproductive here—a direct, cautionary consequence of the frontier degradation \(Table[9](https://arxiv.org/html/2607.08065#S9.T9)\)\.

Table 9:Cost\-aware routing\. TheC\-routed cascade beats random ordering but is dominated by simply always using the mid\-tier model; escalating to the over\-confident frontier model hurts\.

## 10Related work

Self\-consistency was introduced to raise accuracy via majority voting\(Wang et al\.,[2023](https://arxiv.org/html/2607.08065#bib.bib23)\), not as a calibrated confidence; reusing the agreement signal as confidence is the downstream practice we audit\. LLM confidence can be elicited and is often miscalibrated—via self\-evaluation/P\(True\)\(Kadavath et al\.,[2022](https://arxiv.org/html/2607.08065#bib.bib11)\), verbalized confidence\(Lin et al\.,[2022](https://arxiv.org/html/2607.08065#bib.bib14); Tian et al\.,[2023](https://arxiv.org/html/2607.08065#bib.bib21)\), and post\-training calibration regression\(OpenAI,[2023](https://arxiv.org/html/2607.08065#bib.bib16)\)\. Agreement already drives adaptive sampling\(Aggarwal et al\.,[2023](https://arxiv.org/html/2607.08065#bib.bib1)\)and confidence cascades\(Chen et al\.,[2023](https://arxiv.org/html/2607.08065#bib.bib2)\); selective prediction\(Geifman and El\-Yaniv,[2017](https://arxiv.org/html/2607.08065#bib.bib9); Kamath et al\.,[2020](https://arxiv.org/html/2607.08065#bib.bib12)\)thresholds a confidence signal to abstain\. Multiple\-choice answers carry option\-order/selection biases\(Zheng et al\.,[2024](https://arxiv.org/html/2607.08065#bib.bib27); Wei et al\.,[2024](https://arxiv.org/html/2607.08065#bib.bib25); Pezeshkpour and Hruschka,[2024](https://arxiv.org/html/2607.08065#bib.bib18)\)\. Closest to our concern, sampling\-based agreement is itself used as an uncertainty or hallucination signal—SelfCheckGPT\(Manakul et al\.,[2023](https://arxiv.org/html/2607.08065#bib.bib15)\)and semantic\-entropy methods\(Kuhn et al\.,[2023](https://arxiv.org/html/2607.08065#bib.bib13); Farquhar et al\.,[2024](https://arxiv.org/html/2607.08065#bib.bib8)\)—and surveys of LLM confidence estimation treat consistency as one such cue\(Geng et al\.,[2024](https://arxiv.org/html/2607.08065#bib.bib10); Xiong et al\.,[2024](https://arxiv.org/html/2607.08065#bib.bib26)\)\. A separate line shows that a model’s preference for its*own*outputs—self\-enhancement/self\-preference bias in LLM\-as\-judge settings\(Zheng et al\.,[2023](https://arxiv.org/html/2607.08065#bib.bib28); Panickssery et al\.,[2024](https://arxiv.org/html/2607.08065#bib.bib17)\)—can inflate self\-agreement independently of correctness; self\-consistency is the intra\-model analogue, which our cross\-family shared\-error analysis \(§[8](https://arxiv.org/html/2607.08065#S8)\) probes directly\. Judging correctness when reference labels are scarce or contested is itself an active measurement problem\(Ding,[2025b](https://arxiv.org/html/2607.08065#bib.bib5)\), and our runner\-clustered, heterogeneity\-aware inference is in the spirit of clustered estimation over heterogeneous units\(Ding et al\.,[2024](https://arxiv.org/html/2607.08065#bib.bib6),[2025](https://arxiv.org/html/2607.08065#bib.bib7)\)\. Our contribution is an empirical audit of the agreement signal in the confidence role on hard reasoning benchmarks, with a cross\-runner recurrence analysis and an exploratory cross\-family check that separates recurring from stochastic confident\-wrongness and finds confident errors are partly shared across providers\.

## 11Conclusion

Self\-consistency is*not*accuracy, but it is a positive, regime\-dependent empirical proxy for it: useful where agreement is unsaturated—mid\-tier models, and for allocating sample budget—but unreliable as a standalone confidence score, worst for the over\-confident frontier model where high agreement is only weakly tied to correctness\. Along the way it is only marginally improved by chain\-of\-thought on the deployment label, and partly positional on multiple choice; and its confident errors frequently recur across runners and prompts—and, in an exploratory cross\-family check, across providers—so they reflect shared bias more than sampling noise\. The practical upshot is not “never use agreement” but “use it within its regime”: treat it as a coarse, cost\-saving signal, not as a trustworthy confidence threshold, and never as a reason to route hard cases toward the most consistent frontier model\.

## Limitations

Our primary audit is within a single provider \(gpt\-4\.1\); we add an exploratory cross\-family check on three Claude tiers \(§[8](https://arxiv.org/html/2607.08065#S8)\), but it is agent\-mediated—notemperaturecontrol, smallerKK, and internal reasoning we cannot disable—so its absolute values are internal, not directly comparable to the gpt\-4\.1 runs\. Crucially, because the confident errors are largely*shared*across families, cross\-family agreement does not by itself certify correctness: shared pretraining data and shared option\-position priors can produce shared bias rather than independent confirmation\. We study two benchmarks \(four\-option MCQ and integer math\), not open\-ended generation or code\.K=50K\{=\}50leaves substantial measurement noise\. We do not study reasoning\-trained models, whose internal reasoning may already supply the calibration lift we measure for prompted CoT\. Runners wrote their own prompts, adding implementation variance\. Critically, exact model snapshots and run timestamps were not logged and are unrecoverable, so we cannot verify that the replicated runs occurred under an identical backend snapshot; our design establishes recurrence across*different*runners and prompts, not across time—though a controlled re\-run under a single logged snapshot \(§[4](https://arxiv.org/html/2607.08065#S4),20262026\-0505\-2424\) reproduces the over\-confidence direction \(higher agreement at equal accuracy\), while theρ\\rho/ECE gaps are underpowered atK=20K\{=\}20\. Finally, the confidence\-baseline and shuffle studies use5050and4848cases respectively, so their CIs are wide, and a minority of GPQA items appear ambiguous or mislabeled, which inflates apparent confident\-wrongness\. The confidence\-baseline and option\-shuffle controls were run by us under a current model snapshot, so their absolute values are internal comparisons, not directly comparable to the course runs\.

## Ethics Statement

This is a secondary analysis of model\-output data generated by runners completing a graduate course assignment\. All personal identifiers were removed prior to analysis and replaced with anonymous indices; we release only de\-identified per\-runner–per\-case rows \(anonymous runner index, case id, answer\-count distribution, andC/A/M\\textit\{C\}/\\textit\{A\}/\\textit\{M\}\), and no individual is identifiable\. The de\-identified data are available at[https://github\.com/dingkaihua/self\_consistency\_as\_predictor\_of\_accuracy](https://github.com/dingkaihua/self_consistency_as_predictor_of_accuracy)\. We do*not*redistribute GPQA Diamond question text, consistent with its access terms\. The work also carries a deployment\-safety implication: systems that treat self\-consistency as confidence will route confidently\-wrong answers into trusted paths, most severely for frontier models—a risk this paper quantifies\.

## Acknowledgments

I thank the graduate\-course cohort whose submitted model\-output runs constitute the dataset analyzed here\. All analysis, framing, and conclusions are the author’s own\.

The author acknowledges the National Artificial Intelligence Research Resource \(NAIRR\) Pilot and the Jetstream2 cloud resource at Indiana University for contributing to this research result\. This work used Jetstream2 through NAIRR Pilot allocation NAIRR250223; Jetstream2 is supported by the U\.S\. National Science Foundation under award NSF\-OAC 2005506\. The author also thanks the Wharton AI & Analytics Initiative at the University of Pennsylvania for financially supporting this project through its AI Education Innovation Fund\.

## References

- Aggarwal et al\. \(2023\)Pranjal Aggarwal, Aman Madaan, Yiming Yang, and Mausam\. 2023\.[Let’s sample step by step: Adaptive\-consistency for efficient reasoning and coding with LLMs](https://doi.org/10.18653/v1/2023.emnlp-main.761)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pages 12375–12396, Singapore\. Association for Computational Linguistics\.
- Chen et al\. \(2023\)Lingjiao Chen, Matei Zaharia, and James Zou\. 2023\.[FrugalGPT: How to use large language models while reducing cost and improving performance](https://arxiv.org/abs/2305.05176)\.*arXiv preprint arXiv:2305\.05176*\.
- Ding \(2018\)Kaihua Ding\. 2018\.[*Efficient Output\-Based Adaptation Mechanics for High\-Order Computational Fluid Dynamics Methods*](https://deepblue.lib.umich.edu/handle/2027.42/144065)\.Ph\.D\. thesis, University of Michigan\.
- Ding \(2025a\)Kaihua Ding\. 2025a\.[Designing AI\-Resilient Assessments Using Interconnected Problems: A Theoretically Grounded and Empirically Validated Framework](https://doi.org/10.48550/arXiv.2512.10758)\.*Preprint*, arXiv:2512\.10758\.Accepted, IEEE Frontiers in Education \(FIE\) 2026\.
- Ding \(2025b\)Kaihua Ding\. 2025b\.[Variance\-Bounded Evaluation of Entity\-Centric AI Systems Without Ground Truth: Theory and Measurement](https://doi.org/10.48550/arXiv.2509.22751)\.*Preprint*, arXiv:2509\.22751\.
- Ding et al\. \(2024\)Kaihua Ding, Jingsong Cui, Mohammad Soltani, and Jing Jin\. 2024\.[Iterative Causal Segmentation: Filling the Gap between Market Segmentation and Marketing Strategy](https://doi.org/10.48550/arXiv.2405.14743)\.*Preprint*, arXiv:2405\.14743\.
- Ding et al\. \(2025\)Kaihua Ding, Jingsong Cui, Mohammad Soltani, and Jing Jin\. 2025\.Iterative Causal Segmentation\.*PMSA Journal*, pages 21–33\.
- Farquhar et al\. \(2024\)Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal\. 2024\.[Detecting hallucinations in large language models using semantic entropy](https://doi.org/10.1038/s41586-024-07421-0)\.*Nature*, 630\(8017\):625–630\.
- Geifman and El\-Yaniv \(2017\)Yonatan Geifman and Ran El\-Yaniv\. 2017\.[Selective classification for deep neural networks](https://arxiv.org/abs/1705.08500)\.In*Advances in Neural Information Processing Systems 30 \(NIPS\)*\.ArXiv:1705\.08500\.
- Geng et al\. \(2024\)Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych\. 2024\.[A survey of confidence estimation and calibration in large language models](https://arxiv.org/abs/2311.08298)\.In*Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(NAACL\)*, pages 6577–6595, Mexico City, Mexico\. Association for Computational Linguistics\.
- Kadavath et al\. \(2022\)Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield\-Dodds, Nova DasSarma, Eli Tran\-Johnson, Scott Johnston, Sheer El\-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others\. 2022\.[Language models \(mostly\) know what they know](https://arxiv.org/abs/2207.05221)\.*arXiv preprint arXiv:2207\.05221*\.
- Kamath et al\. \(2020\)Amita Kamath, Robin Jia, and Percy Liang\. 2020\.[Selective question answering under domain shift](https://doi.org/10.18653/v1/2020.acl-main.503)\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics \(ACL\)*, pages 5684–5696\. Association for Computational Linguistics\.
- Kuhn et al\. \(2023\)Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar\. 2023\.[Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation](https://arxiv.org/abs/2302.09664)\.In*The Eleventh International Conference on Learning Representations \(ICLR\)*\.
- Lin et al\. \(2022\)Stephanie Lin, Jacob Hilton, and Owain Evans\. 2022\.[Teaching models to express their uncertainty in words](https://arxiv.org/abs/2205.14334)\.*Transactions on Machine Learning Research \(TMLR\)*\.ArXiv:2205\.14334\.
- Manakul et al\. \(2023\)Potsawee Manakul, Adian Liusie, and Mark J\. F\. Gales\. 2023\.[SelfCheckGPT: Zero\-resource black\-box hallucination detection for generative large language models](https://arxiv.org/abs/2303.08896)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pages 9004–9017, Singapore\. Association for Computational Linguistics\.
- OpenAI \(2023\)OpenAI\. 2023\.[GPT\-4 technical report](https://arxiv.org/abs/2303.08774)\.*arXiv preprint arXiv:2303\.08774*\.
- Panickssery et al\. \(2024\)Arjun Panickssery, Samuel R\. Bowman, and Shi Feng\. 2024\.[LLM evaluators recognize and favor their own generations](https://arxiv.org/abs/2404.13076)\.In*Advances in Neural Information Processing Systems 37 \(NeurIPS 2024\)*\.
- Pezeshkpour and Hruschka \(2024\)Pouya Pezeshkpour and Estevam Hruschka\. 2024\.[Large language models sensitivity to the order of options in multiple\-choice questions](https://arxiv.org/abs/2308.11483)\.In*Findings of the Association for Computational Linguistics: NAACL 2024*\. Association for Computational Linguistics\.
- Rein et al\. \(2024\)David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R\. Bowman\. 2024\.[GPQA: A graduate\-level google\-proof Q&A benchmark](https://arxiv.org/abs/2311.12022)\.In*First Conference on Language Modeling \(COLM\)*\.ArXiv:2311\.12022\.
- Shazeer et al\. \(2017\)Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean\. 2017\.Outrageously large neural networks: The sparsely\-gated mixture\-of\-experts layer\.In*International Conference on Learning Representations \(ICLR\)*\.
- Tian et al\. \(2023\)Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D\. Manning\. 2023\.[Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback](https://doi.org/10.18653/v1/2023.emnlp-main.330)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pages 5433–5442, Singapore\. Association for Computational Linguistics\.
- Verga et al\. \(2024\)Pat Verga, Sebastian Hofstätter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis\. 2024\.[Replacing judges with juries: Evaluating LLM generations with a panel of diverse models](https://arxiv.org/abs/2404.18796)\.*Preprint*, arXiv:2404\.18796\.
- Wang et al\. \(2023\)Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V\. Le, Ed H\. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou\. 2023\.[Self\-consistency improves chain of thought reasoning in language models](https://arxiv.org/abs/2203.11171)\.In*The Eleventh International Conference on Learning Representations \(ICLR\)*\.ArXiv:2203\.11171\.
- Wei et al\. \(2022\)Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H\. Chi, Quoc V\. Le, and Denny Zhou\. 2022\.[Chain\-of\-thought prompting elicits reasoning in large language models](https://arxiv.org/abs/2201.11903)\.In*Advances in Neural Information Processing Systems 35 \(NeurIPS\)*\.ArXiv:2201\.11903\.
- Wei et al\. \(2024\)Sheng\-Lun Wei, Cheng\-Kuang Wu, Hen\-Hsen Huang, and Hsin\-Hsi Chen\. 2024\.[Unveiling selection biases: Exploring order and token sensitivity in large language models](https://arxiv.org/abs/2406.03009)\.In*Findings of the Association for Computational Linguistics: ACL 2024*, pages 5598–5621\. Association for Computational Linguistics\.
- Xiong et al\. \(2024\)Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi\. 2024\.[Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs](https://arxiv.org/abs/2306.13063)\.In*The Twelfth International Conference on Learning Representations \(ICLR\)*\.
- Zheng et al\. \(2024\)Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang\. 2024\.[Large language models are not robust multiple choice selectors](https://arxiv.org/abs/2309.03882)\.In*The Twelfth International Conference on Learning Representations \(ICLR\)*\.ArXiv:2309\.03882\.
- Zheng et al\. \(2023\)Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P\. Xing, Hao Zhang, Joseph E\. Gonzalez, and Ion Stoica\. 2023\.[Judging LLM\-as\-a\-judge with MT\-Bench and chatbot arena](https://arxiv.org/abs/2306.05685)\.In*Advances in Neural Information Processing Systems 36 \(NeurIPS 2023\), Datasets and Benchmarks Track*\.

## Appendix AHierarchical cluster bootstrap

ForB=2000B\{=\}2000replicates we resample the5353runner indices with replacement; within each sampled runner we resample its cases with replacement; we recompute the statistic on the pooled resample and take percentile CIs\. This two\-stage scheme is a hierarchical \(nested\) cluster bootstrap on the runner factor: it corrects the dominant runner\-level dependence rather than implementing a full two\-way crossed resample over globally shared cases\. For the paired Axis\-B contrasts we resample \(runner, case\) units while preserving the zero\-shot/CoT pairing and recompute the paired difference\. Holm correction is applied across the family of twelve per\-cellρ​\(C,M\)\\rho\(\\textit\{C\},\\textit\{M\}\)tests; correlation\-differencepp\-values are permutation\-based\.

## Appendix BPrompts and protocol

Canonical zero\-shot and chain\-of\-thought prompts for GPQA and AIME, answer\-extraction rules, and the\_UNPARSEABLE\_handling \(which counts as incorrect\) are documented with the released code\. For the confidence baselines, verbalized confidence used a fixed “Answer/Confidence” template \(ten samples\), and P\(True\) read the True/False token logprobs on the modal answer; for the shuffle control, option contents were permuted with the ground\-truth letter remapped accordingly\.

## Appendix CAdditional tables and robustness

Per\-cell marginal rates, reliability diagrams, full risk–coverage curves, equal\-mass vs\. fixed\-width ECE, the confident\-wrongness threshold sweep, and the adaptive\-sampling and cascade frontiers are provided in the supplementary material\.