Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty

# Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty
Source: [https://arxiv.org/html/2606.17312](https://arxiv.org/html/2606.17312)
Baishali Chaudhury, Mengdie Flora Wang, Hyunji Hayley Park, Rahul Ghosh, Sungmin Hong, Jae Oh Woo AWS Generative AI Innovation Center \{baishch, florawan, parhyunj, rahulgh, hsungmin, jaeohwoo\}@amazon\.com

###### Abstract

Large language models can arrive at the same answer through reasoning paths that are unstable, contradictory, or difficult to rank consistently—a failure mode especially prevalent in multi\-step deductive reasoning\. Existing methods assess reasoning reliability primarily through output dispersion—measuring how much sampled answers differ—but this view discards a complementary signal: whether the model can consistently rank competing reasoning candidates\. We propose structural uncertainty, a consistency\-aware evaluation framework derived from the stability of self\-preference\-induced rankings over sampled reasoning solutions\. Given a query, we generate multiple candidate solutions and ask the same model to judge pairwise preferences among its own outputs\. We aggregate sparse self\-preferences into ranking distributions via Bradley–Terry modeling with PageRank, and decompose the signal into two complementary entropy\-based components—across\-trial ranking instability and within\-trial candidate ambiguity\. Across five LLMs and eight benchmarks, structural signals provide information complementary to answer dispersion: on logical and mathematical reasoning tasks, the combination improves identification of unreliable reasoning instances, while on factual retrieval the structural signal collapses toward uniformity, diagnosing a regime boundary where reasoning\-level consistency evaluation is uninformative\. The two components relate differently to accuracy: within\-trial ambiguity correlates positively with correctness on reasoning tasks—consistent with settings where multiple plausible solution paths remain competitive—while across\-trial instability correlates negatively, signaling unreliable reasoning\. Structural uncertainty is best understood not as a universal confidence estimator, but as a regime\-sensitive evaluator of logical reasoning consistency\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2606.17312v1/images/dispersion_vs_structural_stacked.png)Figure 1:An illustrative example contrasting dispersion\-based and structural consistency evaluation\.\(a\)When all five sampled reasoning candidates agree on the same wrong answer \(20\), dispersion\-based methods report zero entropy and low uncertainty, since they do not use preference\-order information among candidates\.\(b\)Our structural approach asks the model to rank its own reasoning outputs across independent trials\. The rankings are completely unstable \(r3moves from \#1 to \#5\), revealing reasoning inconsistency not reflected by answer dispersion alone\.Large language models have achieved remarkable progress in natural language understanding and generation, yet their logical reasoning capabilities remain a significant bottleneckXionget al\.\([2024](https://arxiv.org/html/2606.17312#bib.bib42)\); Tianet al\.\([2023](https://arxiv.org/html/2606.17312#bib.bib20)\); Kapooret al\.\([2024](https://arxiv.org/html/2606.17312#bib.bib16)\); Zhouet al\.\([2024](https://arxiv.org/html/2606.17312#bib.bib44)\)\. Models frequently produce answers that appear logically plausible yet are internally inconsistent—arriving at the same wrong conclusion through different flawed reasoning paths, or failing to maintain stable preferences when asked to compare their own solutions\. Evaluating logical reasoning reliability requires assessing not only answer\-level correctness but also the*consistency*of the reasoning process itself\. Although we do not directly model cross\-question contradiction, our framework targets a closely related consistency problem: whether the model can stably evaluate competing reasoning candidates for the same query\.

Existing post\-hoc evaluation methods\(Linet al\.,[2023](https://arxiv.org/html/2606.17312#bib.bib3); Farquharet al\.,[2024](https://arxiv.org/html/2606.17312#bib.bib4); Wanget al\.,[2024a](https://arxiv.org/html/2606.17312#bib.bib24); Lyuet al\.,[2025](https://arxiv.org/html/2606.17312#bib.bib2)\)treat sampled responses as exchangeable and assess reliability from how much answers*differ*—i\.e\., output dispersion\. This view captures one dimension of reasoning reliability but discards a complementary signal: the*structural consistency*of preferences among candidate reasoning solutions\. For logical reasoning, this omission is consequential because multiple candidates may share the same final answer yet differ in reasoning quality, coherence, or mutual consistency; collapsing this structure loses information about which solutions the model favors and how stable those preferences are\. Figure[1](https://arxiv.org/html/2606.17312#S1.F1)illustrates this gap: when all sampled responses agree on the same wrong answer, dispersion\-based methods report low uncertainty, while self\-preference rankings across trials can reveal instability not reflected by dispersion alone\.

We propose a consistency\-aware evaluation framework for logical reasoning that measures the stability of self\-preference\-induced rankings over sampled candidate solutions\. Given a query, we sample multiple reasoning candidates and ask the*same model*to judge pairwise preferences among its own outputs\. Beyond how much responses*differ*, this probes whether the model forms a stable or fluctuating preference ordering over competing reasoning paths—a signal that dispersion alone discards\. We aggregate sparse pairwise judgments into ranking distributions using Bradley–Terry modeling\(Bradley and Terry,[1952](https://arxiv.org/html/2606.17312#bib.bib34)\)with PageRank normalization\(Langville and Meyer,[2006](https://arxiv.org/html/2606.17312#bib.bib38)\), repeated across random spanning\-tree comparison trials\. This produces two distinct components of structural uncertainty:*across\-trial ranking instability*\(reasoning instability across elicitation trials\) and*within\-trial candidate ambiguity*\(ambiguity among plausible reasoning candidates within each trial\)\.

Across five LLMs and eight benchmarks, we show that self\-preference\-derived structural signals provide information complementary to output dispersion\. The interaction with accuracy is task\-dependent: on mathematical reasoning, within\-trial ambiguity can correlate positively with correctness—consistent with settings where several plausible solution paths remain competitive—while on factual retrieval, structural signals collapse toward uniformity, diagnosing a regime where reasoning\-level consistency evaluation is uninformative\. Combining structural and dispersion\-based signals improves identification of unreliable reasoning instances on several reasoning and knowledge tasks, though gains are absent in the collapse regime\.

##### Contributions\.

\(1\) A post\-hoc framework for evaluating logical reasoning consistency and reliability\.We propose a model\-agnostic framework that quantifies reasoning consistency through the stability of a model’s self\-preference rankings over its own candidate solutions, providing an observable signal complementary to output dispersion\.\(2\) A structural decomposition of reasoning stability\.We aggregate sparse pairwise self\-preferences into ranking distributions via Bradley–Terry with PageRank, yielding two complementary entropy\-based components: across\-trial ranking instability \(reasoning instability\) and within\-trial candidate ambiguity \(ambiguity among plausible reasoning candidates\)\.\(3\) A regime analysis distinguishing reasoning from retrieval settings\.Across five LLMs and eight benchmarks, we show that structural signals complement dispersion on reasoning tasks, identify when and why the signal collapses on retrieval tasks, and characterize the task conditions under which each signal type is most informative for evaluating logical reasoning consistency\.

## 2Related Work

##### Logical reasoning and self\-consistency in LLMs\.

Chain\-of\-thought prompting and self\-consistency\(Wanget al\.,[2023](https://arxiv.org/html/2606.17312#bib.bib59)\)have become standard approaches for improving and evaluating LLM reasoning\. Self\-consistency measures agreement across multiple sampled reasoning paths, while debate and self\-judge frameworks\(Zhenget al\.,[2023](https://arxiv.org/html/2606.17312#bib.bib57); Kadavathet al\.,[2022](https://arxiv.org/html/2606.17312#bib.bib58)\)demonstrate that models can assess output quality\. However, self\-consistency treats responses as exchangeable and measures only answer\-level agreement, missing structural differences in reasoning quality among candidates\. We complement answer\-level consistency by measuring how stably the model ranks competing reasoning solutions through self\-preference\.

##### Post\-hoc uncertainty and preference\-based evaluation\.

Dispersion\-based methods estimate uncertainty from semantic variation among responses\(Kuhnet al\.,[2023](https://arxiv.org/html/2606.17312#bib.bib60); Linet al\.,[2023](https://arxiv.org/html/2606.17312#bib.bib3); Farquharet al\.,[2024](https://arxiv.org/html/2606.17312#bib.bib4); Kossenet al\.,[2024](https://arxiv.org/html/2606.17312#bib.bib5)\), output density\(Qiu and Miikkulainen,[2024](https://arxiv.org/html/2606.17312#bib.bib6)\), kernelized entropy\(Nikitinet al\.,[2024](https://arxiv.org/html/2606.17312#bib.bib7)\), or self\-consistency entropy\(Wanget al\.,[2024a](https://arxiv.org/html/2606.17312#bib.bib24); Lyuet al\.,[2025](https://arxiv.org/html/2606.17312#bib.bib2)\)\. Comparison\-based methods aggregate pairwise preferences into calibrated scores\(Shrivastavaet al\.,[2025](https://arxiv.org/html/2606.17312#bib.bib10)\)or incorporate richer structure via multi\-dimensional representations\(Chenet al\.,[2025](https://arxiv.org/html/2606.17312#bib.bib12)\), knowledge graphs\(Yuanet al\.,[2025](https://arxiv.org/html/2606.17312#bib.bib11)\), or Minimum Bayes Risk\(Vashurinet al\.,[2025a](https://arxiv.org/html/2606.17312#bib.bib33)\)\. When internal access is available, logit\-based\(Maet al\.,[2025](https://arxiv.org/html/2606.17312#bib.bib13)\), chain\-of\-thought\(Zhang and Zhang,[2025](https://arxiv.org/html/2606.17312#bib.bib14)\), and proxy\-based methods\(Leeet al\.,[2024](https://arxiv.org/html/2606.17312#bib.bib15)\)derive uncertainty from model internals\. Information\-theoretic and Bayesian perspectives motivate principled decomposition\(Abbasi Yadkoriet al\.,[2024](https://arxiv.org/html/2606.17312#bib.bib1); Kendall and Gal,[2017](https://arxiv.org/html/2606.17312#bib.bib40); Woo,[2022](https://arxiv.org/html/2606.17312#bib.bib43);[2023](https://arxiv.org/html/2606.17312#bib.bib30)\)\. Our approach operates in a fully black\-box setting without requiring internal access or model modification\.

##### Consistency and contradiction in LLM outputs\.

A growing body of work addresses logical contradictions and inconsistencies in LLM outputs\. Evaluation frameworks assess alignment between uncertainty and quality\(Huanget al\.,[2024](https://arxiv.org/html/2606.17312#bib.bib21); Yeet al\.,[2024](https://arxiv.org/html/2606.17312#bib.bib22); Vashurinet al\.,[2025b](https://arxiv.org/html/2606.17312#bib.bib32)\), while studies reveal strong task\- and model\-dependence in LLM reliability\(Huanget al\.,[2023](https://arxiv.org/html/2606.17312#bib.bib23); Yanget al\.,[2025](https://arxiv.org/html/2606.17312#bib.bib31)\)\. Our work contributes to this direction by providing a structural lens on reasoning consistency: rather than checking whether outputs contradict each other at the answer level, we measure whether the model can form a stable preference ordering over its own reasoning candidates—directly revealing internal inconsistency in reasoning evaluation\. Unlike approaches that improve reasoning via symbolic modules or external solvers, we focus on post\-hoc evaluation of reasoning consistency in a pure black\-box setting\.

![Refer to caption](https://arxiv.org/html/2606.17312v1/images/method.png)Figure 2:Overview of the consistency\-aware reasoning evaluation framework\. Given a query, we \(1\) generate diverse candidate reasoning solutions, \(2\) elicit pairwise self\-preferences, \(3\) aggregate into a global ranking via pairwise preference modeling \(Bradley–Terry or TrueSkill\) with PageRank, and \(4\) decompose the consistency signal via random spanning tree sampling\.

## 3Method

Our framework quantifies consistency in logical reasoning via the stability of self\-preference\-induced rankings over sampled candidate solutions\. Our notion of consistency is same\-query and candidate\-relative: we ask whether the model forms a stable preference ordering over multiple reasoning solutions to the same problem\. Given a query, we: \(1\) generateNNdiverse candidate reasoning solutions, \(2\) elicit pairwise self\-preferences by asking the model to judge its own outputs, \(3\) aggregate preferences into a global ranking via Bradley–Terry with PageRank, and \(4\) decompose the consistency signal into across\-trial and within\-trial components through random spanning tree sampling\. Figure[2](https://arxiv.org/html/2606.17312#S2.F2)illustrates the pipeline\.

### 3\.1Self\-Preference via Spanning Trees

Given inputxx, we sampleNNcandidatesℛ​\(x\)=\{r1,…,rN\}\\mathcal\{R\}\(x\)=\\\{r\_\{1\},\\ldots,r\_\{N\}\\\}from the model’s conditional distributionpθ\(⋅\|x\)p\_\{\\theta\}\(\\cdot\|x\)via diverse prompting with stochastic decoding\. Rather than comparing all\(N2\)\\binom\{N\}\{2\}pairs, we repeat the following forMMindependent trials \(m=1,…,Mm=1,\\ldots,M\):

Sparse graph sampling\.We draw a uniform random spanning tree𝒯\(m\)\\mathcal\{T\}^\{\(m\)\}over theNNcandidates, yielding a connected graph with exactlyN−1N\{\-\}1edges\. This guarantees global connectivity with minimal comparisons while injecting structural randomness across trials, loosely analogous in spirit to Monte Carlo dropout over graph structure rather than weights\.

Self\-preference elicitation\.For each edge\(i,j\)∈𝒯\(m\)\(i,j\)\\in\\mathcal\{T\}^\{\(m\)\}, we query the*same model*to judge which response is better, optionally obtaining a confidence score\. The consistency of these elicited self\-preference judgments across trials provides an additional uncertainty signal that is not directly available from output dispersion alone\.

Preference aggregation\.We fit a pairwise preference model \(Bradley–Terry or TrueSkill\) on theN−1N\{\-\}1observed comparisons to infer win probabilities for*all*N2N^\{2\}pairs, then aggregate into a global ranking distribution𝝅\(m\)\\boldsymbol\{\\pi\}^\{\(m\)\}via PageRank\. Details follow in Section[3\.2](https://arxiv.org/html/2606.17312#S3.SS2)\.

The ensemble\{𝝅\(m\)\}m=1M\\\{\\boldsymbol\{\\pi\}^\{\(m\)\}\\\}\_\{m=1\}^\{M\}fromMMtrials enables the uncertainty decomposition in Section[3\.3](https://arxiv.org/html/2606.17312#S3.SS3)\.

### 3\.2Preference Aggregation

Bradley–Terry with L2 regularization\.We assign each candidateiia latent utilityθi∈ℝ\\theta\_\{i\}\\in\\mathbb\{R\}and model pairwise preferences as\(Bradley and Terry,[1952](https://arxiv.org/html/2606.17312#bib.bib34)\):

ℙ​\(i≻j\)=exp⁡\(θi\)exp⁡\(θi\)\+exp⁡\(θj\)\.\\mathbb\{P\}\(i\\succ j\)=\\frac\{\\exp\(\\theta\_\{i\}\)\}\{\\exp\(\\theta\_\{i\}\)\+\\exp\(\\theta\_\{j\}\)\}\.\(1\)Since spanning trees admit perfect total orderings, the unregularized maximum likelihood objective is unbounded\(Ford,[1957](https://arxiv.org/html/2606.17312#bib.bib50)\)\. We add anL2L^\{2\}\-penalty for numerical well\-posedness:

ℒreg​\(𝜽\)=∑\(i,j\)∈𝒯\(m\)log⁡ℙ​\(i≻j\)−12​C​‖𝜽‖2,\\mathcal\{L\}\_\{\\text\{reg\}\}\(\\boldsymbol\{\\theta\}\)=\\sum\_\{\(i,j\)\\in\\mathcal\{T\}^\{\(m\)\}\}\\log\\mathbb\{P\}\(i\\succ j\)\-\\frac\{1\}\{2C\}\\\|\\boldsymbol\{\\theta\}\\\|^\{2\},\(2\)whereC\>0C\>0is the inverse regularization strength\. We setC=1C=1based on an ablation sweep \(Appendix[A\.3](https://arxiv.org/html/2606.17312#A1.SS3)\): performance degrades belowC<1C<1but plateaus stably forC≥1C\\geq 1across all models, confirming bounded parameter estimates\. We optimize via MM\-style updates\(Hunter,[2004](https://arxiv.org/html/2606.17312#bib.bib35)\)and evaluate Eq\. \([1](https://arxiv.org/html/2606.17312#S3.E1)\) for all pairs to obtain𝐏\(m\)∈\[0,1\]N×N\\mathbf\{P\}^\{\(m\)\}\\in\[0,1\]^\{N\\times N\}\.

As a robustness check, we also implement a TrueSkill variant\(Herbrichet al\.,[2006](https://arxiv.org/html/2606.17312#bib.bib49)\)with confidence\-weighted updates\. Despite fundamentally different assumptions, both backends produce highly similar uncertainty rankings \(Appendix[A\.4](https://arxiv.org/html/2606.17312#A1.SS4)\), suggesting that the induced ranking distributions are reasonably stable with respect to the choice of preference backend\.

PageRank global ranking\.While BT interpolates sparse comparisons into a dense preference matrix𝐏\(m\)\\mathbf\{P\}^\{\(m\)\}, PageRank summarizes it into a normalized ranking distribution𝝅\(m\)∈ΔN\\boldsymbol\{\\pi\}^\{\(m\)\}\\in\\Delta^\{N\}on which we define the entropy\-based decomposition \(Section[3\.3](https://arxiv.org/html/2606.17312#S3.SS3)\)\. From𝐏\(m\)\\mathbf\{P\}^\{\(m\)\}, we construct a row\-stochastic transition matrix where probability mass flows from weaker to stronger candidates:

Ti​j\(m\)∝Pj​i\(m\),Ti​i\(m\)=0,T^\{\(m\)\}\_\{ij\}\\propto P^\{\(m\)\}\_\{ji\},\\quad T^\{\(m\)\}\_\{ii\}=0,\(3\)with row normalization\. We compute the stationary distribution𝝅\(m\)\\boldsymbol\{\\pi\}^\{\(m\)\}satisfying𝝅\(m\)=\(𝐓\(m\)\)⊤​𝝅\(m\)\\boldsymbol\{\\pi\}^\{\(m\)\}=\(\\mathbf\{T\}^\{\(m\)\}\)^\{\\top\}\\boldsymbol\{\\pi\}^\{\(m\)\}via power iteration\(Brin and Page,[1998](https://arxiv.org/html/2606.17312#bib.bib37); Langville and Meyer,[2006](https://arxiv.org/html/2606.17312#bib.bib38)\)\.

### 3\.3Structural Uncertainty Decomposition

TheMMtrials yield an ensemble\{𝝅\(m\)\}m=1M\\\{\\boldsymbol\{\\pi\}^\{\(m\)\}\\\}\_\{m=1\}^\{M\}with mean𝝅¯=1M​∑m𝝅\(m\)\\bar\{\\boldsymbol\{\\pi\}\}=\\tfrac\{1\}\{M\}\\sum\_\{m\}\\boldsymbol\{\\pi\}^\{\(m\)\}\. We decompose total uncertainty via Shannon entropyH​\[𝒑\]=−∑ipi​log⁡piH\[\\boldsymbol\{p\}\]=\-\\sum\_\{i\}p\_\{i\}\\log p\_\{i\}followingKendall and Gal \([2017](https://arxiv.org/html/2606.17312#bib.bib40)\); Woo \([2022](https://arxiv.org/html/2606.17312#bib.bib43)\):

StructU=H​\[𝝅¯\],\\displaystyle=H\[\\bar\{\\boldsymbol\{\\pi\}\}\],\(4\)StructUwithin\\displaystyle\\text\{StructU\}\_\{\\text\{within\}\}=1M​∑m=1MH​\[𝝅\(m\)\],\\displaystyle=\\tfrac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}H\[\\boldsymbol\{\\pi\}^\{\(m\)\}\],\(5\)StructUacross\\displaystyle\\text\{StructU\}\_\{\\text\{across\}\}=StructU−StructUwithin\.\\displaystyle=\\text\{StructU\}\-\\text\{StructU\}\_\{\\text\{within\}\}\.\(6\)Here,StructUwithin\\mathrm\{StructU\}\_\{\\text\{within\}\}is intended to reflect*within\-trial candidate ambiguity*: it is high when, within a single sparse comparison trial, the ranking distribution spreads its mass over multiple candidates instead of concentrating on a single preferred response\. Meanwhile,StructUacross\\mathrm\{StructU\}\_\{\\text\{across\}\}is intended to reflect*across\-trial ranking instability*: it is high when different sampled comparison trees lead to substantially different ranking distributions across trials\. Figure[3](https://arxiv.org/html/2606.17312#S3.F3)illustrates the two components with contrasting examples\.

Concretely,H​\[𝝅¯\]H\[\\bar\{\\boldsymbol\{\\pi\}\}\]is the entropy of the trial\-averaged ranking; the average1M​∑mH​\[𝝅\(m\)\]\\frac\{1\}\{M\}\\sum\_\{m\}H\[\\boldsymbol\{\\pi\}^\{\(m\)\}\]measures within\-trial spread; and their difference—a Jensen gap—provides a distribution\-level measure of cross\-trial variation\. This decomposition is defined entirely over the observable ranking ensemble induced by self\-preference; we do not claim it identifies underlying data or parameter uncertainty\.

![Refer to caption](https://arxiv.org/html/2606.17312v1/images/structuwithin_structuaccross_v3.png)Figure 3:Across\-trial vs\. within\-trial structural uncertainty: two contrasting examples\.Scenario I \(left\):The model produces a confident, concentrated ranking within each trial \(low within\-trial entropyH​\[𝝅\(m\)\]H\[\\boldsymbol\{\\pi\}^\{\(m\)\}\]\), but the dominant candidate changes across trials as different spanning trees are sampled—indicating substantial across\-trial instability in the induced ranking distribution \(𝐒𝐭𝐫𝐮𝐜𝐭𝐔across↑\\mathbf\{StructU\}\_\{\\mathrm\{across\}\}\\uparrow,𝐒𝐭𝐫𝐮𝐜𝐭𝐔within↓\\mathbf\{StructU\}\_\{\\mathrm\{within\}\}\\downarrow\)\.Scenario II \(right\):The ranking distribution is nearly uniform in every trial \(high within\-trial entropy\), yet remains consistent across trials—indicating the model is stably uncertain rather than inconsistently confident \(𝐒𝐭𝐫𝐮𝐜𝐭𝐔across↓\\mathbf\{StructU\}\_\{\\mathrm\{across\}\}\\downarrow,𝐒𝐭𝐫𝐮𝐜𝐭𝐔within↑\\mathbf\{StructU\}\_\{\\mathrm\{within\}\}\\uparrow\)\. Node size reflects PageRank score𝝅\(m\)\\boldsymbol\{\\pi\}^\{\(m\)\}; arrows indicate self\-preference direction on the sampled spanning tree edges; dashed edges are unobserved pairs\.Combination with self\-consistency\.For practical reasoning reliability evaluation, we also study a simple fixed combination of structural uncertainty with self\-consistency entropySelf\-ConsU=−∑a∈𝒜p​\(a\)​log⁡p​\(a\)\\text\{Self\-ConsU\}=\-\\sum\_\{a\\in\\mathcal\{A\}\}p\(a\)\\log p\(a\)over the answer distribution, where𝒜\\mathcal\{A\}denotes distinct answers\. AbbreviatingStructUacross\\text\{StructU\}\_\{\\text\{across\}\}asSUa\\text\{SU\}\_\{\\text\{a\}\}andStructUwithin\\text\{StructU\}\_\{\\text\{within\}\}asSUw\\text\{SU\}\_\{\\text\{w\}\}:

SUa​\+SC\\displaystyle\\text\{SU\}\_\{\\text\{a\}\}\\text\{\+SC\}=SUa\+Self\-ConsU,\\displaystyle=\\text\{SU\}\_\{\\text\{a\}\}\+\\text\{Self\-ConsU\},\(7\)SUw​\+SC\\displaystyle\\text\{SU\}\_\{\\text\{w\}\}\\text\{\+SC\}=SUw−Self\-ConsU,\\displaystyle=\\text\{SU\}\_\{\\text\{w\}\}\-\\text\{Self\-ConsU\},\(8\)where SC abbreviates Self\-ConsU\. The sign reflects each component’s empirical relationship with accuracy:SUa\\text\{SU\}\_\{\\text\{a\}\}correlates negatively \(instability→\\tofailure\), so it adds with Self\-ConsU;SUw\\text\{SU\}\_\{\\text\{w\}\}correlates positively on reasoning tasks, so it enters subtractively\. This assignment is fixed globally—not tuned per task or model\. We treat this as an empirical fusion rule rather than an intrinsic part of the decomposition; structural uncertainty captures*how*self\-preferences are organized, while Self\-ConsU captures*what*answers disagree\.

## 4Experiments

### 4\.1Experimental Setup

We evaluate five LLMs \(Claude Sonnet 4\.5, GPT\-OSS 20B, Qwen 3 32B, Amazon Nova Premier, DeepSeek R1\) on eight benchmarks grouped by reasoning structure:*mathematical and logical reasoning*\(Math\-Synth, MATH\-500\(Lightmanet al\.,[2024](https://arxiv.org/html/2606.17312#bib.bib54)\), AMC\-23\(He,[2023](https://arxiv.org/html/2606.17312#bib.bib53)\), AIME\-24/25\(HuggingFaceH4,[2024](https://arxiv.org/html/2606.17312#bib.bib51); TIGER\-Lab,[2024](https://arxiv.org/html/2606.17312#bib.bib52)\)\),*reasoning\-adjacent knowledge tasks*\(MMLU\-Pro\(Wanget al\.,[2024b](https://arxiv.org/html/2606.17312#bib.bib55)\), TruthfulQA\(Linet al\.,[2022](https://arxiv.org/html/2606.17312#bib.bib61)\)\), and a*retrieval\-dominant comparison regime*\(HotpotQA\(Yanget al\.,[2018](https://arxiv.org/html/2606.17312#bib.bib56)\)\)\. Math\-Synth is a synthetic arithmetic benchmark with 993 problems \(Appendix[B](https://arxiv.org/html/2606.17312#A2)\); other dataset details and model accuracies are in Table[4](https://arxiv.org/html/2606.17312#A4.T4)\.

We compare against black\-box baselines computed from the sameN=5N\{=\}5samples:Self\-ConsU\(answer entropy\)\(Wanget al\.,[2024a](https://arxiv.org/html/2606.17312#bib.bib24); Lyuet al\.,[2025](https://arxiv.org/html/2606.17312#bib.bib2)\),SemanticU\(embedding dispersion\)\(Qiu and Miikkulainen,[2024](https://arxiv.org/html/2606.17312#bib.bib6); Kossenet al\.,[2024](https://arxiv.org/html/2606.17312#bib.bib5)\), andVerbalizedU\(prompted confidence\)\(Tianet al\.,[2023](https://arxiv.org/html/2606.17312#bib.bib20); Xionget al\.,[2024](https://arxiv.org/html/2606.17312#bib.bib42)\)\. See Appendix[D](https://arxiv.org/html/2606.17312#A4)for details\.

We use selective prediction metrics as an operational measure of whether the proposed consistency signal identifies unreliable logical reasoning instances: questions wherep^corr=1N​∑i𝕀​\[ri​correct\]<τ\\hat\{p\}\_\{\\mathrm\{corr\}\}=\\tfrac\{1\}\{N\}\\sum\_\{i\}\\mathbb\{I\}\[r\_\{i\}\\text\{ correct\}\]<\\tauwithτ=1\.0\\tau\{=\}1\.0\. We report Spearman correlation, AUROC, and area under the risk\-coverage curve \(Sel\-AUC\)\(Shrivastavaet al\.,[2023](https://arxiv.org/html/2606.17312#bib.bib18)\)\.

### 4\.2Results

We evaluate structural uncertainty as a consistency signal for logical reasoning across five models and eight benchmarks, analyzing where it improves reasoning reliability evaluation, how the components relate to accuracy, and when the signal breaks down\. Our key target failure mode is systematic but internally unstable reasoning: candidate solutions may agree at the answer level while remaining inconsistent in how the model ranks them\.

##### Overall Evaluation\.

Table[1](https://arxiv.org/html/2606.17312#S4.T1)reports reasoning reliability evaluation performance \(Sel\-AUC; AUROC in parentheses\)\. The central finding is*task\-dependent complementarity*: on reasoning\-heavy and some knowledge tasks, structural consistency signals add information beyond answer dispersion, while on retrieval\-style tasks the structural signal collapses and provides limited benefit\. Reasoning tasks admit structurally diverse solution paths, making self\-preference consistency informative in logical reasoning regimes\. Specifically, the combined estimator \(StructU\+Self\-ConsU\) achieves highest or second\-highest Sel\-AUC on mathematical reasoning \(Math\-Synth, MATH\-500, AMC\-23\) and knowledge tasks \(MMLU\-Pro, TruthfulQA\), with largest gains where answer\-level agreement is insufficient\. On HotpotQA, retrieval tasks suppress structural diversity, so the signal collapses—dispersion methods dominate for the two strongest models \(Claude: Self\-ConsU 0\.839 vs\. combined 0\.742; DeepSeek: SemanticU 0\.852 vs\. combined 0\.789\)\. This task asymmetry is itself informative: it identifies the regime boundary where reasoning\-level consistency evaluation ceases to be useful, making structural uncertainty a regime\-sensitive evaluator of reasoning consistency rather than a universal confidence estimator\. Among baselines, Self\-ConsU is strongest but blind to systematic reasoning errors; VerbalizedU is inconsistent; SemanticU is weakest except where structural signals collapse\.

DatasetModelStructU \(Ours\)StructU\+Self\-ConsU \(Ours\)BaselineswithinacrosstotalwithinacrosstotalSelf\-ConsUVerbalizedUSemanticUMath\-SynthClaude 4\.5 Sonnet0\.624 \(0\.984\)0\.644 \(0\.936\)0\.596 \(0\.972\)0\.661 \(0\.992\)0\.663 \(0\.990\)0\.655 \(0\.992\)0\.65 \(0\.989\)0\.544 \(0\.924\)0\.430 \(0\.927\)DeepSeek R10\.815 \(0\.900\)0\.790 \(0\.837\)0\.773 \(0\.790\)0\.823 \(0\.929\)0\.820 \(0\.924\)0\.814 \(0\.915\)0\.802 \(0\.899\)0\.793 \(0\.701\)0\.664 \(0\.502\)GPT\-OSS 20B0\.794 \(0\.756\)0\.762 \(0\.668\)0\.661 \(0\.366\)0\.840 \(0\.955\)0\.849 \(0\.956\)0\.849 \(0\.958\)0\.83 \(0\.958\)0\.792 \(0\.742\)0\.528 \(0\.638\)Amazon Nova Premier0\.496 \(0\.953\)0\.370 \(0\.841\)0\.447 \(0\.853\)0\.511 \(0\.997\)0\.498 \(0\.998\)0\.512 \(0\.997\)0\.382 \(0\.948\)0\.389 \(0\.824\)0\.436 \(0\.807\)Qwen 3 32B0\.231 \(0\.794\)0\.190 \(0\.656\)0\.213 \(0\.718\)0\.393 \(0\.998\)0\.391 \(0\.998\)0\.388 \(0\.998\)0\.38 \(0\.995\)0\.279 \(0\.824\)0\.218 \(0\.462\)MATH\-500Claude 4\.5 Sonnet0\.931 \(0\.813\)0\.936 \(0\.789\)0\.932 \(0\.801\)0\.947 \(0\.840\)0\.950 \(0\.843\)0\.948 \(0\.831\)0\.942 \(0\.816\)0\.891 \(0\.686\)0\.783 \(0\.720\)DeepSeek R10\.889 \(0\.652\)0\.890 \(0\.640\)0\.868 \(0\.596\)0\.934 \(0\.776\)0\.936 \(0\.784\)0\.927 \(0\.767\)0\.923 \(0\.759\)0\.870 \(0\.603\)0\.875 \(0\.546\)GPT\-OSS 20B0\.869 \(0\.582\)0\.885 \(0\.529\)0\.886 \(0\.611\)0\.897 \(0\.715\)0\.910 \(0\.718\)0\.906 \(0\.729\)0\.871 \(0\.694\)0\.886 \(0\.645\)0\.86 \(0\.46\)Amazon Nova Premier0\.834 \(0\.747\)0\.726 \(0\.609\)0\.821 \(0\.687\)0\.887 \(0\.873\)0\.880 \(0\.871\)0\.886 \(0\.870\)0\.860 \(0\.839\)0\.883 \(0\.715\)0\.81 \(0\.695\)Qwen 3 32B0\.859 \(0\.798\)0\.780 \(0\.608\)0\.833 \(0\.754\)0\.889 \(0\.882\)0\.882 \(0\.853\)0\.888 \(0\.876\)0\.871 \(0\.817\)0\.880 \(0\.684\)0\.819 \(0\.717\)AMC\-23Claude 4\.5 Sonnet0\.967 \(1\.000\)0\.966 \(0\.963\)0\.962 \(0\.980\)0\.970 \(1\.000\)0\.972 \(1\.000\)0\.970 \(1\.000\)0\.955 \(1\.00\)0\.900 \(0\.604\)0\.853 \(0\.588\)DeepSeek R10\.924 \(0\.680\)0\.915 \(0\.463\)0\.923 \(0\.291\)0\.985 \(1\.000\)0\.985 \(1\.000\)0\.985 \(1\.000\)0\.985 \(1\.00\)0\.877 \(0\.412\)0\.880 \(0\.592\)GPT\-OSS 20B0\.956 \(0\.770\)0\.931 \(0\.533\)0\.954 \(0\.673\)0\.980 \(1\.000\)0\.980 \(1\.000\)0\.980 \(1\.000\)0\.980 \(1\.00\)0\.980 \(0\.583\)0\.884 \(0\.630\)Amazon Nova Premier0\.591 \(0\.865\)0\.538 \(0\.604\)0\.610 \(0\.919\)0\.712 \(1\.000\)0\.716 \(1\.000\)0\.718 \(1\.000\)0\.584 \(1\.00\)0\.419 \(0\.362\)0\.351 \(0\.410\)Qwen 3 32B0\.637 \(0\.643\)0\.566 \(0\.567\)0\.606 \(0\.643\)0\.820 \(1\.000\)0\.821 \(1\.000\)0\.820 \(1\.000\)0\.810 \(1\.00\)0\.637 \(0\.389\)0\.513 \(0\.299\)AIME\-24Claude 4\.5 Sonnet0\.497 \(0\.903\)0\.595 \(0\.852\)0\.508 \(0\.875\)0\.662 \(0\.994\)0\.690 \(0\.989\)0\.661 \(0\.994\)0\.567 \(0\.972\)0\.273 \(0\.144\)0\.45 \(0\.64\)DeepSeek R10\.834 \(0\.752\)0\.820 \(0\.554\)0\.866 \(0\.884\)0\.922 \(1\.000\)0\.925 \(1\.000\)0\.922 \(1\.000\)0\.917 \(1\.00\)0\.799 \(0\.298\)0\.788 \(0\.715\)GPT\-OSS 20B0\.604 \(0\.725\)0\.762 \(0\.714\)0\.756 \(0\.813\)0\.893 \(1\.000\)0\.907 \(1\.000\)0\.896 \(1\.000\)0\.891 \(1\.00\)0\.905 \(0\.319\)0\.876 \(0\.681\)Amazon Nova Premier0\.190 \(—\)0\.267 \(—\)0\.170 \(—\)0\.181 \(—\)0\.210 \(—\)0\.189 \(—\)0\.174 \(–\)0\.125 \(–\)0\.294 \(–\)Qwen 3 32B0\.235 \(—\)0\.243 \(—\)0\.257 \(—\)0\.401 \(—\)0\.428 \(—\)0\.402 \(—\)0\.414 \(–\)0\.355 \(–\)0\.213 \(1\.00\)AIME\-25Claude 4\.5 Sonnet0\.565 \(0\.975\)0\.602 \(0\.969\)0\.221 \(0\.062\)0\.645 \(1\.00\)0\.646 \(1\.00\)0\.656 \(1\.00\)0\.645 \(1\.00\)0\.489 \(0\.175\)0\.524 \(0\.263\)DeepSeek R10\.263 \(0\.413\)0\.244 \(0\.259\)0\.539 \(0\.466\)0\.728 \(1\.00\)0\.740 \(1\.00\)0\.761 \(1\.00\)0\.707 \(1\.00\)0\.564 \(0\.296\)0\.616 \(0\.400\)GPT\-OSS 20B0\.424 \(0\.449\)0\.392 \(0\.324\)0\.696 \(0\.546\)0\.881 \(1\.00\)0\.884 \(1\.00\)0\.886 \(1\.00\)0\.825 \(1\.00\)0\.776 \(0\.370\)0\.768 \(0\.491\)Amazon Nova Premier0\.203 \(–\)0\.149 \(–\)0\.059 \(–\)0\.198 \(–\)0\.218 \(–\)0\.175 \(–\)0\.178 \(–\)0\.106 \(–\)0\.257 \(–\)Qwen 3 32B0\.096 \(0\.862\)0\.179 \(0\.897\)0\.234 \(0\.414\)0\.203 \(1\.00\)0\.282 \(1\.00\)0\.328 \(1\.00\)0\.208 \(1\.00\)0\.192 \(0\.897\)0\.126 \(0\.931\)MMLU\-ProClaude 4\.5 Sonnet0\.924 \(0\.833\)0\.913 \(0\.785\)0\.916 \(0\.792\)0\.944 \(0\.912\)0\.936 \(0\.897\)0\.943 \(0\.908\)0\.900 \(0\.884\)0\.944 \(0\.885\)0\.890 \(0\.602\)DeepSeek R10\.845 \(0\.573\)0\.849 \(0\.500\)0\.855 \(0\.596\)0\.925 \(0\.889\)0\.917 \(0\.882\)0\.924 \(0\.895\)0\.882 \(0\.882\)0\.927 \(0\.796\)0\.870 \(0\.577\)GPT\-OSS 20B0\.765 \(0\.503\)0\.754 \(0\.547\)0\.749 \(0\.475\)0\.889 \(0\.948\)0\.880 \(0\.941\)0\.886 \(0\.945\)0\.830 \(0\.935\)0\.785 \(0\.631\)0\.774 \(0\.573\)Amazon Nova Premier0\.671 \(0\.633\)0\.696 \(0\.507\)0\.679 \(0\.624\)0\.805 \(0\.936\)0\.827 \(0\.948\)0\.800 \(0\.936\)0\.801 \(0\.945\)0\.820 \(0\.775\)0\.811 \(0\.713\)Qwen 3 32B0\.713 \(0\.669\)0\.657 \(0\.567\)0\.685 \(0\.624\)0\.818 \(0\.971\)0\.805 \(0\.964\)0\.817 \(0\.971\)0\.787 \(0\.966\)0\.728 \(0\.724\)0\.742 \(0\.686\)HotpotQAClaude 4\.5 Sonnet0\.686 \(0\.585\)0\.731 \(0\.600\)0\.664 \(0\.564\)0\.698 \(0\.617\)0\.742 \(0\.647\)0\.681 \(0\.604\)0\.839 \(0\.656\)0\.847 \(0\.700\)0\.768 \(0\.576\)DeepSeek R10\.732 \(0\.580\)0\.747 \(0\.514\)0\.744 \(0\.614\)0\.767 \(0\.683\)0\.789 \(0\.666\)0\.771 \(0\.693\)0\.835 \(0\.658\)0\.829 \(0\.708\)0\.852 \(0\.696\)GPT\-OSS 20B0\.724 \(0\.537\)0\.717 \(0\.524\)0\.736 \(0\.568\)0\.815 \(0\.728\)0\.798 \(0\.724\)0\.815 \(0\.734\)0\.813 \(0\.721\)0\.772 \(0\.592\)0\.80 \(0\.707\)Amazon Nova Premier0\.806 \(0\.582\)0\.805 \(0\.589\)0\.787 \(0\.541\)0\.850 \(0\.752\)0\.840 \(0\.748\)0\.855 \(0\.750\)0\.864 \(0\.740\)0\.812 \(0\.647\)0\.830 \(0\.618\)Qwen 3 32B0\.699 \(0\.648\)0\.679 \(0\.528\)0\.691 \(0\.618\)0\.766 \(0\.757\)0\.772 \(0\.713\)0\.778 \(0\.765\)0\.787 \(0\.729\)0\.702 \(0\.630\)0\.707 \(0\.612\)TruthfulQAClaude 4\.5 Sonnet0\.998 \(0\.956\)0\.997 \(0\.912\)0\.998 \(0\.953\)0\.998 \(0\.970\)0\.998 \(0\.966\)0\.998 \(0\.971\)0\.994 \(0\.926\)0\.997 \(0\.949\)0\.979 \(0\.503\)DeepSeek R10\.932 \(0\.564\)0\.941 \(0\.662\)0\.874 \(0\.535\)0\.954 \(0\.884\)0\.957 \(0\.888\)0\.910 \(0\.813\)0\.940 \(0\.844\)0\.949 \(0\.721\)0\.907 \(0\.530\)GPT\-OSS 20B0\.913 \(0\.717\)0\.893 \(0\.611\)0\.909 \(0\.727\)0\.943 \(0\.955\)0\.942 \(0\.955\)0\.941 \(0\.952\)0\.916 \(0\.936\)0\.864 \(0\.608\)0\.856 \(0\.525\)Amazon Nova Premier0\.956 \(0\.806\)0\.930 \(0\.689\)0\.952 \(0\.779\)0\.963 \(0\.914\)0\.951 \(0\.893\)0\.962 \(0\.914\)0\.946 \(0\.862\)0\.953 \(0\.828\)0\.907 \(0\.596\)Qwen 3 32B0\.783 \(0\.510\)0\.787 \(0\.534\)0\.765 \(0\.499\)0\.869 \(0\.956\)0\.867 \(0\.953\)0\.855 \(0\.949\)0\.856 \(0\.936\)0\.822 \(0\.694\)0\.812 \(0\.636\)

Table 1:Selective prediction performance \(Sel\-AUC; AUROC in parentheses\)\.Results using Bradley–Terry with PageRank aggregation\. StructU reports structural uncertainty \(within, across components\)\. StructU\+Self\-ConsU reports the combined estimators\. Baselines: Self\-ConsU \(answer entropy\), VerbalizedU \(prompted confidence\), SemanticU \(embedding dispersion\)\. Bold = best; underline = second\-best per row\.
##### Where Structural Consistency Signals Help\.

Figure[4](https://arxiv.org/html/2606.17312#S4.F4)showsΔ​Sel\-AUC=Sel\-AUC​\(StructU\+Self\-ConsU\)−Sel\-AUC​\(Self\-ConsU\)\\Delta\\text\{Sel\-AUC\}=\\text\{Sel\-AUC\}\(\\textsc\{StructU\+Self\-ConsU\}\)\-\\text\{Sel\-AUC\}\(\\text\{Self\-ConsU\}\)\. Gains are consistent across logical reasoning and knowledge benchmarks, and largest where answer\-level agreement alone is insufficient to identify unreliable reasoning: weaker models on hard contest problems gain the most \(Qwen on AIME\-25:\+12\.0%\+12\.0\\%, Amazon Nova Premier on AMC\-23:\+13\.4%\+13\.4\\%\), as structural consistency rankings surface signal not captured by Self\-ConsU alone\. HotpotQA is the exception—stronger models show negative lift \(Claude Sonnet 4\.5:−9\.7%\-9\.7\\%, DeepSeek R1:−4\.6%\-4\.6\\%\), consistent with the structural collapse on retrieval tasks\.

![Refer to caption](https://arxiv.org/html/2606.17312v1/images/heatmap_selauc.png)Figure 4:Sel\-AUC lift ofStructU\+Self\-ConsUoverSelf\-ConsU\.Δ​Sel\-AUC\\Delta\\text\{Sel\-AUC\}for five models×\\timeseight benchmarks\. Teal = hybrid wins; coral = Self\-ConsU dominates\.
##### Correlation Between Consistency Signal and Reasoning Accuracy\.

Figure[5](https://arxiv.org/html/2606.17312#S4.F5)shows across\-trial and within\-trial components exhibit opposite correlations with accuracy—most pronounced on mathematical reasoning\. On Math\-Synth and MATH\-500, across\-trial instability is negatively correlated with correctness while within\-trial ambiguity is positively correlated \(e\.g\., Claude on MATH\-500:ρacross=−0\.37\\rho\_\{\\text\{across\}\}=\-0\.37vs\.ρwithin=0\.42\\rho\_\{\\text\{within\}\}=0\.42\)\. This asymmetry has a natural interpretation for logical reasoning: ranking instability signals unreliable reasoning, while distributed preference among candidates is consistent with settings where multiple plausible solution paths remain competitive\. The pattern weakens on MMLU\-Pro and collapses on HotpotQA \(near\-zero correlations\), confirming that the consistency signal is regime\-sensitive—informative for logical reasoning but uninformative where reasoning\-level structural diversity is absent\.

![Refer to caption](https://arxiv.org/html/2606.17312v1/images/correlation_heatmap.png)Figure 5:Spearman correlation \(ρ\\rho\) between the two uncertainty components and accuracy\.Each cell reports Spearman rank correlation between per\-question uncertainty and per\-question accuracy \(fraction correct amongN=5N\{=\}5samples\)\. Blue indicates negative correlation; red indicates positive correlation\. Stars denote statistical significance \(p∗<0\.05\{\}^\{\*\}p<0\.05,p∗∗<0\.01\{\}^\{\*\*\}p<0\.01,p∗⁣∗∗<0\.001\{\}^\{\*\*\*\}p<0\.001\)\.
##### Regime Analysis: Reasoning vs\. Retrieval\.

Table[1](https://arxiv.org/html/2606.17312#S4.T1)reveals a clear regime boundary: StructU\+Self\-ConsU consistently outperforms baselines on logical reasoning, whereas on HotpotQA, dispersion methods dominate for the strongest models \(e\.g\., Claude: Self\-ConsU 0\.839 vs\. Hybrid 0\.742; DeepSeek: SemanticU 0\.852 vs\. Hybrid 0\.796\)\. Understanding*when*consistency signals fail is as important as knowing when they succeed, as this clarifies where logical reasoning structure is present versus absent\. Figure[6](https://arxiv.org/html/2606.17312#S4.F6)compares Claude 4\.5 Sonnet on Math\-Synth and HotpotQA under identical experimental conditions \(N=5N\{=\}5,M=5M\{=\}5, similar prompt templates\)\.

Across\-trial separation on reasoning, collapse on retrieval\.On Math\-Synth, across\-trial uncertainty distributions are well\-separated between correct and incorrect questions \(Figure[6](https://arxiv.org/html/2606.17312#S4.F6)a\): incorrect questions exhibit a long right tail beyond 0\.10, while correct questions concentrate near zero\. On HotpotQA, both distributions are compressed near zero—structural uncertainty produces no across\-trial signal regardless of correctness\. In the joint space \(Figure[6](https://arxiv.org/html/2606.17312#S4.F6)c\), Math\-Synth shows correctness separation along the across\-trial axis, while HotpotQA collapses into a degenerate cluster with no separation\.

Near\-uniform preference distributions on factual retrieval\.The within\-trial distributions \(Figure[6](https://arxiv.org/html/2606.17312#S4.F6)b\) reveal the mechanism: HotpotQA responses cluster at maximum entropy \(log⁡5≈1\.61\\log 5\\approx 1\.61, dotted line\) for both correct and incorrect questions, indicating near\-uniform PageRank distributions\. The model assigns approximately equal preference to all five responses, producing pairwise confidences near 50% and identical rankings across spanning tree samples\.

When does self\-preference carry signal?Logical and mathematical reasoning elicits*structurally diverse*solution paths—step\-by\-step computation, bracket decomposition, estimation\-then\-verification—enabling meaningful self\-preference discrimination and making the consistency signal informative\. In our setting, factual retrieval often elicits more*structurally homogeneous*reasoning chains across prompt templates, so self\-preference reflects only stylistic variation and the structural signal vanishes\. The “HotpotQA signature” in Figure[6](https://arxiv.org/html/2606.17312#S4.F6)c serves as a practical diagnostic: it identifies the regime boundary where logical reasoning structure is absent and dispersion\-based methods should be preferred\. This collapse is not merely a failure case; it is a substantively useful boundary result that distinguishes tasks genuinely supporting reasoning\-consistency analysis from those dominated by retrieval\-induced homogeneity\.

ModelStructUwithin\{\}\_\{\\text\{within\}\}\(Within\-trial\) AUROCStructUacross\{\}\_\{\\text\{across\}\}\(Across\-trial\) AUROCRealRandom↓\\downarrowDropRealRandom↓\\downarrowDropClaude 4\.5 Sonnet0\.9840\.488±\\pm0\.0160\.4960\.9230\.510±\\pm0\.0190\.413DeepSeek R10\.8960\.743±\\pm0\.0050\.1530\.8400\.747±\\pm0\.0010\.093GPT\-OSS 20B0\.7560\.557±\\pm0\.0480\.1990\.6680\.588±\\pm0\.0540\.080Amazon Nova Premier0\.9430\.530±\\pm0\.0050\.4130\.8660\.482±\\pm0\.0140\.384Qwen 3 32B0\.8500\.511±\\pm0\.0210\.3390\.7320\.512±\\pm0\.0200\.220Mean0\.8860\.566±\\pm0\.0220\.3200\.8060\.568±\\pm0\.0220\.238

Table 2:Real vs\. randomized preferences on Math\-Synth \(BT\+PageRank\)\.Randomization tests whether gains depend on elicited self\-preference content rather than fixed aggregation structure: winner direction and confidence scores are randomized while all other pipeline components are held fixed\. Random AUROC is mean±\\pmstd over three runs;↓\\downarrowindicates drop from real to random\.
##### Ablation Studies\.

To test whether structural uncertainty reflects elicited preference signal rather than pipeline artifacts, we replace real self\-preference judgments with random comparisons on Math\-Synth: winner direction is randomized uniformly and confidence scores are sampled uniformly from\[51,99\]\[51,99\], while all other pipeline components \(spanning tree topology, BT fitting withC=1C\{=\}1, PageRank aggregation, entropy computation\) are held fixed\. Table[2](https://arxiv.org/html/2606.17312#S4.T2)shows AUROC drops substantially for both uncertainty components \(mean drop: 0\.320 for within\-trial, 0\.238 for across\-trial\), with three models collapsing to near\-chance level \(Claude: 0\.984 → 0\.488; Nova: 0\.943 → 0\.530; Qwen: 0\.850 → 0\.511\)\. These ablations suggest that discriminative performance depends materially on the elicited self\-preference judgments rather than on the aggregation structure alone\. Performance plateaus atM≈5M\{\\approx\}5trials and remains stable throughM=20M\{=\}20\(Figure[8](https://arxiv.org/html/2606.17312#S4.F8)\)\. TheC=1C\{=\}1regularization choice is validated as performance degrades forC<1C<1but remains stable forC≥1C\\geq 1across all models\. PageRank smoothing provides mild regularization benefit \(\+0\.015\+0\.015Sel\-AUC average\)\. Increasing response count fromN=5N\{=\}5toN=10N\{=\}10degrades performance, indicating high\-temperature samples introduce noise \(Figure[7](https://arxiv.org/html/2606.17312#S4.F7)\)\.

![Refer to caption](https://arxiv.org/html/2606.17312v1/images/figure_StructU_within_StructU_accross_final.png)Figure 6:Regime analysis: reasoning consistency vs\. retrieval collapse\.Claude 4\.5 Sonnet on Math\-Synth \(reasoning\) and HotpotQA \(retrieval\), conditioned on correctness \(BT \+ PageRank\)\.\(a\)StructUacross: Math\-Synth shows correctness separation; HotpotQA concentrates near zero for both\.\(b\)StructUwithin: HotpotQA clusters at maximum entropy \(log⁡5≈1\.61\\log 5\\approx 1\.61, dotted line\)\.\(c\)Joint space: reasoning tasks separate along the across\-trial axis; retrieval tasks collapse into a degenerate cluster, diagnosing the regime boundary where logical reasoning structure is absent\.![Refer to caption](https://arxiv.org/html/2606.17312v1/images/aime_ablation_responses.png)Figure 7:Ablation study on AIME benchmarks: effect of number of sampled responses showing performance degrades withN=10N\{=\}10\.![Refer to caption](https://arxiv.org/html/2606.17312v1/images/abalation_iterations.png)Figure 8:Ablation study on AIME benchmarks: effect of number of trials showing performance plateaus atM≈5M\{\\approx\}5\.

## 5Conclusion

We introduced structural uncertainty, a consistency\-aware evaluation framework for logical reasoning that measures the stability of self\-preference\-induced rankings over sampled LLM reasoning candidates\. By eliciting pairwise self\-preferences and aggregating them via Bradley–Terry with PageRank, we decompose the signal into across\-trial ranking instability and within\-trial candidate ambiguity without requiring model internals\.

Across five LLMs and eight benchmarks, the central finding is that structural uncertainty provides a consistency\-aware lens on logical reasoning: combining structural and dispersion\-based signals improves identification of unreliable reasoning instances on several reasoning and knowledge tasks, with largest gains where systematic errors produce consistent but incorrect answers—a failure mode invisible to answer dispersion alone\. The two components relate differently to accuracy—across\-trial instability signals unreliable reasoning, while within\-trial ambiguity correlates positively on mathematical reasoning, consistent with settings where multiple plausible solution paths remain competitive\. Conversely, on factual retrieval \(HotpotQA\), the structural signal collapses where reasoning\-level structural diversity is absent; this collapse itself clarifies the regime boundary where logical reasoning structure is not present\.

Structural uncertainty is best understood not as a universal confidence estimator, but as a regime\-sensitive evaluator of logical reasoning consistency—reframing the question from*how much do responses differ*to*how consistently does the model rank competing reasoning solutions*\. These two complementary views of the same response set help practitioners assess reasoning reliability and identify when consistency\-based evaluation is informative versus when answer dispersion should be preferred\. More broadly, our results suggest that benchmarking logical reasoning should account not only for answer agreement, but also for the structural stability of model\-internal preferences over competing solution paths\.

## Limitations

Our approach requiresNNgenerations andM​\(N−1\)M\(N\{\-\}1\)pairwise comparisons per question \(N=5N\{=\}5,M=5M\{=\}5\), increasing inference cost\. Since models judge their own outputs, self\-preferences can inherit model\-specific biases and should be interpreted as behavioral signals rather than guaranteed correctness measures\. The decomposition is empirical: the two components are observable signals under a fixed protocol, not universally identifiable uncertainty sources—candidate diversity, prompt design, and task structure all affect the signal\. The method is most informative when responses differ in reasoning quality; when variation is stylistic, the preference graph collapses \(as on HotpotQA\)\. Accordingly, our method should be interpreted as an evaluator of reasoning consistency under a fixed elicitation protocol, rather than as a complete measure of logical validity\. Our evaluation focuses on short\-answer tasks; extending to long\-form or open\-ended generation remains future work\.

## References

- To believe or not to believe your llm: iterative prompting for estimating epistemic uncertainty\.Advances in Neural Information Processing Systems37,pp\. 58077–58117\.Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px2.p1.1)\.
- R\. A\. Bradley and M\. E\. Terry \(1952\)Rank analysis of incomplete block designs: I\. the method of paired comparisons\.Biometrika39\(3\-4\),pp\. 324–345\.External Links:[Document](https://dx.doi.org/10.1093/biomet/39.3-4.324)Cited by:[§1](https://arxiv.org/html/2606.17312#S1.p3.1),[§3\.2](https://arxiv.org/html/2606.17312#S3.SS2.p1.2)\.
- S\. Brin and L\. Page \(1998\)The anatomy of a large\-scale hypertextual Web search engine\.InComputer Networks and ISDN Systems,Vol\.30,pp\. 107–117\.External Links:[Document](https://dx.doi.org/10.1016/S0169-7552%2898%2900110-X)Cited by:[§3\.2](https://arxiv.org/html/2606.17312#S3.SS2.p3.5)\.
- T\. Chen, X\. Liu, L\. Da, J\. Chen, V\. Papalexakis, and H\. Wei \(2025\)Uncertainty quantification of large language models through multi\-dimensional responses\.arXiv preprint arXiv:2502\.16820\.Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Farquhar, J\. Kossen, L\. Kuhn, and Y\. Gal \(2024\)Detecting hallucinations in large language models using semantic entropy\.Nature630\(8017\),pp\. 625–630\.Cited by:[§1](https://arxiv.org/html/2606.17312#S1.p2.1),[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px2.p1.1)\.
- L\. R\. Ford \(1957\)Solution of a ranking problem from binary comparisons\.The American Mathematical Monthly64\(8\),pp\. 28–33\.Cited by:[Figure 9](https://arxiv.org/html/2606.17312#A1.F9),[§A\.2](https://arxiv.org/html/2606.17312#A1.SS2.SSS0.Px2.p1.4),[§3\.2](https://arxiv.org/html/2606.17312#S3.SS2.p1.3)\.
- Z\. He \(2023\)AMC23: american mathematics competitions dataset\.Note:HuggingFace DatasetExternal Links:[Link](https://huggingface.co/datasets/zwhe99/amc23)Cited by:[§4\.1](https://arxiv.org/html/2606.17312#S4.SS1.p1.1)\.
- R\. Herbrich, T\. Minka, and T\. Graepel \(2006\)TrueSkill™: a bayesian skill rating system\.Advances in neural information processing systems19\.Cited by:[§A\.4](https://arxiv.org/html/2606.17312#A1.SS4.p1.2),[§3\.2](https://arxiv.org/html/2606.17312#S3.SS2.p2.1)\.
- X\. Huang, S\. Li, M\. Yu, M\. Sesia, H\. Hassani, I\. Lee, O\. Bastani, and E\. Dobriban \(2024\)Uncertainty in language models: assessment through rank\-calibration\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 2851–2873\.Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Huang, J\. Song, Z\. Wang, S\. Zhao, H\. Chen, F\. Juefei\-Xu, and L\. Ma \(2023\)Look before you leap: an exploratory study of uncertainty measurement for large language models\.arXiv preprint arXiv:2307\.10236\.Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px3.p1.1)\.
- HuggingFaceH4 \(2024\)AIME 2024: american invitational mathematics examination dataset\.Note:HuggingFace DatasetExternal Links:[Link](https://huggingface.co/datasets/HuggingFaceH4/aime_2024)Cited by:[§4\.1](https://arxiv.org/html/2606.17312#S4.SS1.p1.1)\.
- D\. R\. Hunter \(2004\)MM algorithms for generalized Bradley–Terry models\.The Annals of Statistics32\(1\),pp\. 384–406\.External Links:[Document](https://dx.doi.org/10.1214/aos/1079120141)Cited by:[§A\.2](https://arxiv.org/html/2606.17312#A1.SS2.SSS0.Px3.p1.4),[§3\.2](https://arxiv.org/html/2606.17312#S3.SS2.p1.8),[3](https://arxiv.org/html/2606.17312#alg1.l3)\.
- S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DaSilva, E\. Elhage,et al\.\(2022\)Language models \(mostly\) know what they know\.arXiv preprint arXiv:2207\.05221\.Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Kapoor, N\. Gruver, M\. Roberts, A\. Pal, S\. Dooley, M\. Goldblum, and A\. Wilson \(2024\)Calibration\-tuning: teaching large language models to know what they don’t know\.InProceedings of the 1st Workshop on Uncertainty\-Aware NLP \(UncertaiNLP 2024\),pp\. 1–14\.Cited by:[§1](https://arxiv.org/html/2606.17312#S1.p1.1)\.
- A\. Kendall and Y\. Gal \(2017\)What uncertainties do we need in Bayesian deep learning for computer vision?\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/1703.04977)Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2606.17312#S3.SS3.p1.4)\.
- J\. Kossen, J\. Han, M\. Razzak, L\. Schut, S\. Malik, and Y\. Gal \(2024\)Semantic entropy probes: robust and cheap hallucination detection in llms\.arXiv preprint arXiv:2406\.15927\.Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2606.17312#S4.SS1.p2.1)\.
- L\. Kuhn, Y\. Gal, and S\. Farquhar \(2023\)Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation\.InProceedings of the Eleventh International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px2.p1.1)\.
- A\. N\. Langville and C\. D\. Meyer \(2006\)Google’s PageRank and beyond: the science of search engine rankings\.Princeton University Press\.External Links:ISBN 0691122024Cited by:[§1](https://arxiv.org/html/2606.17312#S1.p3.1),[§3\.2](https://arxiv.org/html/2606.17312#S3.SS2.p3.5)\.
- J\. Lee, J\. O\. Woo, J\. Seok, P\. Hassanzadeh, W\. Jang, J\. Son, S\. Didari, B\. Gutow, H\. Hao, H\. Moon, W\. Hu, Y\. Kwon, T\. Lee, and S\. Min \(2024\)Improving instruction following in language models through proxy\-based uncertainty estimation\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 27009–27036\.Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px2.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2024\)Let’s verify step by step\.InProceedings of the Twelfth International Conference on Learning Representations,Cited by:[§4\.1](https://arxiv.org/html/2606.17312#S4.SS1.p1.1)\.
- S\. Lin, J\. Hilton, and O\. Evans \(2022\)TruthfulQA: measuring how models mimic human falsehoods\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3214–3252\.Cited by:[§4\.1](https://arxiv.org/html/2606.17312#S4.SS1.p1.1)\.
- Z\. Lin, S\. Trivedi, and J\. Sun \(2023\)Generating with confidence: uncertainty quantification for black\-box large language models\.arXiv preprint arXiv:2305\.19187\.Cited by:[§1](https://arxiv.org/html/2606.17312#S1.p2.1),[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px2.p1.1)\.
- Q\. Lyu, K\. Shridhar, C\. Malaviya, L\. Zhang, Y\. Elazar, N\. Tandon, M\. Apidianaki, M\. Sachan, and C\. Callison\-Burch \(2025\)Calibrating large language models with sample consistency\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 19260–19268\.Cited by:[§1](https://arxiv.org/html/2606.17312#S1.p2.1),[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2606.17312#S4.SS1.p2.1)\.
- H\. Ma, J\. Chen, J\. T\. Zhou, G\. Wang, and C\. Zhang \(2025\)Estimating llm uncertainty with evidence\.arXiv preprint arXiv:2502\.00290\.Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Nikitin, J\. Kossen, Y\. Gal, and P\. Marttinen \(2024\)Kernel language entropy: fine\-grained uncertainty quantification for llms from semantic similarities\.Advances in Neural Information Processing Systems37,pp\. 8901–8929\.Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Qiu and R\. Miikkulainen \(2024\)Semantic density: uncertainty quantification for large language models through confidence measurement in semantic space\.Advances in neural information processing systems37,pp\. 134507–134533\.Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2606.17312#S4.SS1.p2.1)\.
- V\. Shrivastava, A\. Kumar, and P\. Liang \(2025\)Language models prefer what they know: relative confidence estimation via confidence preferences\.arXiv preprint arXiv:2502\.01126\.Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px2.p1.1)\.
- V\. Shrivastava, P\. Liang, and A\. Kumar \(2023\)Llamas know what gpts don’t show: surrogate models for confidence estimation\.arXiv preprint arXiv:2311\.08877\.Cited by:[§4\.1](https://arxiv.org/html/2606.17312#S4.SS1.p3.2)\.
- K\. Tian, E\. Mitchell, A\. Zhou, A\. Sharma, R\. Rafailov, H\. Yao, C\. Finn, and C\. D\. Manning \(2023\)Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 5433–5442\.Cited by:[§1](https://arxiv.org/html/2606.17312#S1.p1.1),[§4\.1](https://arxiv.org/html/2606.17312#S4.SS1.p2.1)\.
- TIGER\-Lab \(2024\)AIME25: a benchmark for mathematical reasoning\.Note:HuggingFace DatasetExternal Links:[Link](https://huggingface.co/datasets/TIGER-Lab/AIME25)Cited by:[§4\.1](https://arxiv.org/html/2606.17312#S4.SS1.p1.1)\.
- A\. Vashurin, M\. Vikhreva, T\. Kocmi, and A\. Malinin \(2025a\)CoCoA: a minimum Bayes risk framework bridging confidence and consistency for uncertainty quantification in large language models\.InAdvances in Neural Information Processing Systems,Vol\.38\.Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px2.p1.1)\.
- R\. Vashurin, E\. Fadeeva, A\. Vazhentsev, L\. Rvanova, D\. Vasilev, A\. Tsvigun, S\. Petrakov, R\. Xing, A\. Sadallah, K\. Grishchenkov,et al\.\(2025b\)Benchmarking uncertainty quantification methods for large language models with lm\-polygraph\.Transactions of the Association for Computational Linguistics13,pp\. 220–248\.Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Wang, L\. Song, Y\. Tian, B\. Peng, L\. Jin, H\. Mi, J\. Su, and D\. Yu \(2024a\)Self\-consistency boosts calibration for math reasoning\.arXiv preprint arXiv:2403\.09849\.Cited by:[§A\.7](https://arxiv.org/html/2606.17312#A1.SS7.p1.2),[§1](https://arxiv.org/html/2606.17312#S1.p2.1),[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2606.17312#S4.SS1.p2.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023\)Self\-consistency improves chain of thought reasoning in language models\.InProceedings of the Eleventh International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang,et al\.\(2024b\)MMLU\-Pro: a more robust and challenging multi\-task language understanding benchmark\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§4\.1](https://arxiv.org/html/2606.17312#S4.SS1.p1.1)\.
- D\. B\. Wilson \(1996\)Generating random spanning trees more quickly than the cover time\.InProceedings of the Twenty\-Eighth Annual ACM Symposium on Theory of Computing,STOC ’96,New York, NY, USA,pp\. 296–303\.External Links:[Document](https://dx.doi.org/10.1145/237814.237880)Cited by:[§A\.1](https://arxiv.org/html/2606.17312#A1.SS1.p2.1)\.
- J\. O\. Woo \(2022\)Analytic mutual information in bayesian neural networks\.In2022 IEEE International Symposium on Information Theory \(ISIT\),pp\. 300–305\.Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2606.17312#S3.SS3.p1.4)\.
- J\. O\. Woo \(2023\)Active learning in bayesian neural networks with balanced entropy learning principle\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ZTMuZ68B1g)Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Xiong, Z\. Hu, X\. Lu, Y\. Li, J\. Fu, J\. He, and B\. Hooi \(2024\)Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs\.InProceedings of the Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.17312#S1.p1.1),[§4\.1](https://arxiv.org/html/2606.17312#S4.SS1.p2.1)\.
- Y\. Yang, H\. Yoo, and H\. Lee \(2025\)Maqa: evaluating uncertainty quantification in llms regarding data uncertainty\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 5846–5863\.Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning \(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InConference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§4\.1](https://arxiv.org/html/2606.17312#S4.SS1.p1.1)\.
- F\. Ye, M\. Yang, J\. Pang, L\. Wang, D\. Wong, E\. Yilmaz, S\. Shi, and Z\. Tu \(2024\)Benchmarking llms via uncertainty quantification\.Advances in Neural Information Processing Systems37,pp\. 15356–15385\.Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Yuan, L\. Tao, H\. Lu, M\. Khushi, I\. Razzak, M\. Dras, J\. Yang, and U\. Naseem \(2025\)KG\-uq: knowledge graph\-based uncertainty quantification for long text in large language models\.InCompanion Proceedings of the ACM on Web Conference 2025,pp\. 2071–2077\.Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px2.p1.1)\.
- B\. Zhang and R\. Zhang \(2025\)CoT\-UQ: improving response\-wise uncertainty quantification in LLMs with chain\-of\-thought\.InFindings of the Association for Computational Linguistics: ACL 2025,Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§2](https://arxiv.org/html/2606.17312#S2.SS0.SSS0.Px1.p1.1)\.
- K\. Zhou, J\. D\. Hwang, X\. Ren, and M\. Sap \(2024\)Relying on the unreliable: the impact of language models’ reluctance to express uncertainty\.arXiv preprint arXiv:2401\.06730\.Cited by:[§1](https://arxiv.org/html/2606.17312#S1.p1.1)\.

## Appendix AAdditional Method Details

### A\.1Random Spanning Tree Comparison Graphs

A complete comparison graph onNNcandidates requires\(N2\)\\binom\{N\}\{2\}judge calls\. We instead sample a connected sparse graph per trial by drawing a uniform random spanning tree𝒯\(m\)\\mathcal\{T\}^\{\(m\)\}on theNNnodes and querying only itsN−1N\{\-\}1edges\. This guarantees connectivity \(needed for global ranking\) while reducing comparisons toO​\(N\)O\(N\)per trial\.

We sample uniform random spanning trees using Wilson’s algorithm\[Wilson,[1996](https://arxiv.org/html/2606.17312#bib.bib48)\]based on loop\-erased random walks\. In dense graphs, Wilson sampling runs in expectedO​\(N\)O\(N\)time and produces an unbiased sample from the uniform distribution over spanning trees\.

### A\.2Bradley–Terry with L2 Regularization

##### Model\.

BT assigns each responseiia latent utilityθi∈ℝ\\theta\_\{i\}\\in\\mathbb\{R\}\. The probability thatiiis preferred overjjis

ℙ​\(i≻j\)=exp⁡\(θi\)exp⁡\(θi\)\+exp⁡\(θj\)\.\\mathbb\{P\}\(i\\succ j\)=\\frac\{\\exp\(\\theta\_\{i\}\)\}\{\\exp\(\\theta\_\{i\}\)\+\\exp\(\\theta\_\{j\}\)\}\.\(9\)

##### Regularization\.

Spanning trees are cycle\-free and admit perfect total orderings, making the unregularized BT log\-likelihood unbounded\[Ford,[1957](https://arxiv.org/html/2606.17312#bib.bib50)\]\. We add an L2 penalty:

ℒreg​\(𝜽\)=∑\(i,j\)∈𝒯\(m\)log⁡ℙ​\(i≻j\)−12​C​‖𝜽‖2,\\mathcal\{L\}\_\{\\text\{reg\}\}\(\\boldsymbol\{\\theta\}\)=\\sum\_\{\(i,j\)\\in\\mathcal\{T\}^\{\(m\)\}\}\\log\\mathbb\{P\}\(i\\succ j\)\-\\frac\{1\}\{2C\}\\\|\\boldsymbol\{\\theta\}\\\|^\{2\},\(10\)whereC\>0C\>0is inverse regularization strength \(largerCC= weaker penalty\)\. The L2 term ensures strict concavity and a unique finite maximizer\. We setC=1C\{=\}1; see Appendix[A\.3](https://arxiv.org/html/2606.17312#A1.SS3)\.

##### Estimation\.

We maximizeℒreg\\mathcal\{L\}\_\{\\text\{reg\}\}via MM\-style updates\[Hunter,[2004](https://arxiv.org/html/2606.17312#bib.bib35)\]with L2 gradient correction−1C​𝜽\-\\frac\{1\}\{C\}\\boldsymbol\{\\theta\}\. Convergence criterion:‖𝜽t\+1−𝜽t‖∞<10−6\\\|\\boldsymbol\{\\theta\}\_\{t\+1\}\-\\boldsymbol\{\\theta\}\_\{t\}\\\|\_\{\\infty\}<10^\{\-6\}\. After convergence, re\-center utilities:θi←θi−1N​∑kθk\\theta\_\{i\}\\leftarrow\\theta\_\{i\}\-\\frac\{1\}\{N\}\\sum\_\{k\}\\theta\_\{k\}\.

##### Pairwise probabilities\.

After fitting, set trial\-specific win probabilities asPi​j\(m\)=ℙ​\(i≻j;𝜽^\)P^\{\(m\)\}\_\{ij\}=\\mathbb\{P\}\(i\\succ j;\\hat\{\\boldsymbol\{\\theta\}\}\)via Eq\. \([9](https://arxiv.org/html/2606.17312#A1.E9)\) for all pairs\(i,j\)\(i,j\), withPi​i\(m\)=0P^\{\(m\)\}\_\{ii\}=0\.

### A\.3Sensitivity Analysis: L2 Regularization StrengthCC

Figure[9](https://arxiv.org/html/2606.17312#A1.F9)reports Sel\-AUC sensitivity to inverse regularization strengthC∈\{0\.1,0\.5,1,3,5,10\}C\\in\\\{0\.1,0\.5,1,3,5,10\\\}on Math\-Synth\. Performance degrades at lowCC\(over\-regularization suppresses preference signal\) and remains stable in the plateau regionC∈\[1,5\]C\\in\[1,5\]\. We setC=1C\{=\}1throughout all experiments, corresponding to the onset of the stable plateau across all models and uncertainty components\.

![Refer to caption](https://arxiv.org/html/2606.17312v1/images/combined_epistemic_all_models.png)\(a\)StructUacross\{\}\_\{\\text\{across\}\}across\-trial
![Refer to caption](https://arxiv.org/html/2606.17312v1/images/combined_aleatoric_all_models.png)\(b\)StructUwithin\{\}\_\{\\text\{within\}\}\(within\-trial\)
![Refer to caption](https://arxiv.org/html/2606.17312v1/images/combined_total_all_models.png)\(c\)StructUtotal\{\}\_\{\\text\{total\}\}\(Total\)

Figure 9:Sensitivity of Sel\-AUC to inverse regularization strengthCC\(BT\+PageRank, Math\-Synth\)\.Each panel shows one uncertainty component across all five models as a function ofC∈\{0\.1,0\.5,1,3,5,10\}C\\in\\\{0\.1,0\.5,1,3,5,10\\\}, where largerCCcorresponds to weaker regularization\.\(a\) Across\-trial\(StructUacross\{\}\_\{\\text\{across\}\}\): models with stronger preference signal \(DeepSeek R1, GPT\-OSS 20B, Amazon Nova Premier\) show the largest degradation at lowCC, where over\-regularization suppresses trial\-to\-trial ranking variance\.\(b\) Within\-trial\(StructUwithin\{\}\_\{\\text\{within\}\}\): more stable acrossCCbut still degrades atC=0\.1C\{=\}0\.1for stronger models, as compressed utilities produce artificially uniform within\-trial PageRank distributions\.\(c\) Total\(StructUtotal\{\}\_\{\\text\{total\}\}\): reflects the combined effect\. Across all three panels and all five models, performance plateaus stably forC≥1C\\geq 1with no degradation observed atC=10C\{=\}10\(weakest regularization tested\), directly confirming that BT parameters remain bounded and do not exhibit the divergence predicted for unregularized spanning tree MLE\[Ford,[1957](https://arxiv.org/html/2606.17312#bib.bib50)\]\. We fixC=1C\{=\}1throughout all experiments as a conservative choice at the boundary of the stable regime\.
### A\.4Confidence\-Weighted TrueSkill

TrueSkill\[Herbrichet al\.,[2006](https://arxiv.org/html/2606.17312#bib.bib49)\]represents each responseiiwith Gaussian ratingri=\(μi,σi\)r\_\{i\}=\(\\mu\_\{i\},\\sigma\_\{i\}\)and updates ratings sequentially from pairwise outcomes\. We extend with confidence\-weighted fractional updates\.

##### Inputs and filtering\.

Judge returns matches\(w,ℓ,cw​ℓ\)\(w,\\ell,c\_\{w\\ell\}\)with confidencecw​ℓ∈\[0,100\]c\_\{w\\ell\}\\in\[0,100\]\. Convert to probabilitypw​ℓ=cw​ℓ/100p\_\{w\\ell\}=c\_\{w\\ell\}/100, retain onlypw​ℓ\>0\.5p\_\{w\\ell\}\>0\.5, discardpw​ℓ=0\.5p\_\{w\\ell\}=0\.5\.

##### Confidence weighting\.

Fractional evidence weight:d=2​\(pw​ℓ−0\.5\)d=2\(p\_\{w\\ell\}\-0\.5\),w=max⁡\(dγ,ϵ\)w=\\max\(d^\{\\gamma\},\\epsilon\)whereγ\\gammacontrols curvature,ϵ=10−6\\epsilon=10^\{\-6\}\.

##### Natural\-parameter blending\.

Forr=\(μ,σ\)r=\(\\mu,\\sigma\), define natural parameters:

λ=1σ2,η=μσ2\.\\lambda=\\tfrac\{1\}\{\\sigma^\{2\}\},\\quad\\eta=\\tfrac\{\\mu\}\{\\sigma^\{2\}\}\.\(11\)Compute full posterior\(rwfull,rℓfull\)=rate\_1vs1​\(rw,rℓ\)\(r\_\{w\}^\{\\text\{full\}\},r\_\{\\ell\}^\{\\text\{full\}\}\)=\\texttt\{rate\\\_1vs1\}\(r\_\{w\},r\_\{\\ell\}\), then blend:

λnew=λ\+w​\(λfull−λ\),ηnew=η\+w​\(ηfull−η\),\\lambda^\{\\text\{new\}\}=\\lambda\+w\(\\lambda^\{\\text\{full\}\}\-\\lambda\),\\quad\\eta^\{\\text\{new\}\}=\\eta\+w\(\\eta^\{\\text\{full\}\}\-\\eta\),\(12\)applied to winner and loser\. Convert back:σ2=1/λnew\\sigma^\{2\}=1/\\lambda^\{\\text\{new\}\},μ=ηnew/λnew\\mu=\\eta^\{\\text\{new\}\}/\\lambda^\{\\text\{new\}\}\. Perform sequential updates for multiple epochs with randomized order\.

##### Win probabilities and strengths\.

Given final ratings and environment parameterβ\\beta:

Pi​j=Φ​\(μi−μj2​β2\+σi2\+σj2\),P\_\{ij\}=\\Phi\\\!\\left\(\\tfrac\{\\mu\_\{i\}\-\\mu\_\{j\}\}\{\\sqrt\{2\\beta^\{2\}\+\\sigma\_\{i\}^\{2\}\+\\sigma\_\{j\}^\{2\}\}\}\\right\),\(13\)whereΦ\\Phiis the standard normal CDF\. Export strengths:si=exp⁡\(μi−μ¯β\)s\_\{i\}=\\exp\\\!\\bigl\(\\tfrac\{\\mu\_\{i\}\-\\bar\{\\mu\}\}\{\\beta\}\\bigr\)whereμ¯=1N​∑kμk\\bar\{\\mu\}=\\tfrac\{1\}\{N\}\\sum\_\{k\}\\mu\_\{k\}\.

### A\.5PageRank Aggregation Details

Given𝐏\(m\)\\mathbf\{P\}^\{\(m\)\}, construct row\-stochastic transition matrix𝐓\(m\)\\mathbf\{T\}^\{\(m\)\}moving fromiitojjproportional to the probabilityjjbeatsii:

Ti​j\(m\)=Pj​i\(m\)∑k≠iPk​i\(m\),Ti​i\(m\)=0\.T^\{\(m\)\}\_\{ij\}=\\tfrac\{P^\{\(m\)\}\_\{ji\}\}\{\\sum\_\{k\\neq i\}P^\{\(m\)\}\_\{ki\}\},\\qquad T^\{\(m\)\}\_\{ii\}=0\.\(14\)Compute𝝅\(m\)\\boldsymbol\{\\pi\}^\{\(m\)\}by power iteration with damping factord=0\.85d=0\.85and teleportation vector𝒗=1N​𝟏\\boldsymbol\{v\}=\\frac\{1\}\{N\}\\mathbf\{1\}:

𝝅\(m\)=d​\(𝐓\(m\)\)⊤​𝝅\(m\)\+\(1−d\)​𝒗\.\\boldsymbol\{\\pi\}^\{\(m\)\}=d\\,\(\\mathbf\{T\}^\{\(m\)\}\)^\{\\top\}\\boldsymbol\{\\pi\}^\{\(m\)\}\+\(1\-d\)\\boldsymbol\{v\}\.\(15\)Stop when‖𝝅t\+1−𝝅t‖1≤10−6\\\|\\boldsymbol\{\\pi\}\_\{t\+1\}\-\\boldsymbol\{\\pi\}\_\{t\}\\\|\_\{1\}\\leq 10^\{\-6\}and renormalize to sum to 1\.

### A\.6Uncertainty Decomposition Details

RunningMMtrials yields PageRank distributions\{𝝅\(1\),…,𝝅\(M\)\}\\\{\\boldsymbol\{\\pi\}^\{\(1\)\},\\ldots,\\boldsymbol\{\\pi\}^\{\(M\)\}\\\}where each𝝅\(m\)∈ΔN\\boldsymbol\{\\pi\}^\{\(m\)\}\\in\\Delta^\{N\}is a distribution overNNcandidates\. Letω\\omegadenote stochastic trial factors\. The identityH​\(𝝅\)=I​\(ω;𝝅\)\+H​\(𝝅∣ω\)H\(\\boldsymbol\{\\pi\}\)=I\(\\omega;\\boldsymbol\{\\pi\}\)\+H\(\\boldsymbol\{\\pi\}\\mid\\omega\)motivates decomposing into across\-trial \(mutual information\) and within\-trial \(conditional entropy\) components\.

Define mean distribution𝝅¯=1M​∑m𝝅\(m\)\\bar\{\\boldsymbol\{\\pi\}\}=\\tfrac\{1\}\{M\}\\sum\_\{m\}\\boldsymbol\{\\pi\}^\{\(m\)\}\. Total structural uncertainty:

StructU=H​\[𝝅¯\]=−∑i=1Nπ¯i​log⁡π¯i\.\\text\{StructU\}=H\[\\bar\{\\boldsymbol\{\\pi\}\}\]=\-\\sum\_\{i=1\}^\{N\}\\bar\{\\pi\}\_\{i\}\\log\\bar\{\\pi\}\_\{i\}\.\(16\)Within\-trial uncertainty captures candidate ambiguity:

StructUwithin=1M​∑m=1MH​\[𝝅\(m\)\]\.\\text\{StructU\}\_\{\\text\{within\}\}=\\tfrac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}H\[\\boldsymbol\{\\pi\}^\{\(m\)\}\]\.\(17\)
Across\-trial uncertainty measures ranking instability:

StructUacross=StructU−StructUwithin\.\\text\{StructU\}\_\{\\text\{across\}\}=\\text\{StructU\}\-\\text\{StructU\}\_\{\\text\{within\}\}\.\(18\)
For the combined estimator, compute Self\-ConsU from the answer distribution\. The sign convention is fixed globally:StructUacross\\text\{StructU\}\_\{\\text\{across\}\}, which correlates negatively with accuracy, adds with Self\-ConsU;StructUwithin\\text\{StructU\}\_\{\\text\{within\}\}, which correlates positively on reasoning tasks, enters subtractively:

StructU\+Self\-ConsUacross=StructUacross\+Self\-ConsU\\text\{StructU\+Self\-ConsU\}\_\{\\text\{across\}\}=\\text\{StructU\}\_\{\\text\{across\}\}\+\\text\{Self\-ConsU\}\(19\)StructU\+Self\-ConsUwithin=StructUwithin−Self\-ConsU\.\\text\{StructU\+Self\-ConsU\}\_\{\\text\{within\}\}=\\text\{StructU\}\_\{\\text\{within\}\}\-\\text\{Self\-ConsU\}\.\(20\)

### A\.7Self\-Consistency: Linear vs\. Entropy Formulation

The original self\-consistency method\[Wanget al\.,[2024a](https://arxiv.org/html/2606.17312#bib.bib24)\]measures agreement via the majority vote proportion\. For uncertainty quantification, this is typically inverted tousc=1−maxa⁡p​\(a\)u\_\{\\text\{sc\}\}=1\-\\max\_\{a\}p\(a\), where higher values indicate greater disagreement\. We instead use Shannon entropyH​\[p\]=−∑ap​\(a\)​log⁡p​\(a\)H\[p\]=\-\\sum\_\{a\}p\(a\)\\log p\(a\), which:

- •Captures the full distribution shape rather than only the mode,
- •Provides a principled information\-theoretic measure,
- •Enables natural combination with our entropy\-based structural uncertainty\.

Empirically, both formulations correlate strongly with correctness \(Spearmanρ\>0\.95\\rho\>0\.95across datasets\), but entropy slightly outperforms the linear measure when combined with StructU\.

Algorithm 1Bradley–Terry with L2 Regularization \(per trialmm\)0:Edge set

ℰ\(m\)\\mathcal\{E\}^\{\(m\)\}\(spanning tree\); inverse regularization

C\>0C\>0\(default

C=1C\{=\}1\)

0:BT utilities

𝜽^\(m\)\\hat\{\\boldsymbol\{\\theta\}\}^\{\(m\)\}, pairwise probabilities

𝐏\(m\)\\mathbf\{P\}^\{\(m\)\}
1:Initialize

θi←0\\theta\_\{i\}\\leftarrow 0for all

ii
2:repeat

3:Maximize regularized BT log\-likelihood via MM updates\[Hunter,[2004](https://arxiv.org/html/2606.17312#bib.bib35)\]:

4:

ℒreg​\(𝜽\)=∑\(i,j\)∈ℰ\(m\)log⁡exp⁡\(θi\)exp⁡\(θi\)\+exp⁡\(θj\)−12​C​‖𝜽‖2\\mathcal\{L\}\_\{\\text\{reg\}\}\(\\boldsymbol\{\\theta\}\)=\\sum\_\{\(i,j\)\\in\\mathcal\{E\}^\{\(m\)\}\}\\log\\tfrac\{\\exp\(\\theta\_\{i\}\)\}\{\\exp\(\\theta\_\{i\}\)\+\\exp\(\\theta\_\{j\}\)\}\-\\tfrac\{1\}\{2C\}\\\|\\boldsymbol\{\\theta\}\\\|^\{2\}
5:until

‖𝜽t\+1−𝜽t‖∞<10−6\\\|\\boldsymbol\{\\theta\}\_\{t\+1\}\-\\boldsymbol\{\\theta\}\_\{t\}\\\|\_\{\\infty\}<10^\{\-6\}
6:Re\-center:

θi←θi−1N​∑kθk\\theta\_\{i\}\\leftarrow\\theta\_\{i\}\-\\tfrac\{1\}\{N\}\\sum\_\{k\}\\theta\_\{k\}
7:forall pairs

\(i,j\)\(i,j\),

i≠ji\\neq jdo

8:

Pi​j\(m\)←exp⁡\(θi\)exp⁡\(θi\)\+exp⁡\(θj\)P^\{\(m\)\}\_\{ij\}\\leftarrow\\tfrac\{\\exp\(\\theta\_\{i\}\)\}\{\\exp\(\\theta\_\{i\}\)\+\\exp\(\\theta\_\{j\}\)\}
9:endfor

10:

Pi​i\(m\)←0P^\{\(m\)\}\_\{ii\}\\leftarrow 0for all

ii
11:return

𝜽^\(m\),𝐏\(m\)\\hat\{\\boldsymbol\{\\theta\}\}^\{\(m\)\},\\mathbf\{P\}^\{\(m\)\}

Algorithm 2Pairwise Preference Modeling \(Confidence\-Weighted TrueSkill\) \(per trialmm\)0:Matches

ℳ\(m\)=\{\(w,ℓ,cw​ℓ\)\}\\mathcal\{M\}^\{\(m\)\}=\\\{\(w,\\ell,c\_\{w\\ell\}\)\\\}, nodes

\{1,…,N\}\\\{1,\\dots,N\\\}, epochs

EE, curvature

γ\\gamma, TS params

\(β,τ,draw\_prob\)\(\\beta,\\tau,\\texttt\{draw\\\_prob\}\)
0:Ratings

\{\(μi,σi\)\}\\\{\(\\mu\_\{i\},\\sigma\_\{i\}\)\\\}and pairwise probabilities

𝐏\(m\)\\mathbf\{P\}^\{\(m\)\}
1:Initialize ratings

ri←\(μ0,σ0\)r\_\{i\}\\leftarrow\(\\mu\_\{0\},\\sigma\_\{0\}\)
2:Filter: keep

p=c/100\>0\.5p=c/100\>0\.5, drop

p=0\.5p=0\.5
3:for

e=1e=1to

EEdo

4:Shuffle retained matches

5:foreach match

\(w,ℓ,p\)\(w,\\ell,p\)do

6:

d←2​\(p−0\.5\)d\\leftarrow 2\(p\-0\.5\);

w​g​t←max⁡\(dγ,ϵ\)wgt\\leftarrow\\max\(d^\{\\gamma\},\\epsilon\)
7:

\(rwf​u​l​l,rℓf​u​l​l\)←rate\_1vs1​\(rw,rℓ\)\(r\_\{w\}^\{full\},r\_\{\\ell\}^\{full\}\)\\leftarrow\\texttt\{rate\\\_1vs1\}\(r\_\{w\},r\_\{\\ell\}\)
8:Convert

rw,rwf​u​l​lr\_\{w\},r\_\{w\}^\{full\}to naturals

\(λ,η\)\(\\lambda,\\eta\)via

λ=1/σ2\\lambda=1/\\sigma^\{2\},

η=μ/σ2\\eta=\\mu/\\sigma^\{2\}
9:Blend:

\(λ,η\)←\(λ,η\)\+w​g​t​\(\(λf​u​l​l,ηf​u​l​l\)−\(λ,η\)\)\(\\lambda,\\eta\)\\leftarrow\(\\lambda,\\eta\)\+wgt\\big\(\(\\lambda^\{full\},\\eta^\{full\}\)\-\(\\lambda,\\eta\)\\big\)
10:Convert back to

\(μ,σ\)\(\\mu,\\sigma\)via

σ2=1/λ\\sigma^\{2\}=1/\\lambda,

μ=η/λ\\mu=\\eta/\\lambda
11:Apply the same natural\-parameter blending steps to loser

ℓ\\ell
12:endfor

13:endfor

14:forall ordered pairs

\(i,j\)\(i,j\),

i≠ji\\neq jdo

15:

Pi​j\(m\)←Φ​\(μi−μj2​β2\+σi2\+σj2\)P^\{\(m\)\}\_\{ij\}\\leftarrow\\Phi\\\!\\left\(\\frac\{\\mu\_\{i\}\-\\mu\_\{j\}\}\{\\sqrt\{2\\beta^\{2\}\+\\sigma\_\{i\}^\{2\}\+\\sigma\_\{j\}^\{2\}\}\}\\right\)
16:endfor

17:Set

Pi​i\(m\)←0P^\{\(m\)\}\_\{ii\}\\leftarrow 0for all

ii
18:returnratings and

𝐏\(m\)\\mathbf\{P\}^\{\(m\)\}

Algorithm 3PageRank Aggregation \(per trialmm\)0:

𝐏\(m\)∈\[0,1\]N×N\\mathbf\{P\}^\{\(m\)\}\\in\[0,1\]^\{N\\times N\}, damping

d∈\[0,1\)d\\in\[0,1\), tolerance

ε\\varepsilon
0:Stationary distribution

𝝅\(m\)∈ΔN\\boldsymbol\{\\pi\}^\{\(m\)\}\\in\\Delta^\{N\}
1:

Ti​j←Pj​i/∑k≠iPk​iT\_\{ij\}\\leftarrow P\_\{ji\}/\\sum\_\{k\\neq i\}P\_\{ki\}for

i≠ji\\neq j;

Ti​i←0T\_\{ii\}\\leftarrow 0
2:

𝒗←1N​𝟏\\boldsymbol\{v\}\\leftarrow\\tfrac\{1\}\{N\}\\mathbf\{1\};

𝝅←𝒗\\boldsymbol\{\\pi\}\\leftarrow\\boldsymbol\{v\}
3:repeat

4:

𝝅new←d​𝐓⊤​𝝅\+\(1−d\)​𝒗\\boldsymbol\{\\pi\}\_\{\\text\{new\}\}\\leftarrow d\\,\\mathbf\{T\}^\{\\top\}\\boldsymbol\{\\pi\}\+\(1\-d\)\\boldsymbol\{v\}
5:Normalize:

𝝅new←𝝅new/‖𝝅new‖1\\boldsymbol\{\\pi\}\_\{\\text\{new\}\}\\leftarrow\\boldsymbol\{\\pi\}\_\{\\text\{new\}\}/\\\|\\boldsymbol\{\\pi\}\_\{\\text\{new\}\}\\\|\_\{1\}
6:

Δ←‖𝝅new−𝝅‖1\\Delta\\leftarrow\\\|\\boldsymbol\{\\pi\}\_\{\\text\{new\}\}\-\\boldsymbol\{\\pi\}\\\|\_\{1\};

𝝅←𝝅new\\boldsymbol\{\\pi\}\\leftarrow\\boldsymbol\{\\pi\}\_\{\\text\{new\}\}
7:until

Δ≤ε\\Delta\\leq\\varepsilon
8:return

𝝅\(m\)\\boldsymbol\{\\pi\}^\{\(m\)\}

## Appendix BSynthetic Arithmetic Dataset Generation

Math\-Synthis a synthetic arithmetic benchmark isolating computational complexity through systematically varied arithmetic expressions\. Unlike existing benchmarks that conflate conceptual understanding with multi\-step reasoning, we generate expressions whose answer has exactlydddigits \(ford∈\{1,…,14\}d\\in\\\{1,\\ldots,14\\\}\), ensuring answer magnitude serves as a proxy for computational complexity\.

##### Generation and validation pipeline\.

We employ Claude 3\.7 Sonnet \(temperature=0\.9, top\_p=0\.9\) to generate expressions satisfying the digit\-length constraint\. Each problem is stored as JSON with fields:id,question\(arithmetic expression\),answer\(ground\-truth integer\),python\_code\(executable verification\),num\_terms\(operator count \+ 1\), andnum\_digits\(targetdd\)\. The prompt requests 25 problems per digit length with 10 few\-shot examples demonstrating nested negations \(e\.g\.,−\(−\(−\(−\(−\(−\(⋯\)\)\)\)\)\)\-\(\-\(\-\(\-\(\-\(\-\(\\cdots\)\)\)\)\)\)\), complex operator precedence, and exact digit\-length compliance\. Example: “What is−\(−\(−\(−\(−\(−\(−\(−500\)×200\)\)\)\)\)\+−\(−\(−1\)\)\-\(\-\(\-\(\-\(\-\(\-\(\-\(\-500\)\\times 200\)\)\)\)\)\+\-\(\-\(\-1\)\)?” \(answer: 100,001; 6 digits\)\.

Post\-processing ensures correctness: \(i\) parse generated JSON; \(ii\) deduplicate by exact question match; \(iii\) executepython\_code, overwriteanswerif mismatched, drop execution errors; \(iv\) retain only examples satisfyingNumDigits​\(\|y\|\)=d\\texttt\{NumDigits\}\(\|y\|\)=d\. Algorithm[4](https://arxiv.org/html/2606.17312#alg4)formalizes this pipeline\. The final dataset contains 993 verified examples spanningd∈\{1,…,14\}d\\in\\\{1,\\ldots,14\\\}with deterministic ground truth\.

Algorithm 4Generate and Validate Math\-Synth for Digit Lengthdd0:Digit length

dd, batches

BB, optional target size

KK, RNG seed

0:JSONL dataset

𝒟d\\mathcal\{D\}\_\{d\}with exactly\-

dd\-digit answers

1:

Pool←\[\]\\textsc\{Pool\}\\leftarrow\[\\,\]
2:for

b=1b=1to

BBdo

3:

t←LLMGenerate​\(Prompt​\(d\),temp=0\.9\)t\\leftarrow\\textsc\{LLMGenerate\}\(\\textsc\{Prompt\}\(d\),\\text\{temp\}\{=\}0\.9\)
4:

Pool←Pool∪ParseJSON​\(t\)\\textsc\{Pool\}\\leftarrow\\textsc\{Pool\}\\cup\\textsc\{ParseJSON\}\(t\)
5:endfor

6:

Unique←DedupByQuestion​\(Pool\)\\textsc\{Unique\}\\leftarrow\\textsc\{DedupByQuestion\}\(\\textsc\{Pool\}\)
7:

Valid←\[\]\\textsc\{Valid\}\\leftarrow\[\\,\]
8:foreach

e∈Uniquee\\in\\textsc\{Unique\}do

9:

\(ok,y\)←ExecPython\(e\.python\_code\)\(\\textsc\{ok\},y\)\\leftarrow\\textsc\{ExecPython\}\(e\.\\texttt\{python\\\_code\}\)
10:ifokand

NumDigits​\(\|y\|\)=d\\textsc\{NumDigits\}\(\|y\|\)=dthen

11:

e\.answer←str​\(y\)e\.\\texttt\{answer\}\\leftarrow\\texttt\{str\}\(y\); append

eetoValid

12:endif

13:endfor

14:if

KKspecified and

\|Valid\|\>K\|\\textsc\{Valid\}\|\>Kthen

15:

𝒟d←RandomSample​\(Valid,K\)\\mathcal\{D\}\_\{d\}\\leftarrow\\textsc\{RandomSample\}\(\\textsc\{Valid\},K\)
16:else

17:

𝒟d←Valid\\mathcal\{D\}\_\{d\}\\leftarrow\\textsc\{Valid\}
18:endif

19:return

ReindexAndWriteJSONL​\(𝒟d\)\\textsc\{ReindexAndWriteJSONL\}\(\\mathcal\{D\}\_\{d\}\)

## Appendix CExperimental Protocol and Prompt Engineering

### C\.1Overall Evaluation Pipeline

We follow the multi\-path generation andMM\-trial procedure described in Section[3](https://arxiv.org/html/2606.17312#S3)\. For each question, we sampleN=5N\{=\}5candidate responses using the diverse prompt templates in Appendix[E\.6\.1](https://arxiv.org/html/2606.17312#A5.SS6.SSS1), and repeat the complete generation→\\rightarrowcomparison→\\rightarrowranking pipeline forM=5M\{=\}5independent trials\. All models and datasets use identical decoding hyperparameters \(Appendix[C\.2](https://arxiv.org/html/2606.17312#A3.SS2)\) to ensure fair comparison\.

Within each trialmm, we obtain pairwise self\-preference judgments by prompting the same model to compare its own outputs with deterministic evaluation settings \(temperature=0\.0\)\. Each judgment includes a confidence score on a0–100100scale, where100100indicates maximal certainty and5050indicates no preference\. The resulting set of pairwise comparisons is held fixed and reused across all preference\-based uncertainty estimators, ensuring controlled comparisons\. Each trial produces\|ℰ\(m\)\|=N−1=4\|\\mathcal\{E\}^\{\(m\)\}\|=N\{\-\}1=4judged pairs \(spanning tree edges\), yielding4​M=204M\{=\}20total comparisons per question\.

### C\.2Decoding Hyperparameters

Table[3](https://arxiv.org/html/2606.17312#A3.T3)summarizes decoding settings for response generation and self\-preference evaluation\.

ParameterGenerationEvaluationTemperature0\.70\.0Top\-p0\.95–Max tokens40968192Table 3:Decoding hyperparameters\.For response generation, we use stochastic decoding \(temperature=0\.7\) to ensure diversity acrossN=5N\{=\}5candidates\. Each response uses a different prompt template \(Appendix[E\.6\.1](https://arxiv.org/html/2606.17312#A5.SS6.SSS1)\) eliciting distinct reasoning strategies\.

For self\-preference evaluation, we use deterministic decoding \(temperature=0\.0\) to ensure consistent judgments\. The same model that generates responses judges pairwise preferences among its own outputs\. Confidence scores are extracted from structured output and used by TrueSkill \(Appendix[A\.4](https://arxiv.org/html/2606.17312#A1.SS4)\) but not Bradley–Terry\. Fallback confidence is 50 \(neutral\) when preference cannot be determined\.

## Appendix DBaseline Uncertainty Estimators

We compare against representative black\-box uncertainty estimators operating solely on sampled outputs\. All methods use the sameN=5N=5responses per question\. No method accesses token probabilities or hidden states\.

##### Self\-Consistency Uncertainty \(Self\-ConsU\)\.

For inputxx, generateNNresponsesℛ​\(x\)=\{r1,…,rN\}\\mathcal\{R\}\(x\)=\\\{r\_\{1\},\\dots,r\_\{N\}\\\}and extract final answers\{a1,…,aN\}\\\{a\_\{1\},\\dots,a\_\{N\}\\\}\. Let𝒜\\mathcal\{A\}denote unique answers with countn​\(a\)n\(a\)for eacha∈𝒜a\\in\\mathcal\{A\}\. Form empirical distributionp​\(a\)=n​\(a\)/Np\(a\)=n\(a\)/Nand compute Shannon entropy:

Self\-ConsU​\(x\)=−∑a∈𝒜p​\(a\)​log⁡p​\(a\)\.\\text\{Self\-ConsU\}\(x\)=\-\\sum\_\{a\\in\\mathcal\{A\}\}p\(a\)\\log p\(a\)\.\(21\)Self\-ConsU is low when the model repeatedly produces the same answer \(high self\-consistency\) and increases as probability mass spreads across multiple distinct answers\.

##### Semantic Dispersion \(SemanticU\)\.

SampleK=5K=5responses\{r\(1\),…,r\(5\)\}\\\{r^\{\(1\)\},\\dots,r^\{\(5\)\}\\\}containing full solution text\. Embed each using a sentence\-embedding model \(KaLM\), yielding vectors\{e\(1\),…,e\(5\)\}\\\{e^\{\(1\)\},\\dots,e^\{\(5\)\}\\\}\. Compute pairwise cosine distances:

di​j=1−cos⁡\(e\(i\),e\(j\)\),1≤i<j≤5,d\_\{ij\}=1\-\\cos\(e^\{\(i\)\},e^\{\(j\)\}\),\\quad 1\\leq i<j\\leq 5,\(22\)wherecos⁡\(a,b\)=a⊤​b/\(‖a‖​‖b‖\)\\cos\(a,b\)=a^\{\\top\}b/\(\\\|a\\\|\\,\\\|b\\\|\)\. Summarize via mean and variance over the\(52\)=10\\binom\{5\}\{2\}=10distances:

SD\-Mean=110​∑1≤i<j≤5di​j,\\displaystyle=\\tfrac\{1\}\{10\}\\sum\_\{1\\leq i<j\\leq 5\}d\_\{ij\},\(23\)SD\-Var=110​∑1≤i<j≤5\(di​j−SD\-Mean\)2\.\\displaystyle=\\tfrac\{1\}\{10\}\\sum\_\{1\\leq i<j\\leq 5\}\(d\_\{ij\}\-\\text\{SD\-Mean\}\)^\{2\}\.\(24\)Higher SD values indicate greater semantic disagreement and serve as a lightweight uncertainty proxy\.

##### Verbalized Confidence \(VerbalizedU\)\.

For each questionqq, sampleN=5N=5solutions\{s1,…,s5\}\\\{s\_\{1\},\\dots,s\_\{5\}\\\}\. Each candidatesis\_\{i\}is evaluated by a verifier LLMVVusing deterministic decoding \(temperatureT=0T=0\), producing binary verdictvi∈\{PASS,FAIL\}v\_\{i\}\\in\\\{\\text\{PASS\},\\text\{FAIL\}\\\}and confidenceci∈\[0,1\]c\_\{i\}\\in\[0,1\]\. Convert to per\-candidate uncertainty:

uverify\(i\)=1−ci,u\_\{\\text\{verify\}\}^\{\(i\)\}=1\-c\_\{i\},\(25\)and aggregate to question\-level uncertainty:

uverify,mean=1N​∑i=1Nuverify\(i\)\.u\_\{\\text\{verify,mean\}\}=\\tfrac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}u\_\{\\text\{verify\}\}^\{\(i\)\}\.\(26\)This produces a scalar uncertainty estimate per question directly comparable to other estimators\.

### D\.1Full Results: TrueSkill \+ PageRank

Table[5](https://arxiv.org/html/2606.17312#A4.T5)presents complete results using TrueSkill with confidence\-weighted updates and PageRank aggregation\.This serves as critical validation of our structural uncertainty framework: despite fundamentally different modeling assumptions—Bradley–Terry \(Appendix[A\.2](https://arxiv.org/html/2606.17312#A1.SS2)\) estimates deterministic utility differences from observed comparisons, while TrueSkill \(Appendix[A\.4](https://arxiv.org/html/2606.17312#A1.SS4)\) maintains per\-candidate Bayesian variance estimates updated through confidence\-weighted fractional blending—both backends produce highly consistent structural uncertainty estimates and selective prediction rankings\.

##### Cross\-backend validation\.

The Spearman correlation between BT\+PageRank and TS\+PageRank Sel\-AUC scores exceedsρ=0\.95\\rho=0\.95across all 40 model\-dataset pairs \(5 models×\\times8 datasets\)\. Method rank agreement \(which backend’s hybrid variant ranks first\) is 89% for StructU and 91% for StructU\+Self\-ConsU\. Mean absolute difference in Sel\-AUC is 0\.012 for StructU and 0\.015 for StructU\+Self\-ConsU\. This consistency confirms that the across\-trial–within\-trial decomposition is not an artifact of a specific preference model but reflects genuine structural properties of the ranking distribution\.

##### Task\-dependent backend sensitivity\.

While both backends produce consistent overall rankings, they exhibit complementary strengths across task types\. On mathematical reasoning benchmarks \(Math\-Synth, MATH\-500, AMC\-23\), BT\+PageRank better captures within\-trial uncertainty \(within\-trial ambiguity\) through deterministic preference strengths, outperforming TS\+PageRank in 10 of 15 configurations for StructUwithin\{\}\_\{\\text\{within\}\}\. On knowledge\-intensive tasks \(MMLU\-Pro\) and contest benchmarks \(AIME\-24/25\), TS\+PageRank better isolates across\-trial uncertainty \(across\-trial instability\) through variance modeling, achieving superior StructUacross\{\}\_\{\\text\{across\}\}performance in 12 of 15 configurations\. On HotpotQA, both backends exhibit structural collapse \(Table[1](https://arxiv.org/html/2606.17312#S4.T1)\), with neither dominating—consistent with near\-uniform preference graphs rendering backend choice irrelevant when structural signals are degenerate\.

##### Hybrid performance patterns\.

The StructU\+Self\-ConsU hybrids show even stronger backend consistency, with task\-specific exceptions that reveal mechanistic insights\. On MMLU\-Pro, TS\+PageRank hybrids exhibit dramatic advantages \(GPT\-OSS: \+0\.123 Sel\-AUC vs BT\+PageRank; Nova: \+0\.156; Qwen: \+0\.105\), suggesting confidence\-aware variance modeling is particularly valuable when knowledge\-intensive questions admit multiple defensible framings\. On mathematical reasoning, both backends achieve near\-parity \(within 0\.01 Sel\-AUC in 85% of cases\), confirming that deterministic correctness criteria make backend choice less critical\. On HotpotQA, hybrids degrade performance for strongest models \(Claude, DeepSeek\) with both backends, confirming that degenerate preference graphs introduce noise rather than signal regardless of modeling choice\.

Math BenchmarksMMLU\-ProFactualModelSynthMath500AMC23AIME24AIME25OverallChemPhysMathLawEngHotpotTruthfulClaude Sonnet 4\.539\.988\.486\.540\.033\.384\.988\.090\.192\.174\.676\.07398\.5DeepSeek R170\.987\.194\.078\.247\.383\.886\.689\.191\.068\.780\.172\.990\.9GPT\-OSS 20B66\.186\.593\.569\.663\.372\.681\.681\.689\.443\.459\.670\.6484\.4Nova Premier29\.073\.250\.516\.014\.069\.071\.974\.978\.850\.461\.076\.189\.4Qwen 3 32B21\.774\.156\.020\.714\.764\.667\.771\.475\.544\.060\.16675\.3

Table 4:Model accuracies \(percent correct\) across benchmarks\.

### D\.2Full Results: TrueSkill \+ PageRank

Table[5](https://arxiv.org/html/2606.17312#A4.T5)presents complete results using TrueSkill with PageRank aggregation\. The overall performance patterns mirror Bradley–Terry results \(Table[1](https://arxiv.org/html/2606.17312#S4.T1)in main paper\): StructU\+Self\-ConsU achieves highest performance on mathematical reasoning and knowledge tasks, while structural signals collapse on HotpotQA\. Key differences: TrueSkill hybrids show larger gains on MMLU\-Pro \(GPT\-OSS: \+0\.123 vs BT\+PR; Nova: \+0\.156\) due to confidence\-weighted variance modeling, while BT\+PR better captures within\-trial uncertainty on math benchmarks\.

DatasetModelStructU \(TrueSkill\+PageRank\)StructU\+Self\-ConsU \(TrueSkill\+PageRank\)BaselineswithinacrosstotalwithinacrosstotalSelf\-ConsUVerbalizedUSemanticUMath\-SynthClaude 4\.5 Sonnet0\.627 \(0\.978\)0\.641 \(0\.958\)0\.580 \(0\.940\)0\.660 \(0\.992\)0\.661 \(0\.991\)0\.660 \(0\.992\)0\.65 \(0\.985\)0\.544 \(0\.924\)0\.430 \(0\.927\)DeepSeek R10\.816 \(0\.879\)0\.792 \(0\.821\)0\.739 \(0\.709\)0\.823 \(0\.924\)0\.825 \(0\.925\)0\.805 \(0\.909\)0\.802 \(0\.899\)0\.793 \(0\.701\)0\.664 \(0\.502\)GPT\-OSS 20B0\.809 \(0\.750\)0\.772 \(0\.661\)0\.679 \(0\.419\)0\.854 \(0\.961\)0\.849 \(0\.959\)0\.842 \(0\.956\)0\.83 \(0\.958\)0\.792 \(0\.742\)0\.528 \(0\.638\)Amazon Nova Premier0\.417 \(0\.812\)0\.275 \(0\.382\)0\.374 \(0\.682\)0\.501 \(0\.997\)0\.501 \(0\.997\)0\.504 \(0\.998\)0\.382 \(0\.948\)0\.389 \(0\.807\)0\.436 \(0\.824\)Qwen 3 32B0\.254 \(0\.736\)0\.219 \(0\.602\)0\.225 \(0\.614\)0\.407 \(0\.996\)0\.411 \(0\.996\)0\.404 \(0\.995\)0\.38 \(0\.995\)0\.279 \(0\.824\)0\.218 \(0\.462\)MATH\-500Claude 4\.5 Sonnet0\.930 \(0\.800\)0\.932 \(0\.783\)0\.931 \(0\.779\)0\.946 \(0\.832\)0\.950 \(0\.834\)0\.947 \(0\.824\)0\.942 \(0\.816\)0\.891 \(0\.686\)0\.783 \(0\.720\)DeepSeek R10\.893 \(0\.653\)0\.899 \(0\.651\)0\.870 \(0\.591\)0\.939 \(0\.789\)0\.942 \(0\.800\)0\.928 \(0\.771\)0\.923 \(0\.759\)0\.870 \(0\.546\)0\.875 \(0\.603\)GPT\-OSS 20B0\.874 \(0\.602\)0\.888 \(0\.540\)0\.896 \(0\.611\)0\.900 \(0\.725\)0\.904 \(0\.711\)0\.905 \(0\.726\)0\.871 \(0\.694\)0\.886 \(0\.645\)0\.860 \(0\.460\)Amazon Nova Premier0\.785 \(0\.673\)0\.713 \(0\.559\)0\.773 \(0\.587\)0\.886 \(0\.871\)0\.878 \(0\.869\)0\.885 \(0\.865\)0\.860 \(0\.839\)0\.883 \(0\.715\)0\.810 \(0\.695\)Qwen 3 32B0\.827 \(0\.740\)0\.729 \(0\.506\)0\.799 \(0\.691\)0\.886 \(0\.870\)0\.886 \(0\.851\)0\.888 \(0\.867\)0\.871 \(0\.817\)0\.880 \(0\.684\)0\.819 \(0\.717\)AMC\-23Claude 4\.5 Sonnet0\.967 \(0\.997\)0\.969 \(0\.987\)0\.956 \(0\.960\)0\.972 \(1\.000\)0\.972 \(1\.000\)0\.971 \(1\.000\)0\.955 \(1\.00\)0\.900 \(0\.604\)0\.853 \(0\.588\)DeepSeek R10\.947 \(0\.749\)0\.953 \(0\.543\)0\.914 \(0\.229\)0\.985 \(1\.000\)0\.985 \(1\.000\)0\.985 \(1\.000\)0\.985 \(1\.00\)0\.877 \(0\.412\)0\.880 \(0\.592\)GPT\-OSS 20B0\.948 \(0\.737\)0\.929 \(0\.507\)0\.948 \(0\.603\)0\.980 \(1\.000\)0\.980 \(1\.000\)0\.980 \(1\.000\)0\.980 \(1\.00\)0\.980 \(0\.583\)0\.884 \(0\.630\)Amazon Nova Premier0\.526 \(0\.811\)0\.558 \(0\.486\)0\.570 \(0\.757\)0\.715 \(1\.000\)0\.710 \(1\.000\)0\.714 \(1\.000\)0\.584 \(1\.00\)0\.419 \(0\.362\)0\.351 \(0\.410\)Qwen 3 32B0\.610 \(0\.755\)0\.516 \(0\.561\)0\.610 \(0\.727\)0\.818 \(1\.000\)0\.821 \(1\.000\)0\.820 \(1\.000\)0\.810 \(1\.00\)0\.637 \(0\.389\)0\.513 \(0\.299\)AIME\-24Claude 4\.5 Sonnet0\.557 \(0\.932\)0\.620 \(0\.955\)0\.555 \(0\.909\)0\.681 \(1\.000\)0\.699 \(1\.000\)0\.682 \(1\.000\)0\.567 \(0\.972\)0\.273 \(0\.144\)0\.450 \(0\.640\)DeepSeek R10\.857 \(0\.843\)0\.813 \(0\.455\)0\.891 \(0\.909\)0\.925 \(1\.0\)0\.925 \(1\.0\)0\.925 \(1\.0\)0\.917 \(1\.00\)0\.799 \(0\.298\)0\.788 \(0\.715\)GPT\-OSS 20B0\.601 \(0\.742\)0\.757 \(0\.648\)0\.757 \(0\.786\)0\.893 \(1\.000\)0\.904 \(1\.000\)0\.897 \(1\.000\)0\.905 \(1\.000\)0\.891 \(0\.319\)0\.876 \(0\.681\)Amazon Nova Premier0\.233 \(—\)0\.231 \(—\)0\.194 \(—\)0\.196 \(—\)0\.205 \(—\)0\.191 \(—\)0\.174 \(—\)—0\.294 \(—\)Qwen 3 32B0\.306 \(—\)0\.230 \(—\)0\.287 \(—\)0\.423 \(—\)0\.420 \(—\)0\.426 \(—\)0\.414 \(—\)—0\.247 \(—\)AIME\-25Claude 4\.5 Sonnet0\.580 \(0\.988\)0\.599 \(1\.00\)0\.204 \(0\.081\)0\.645 \(1\.00\)0\.643 \(1\.00\)0\.646 \(1\.00\)0\.645 \(1\.00\)0\.489 \(0\.175\)0\.524 \(0\.263\)DeepSeek R10\.360 \(0\.630\)0\.380 \(0\.466\)0\.523 \(0\.397\)0\.735 \(1\.00\)0\.748 \(1\.00\)0\.754 \(1\.00\)0\.707 \(1\.00\)0\.564 \(0\.400\)0\.616 \(0\.296\)GPT\-OSS 20B0\.424 \(0\.546\)0\.407 \(0\.370\)0\.634 \(0\.417\)0\.880 \(1\.00\)0\.886 \(1\.00\)0\.886 \(1\.00\)0\.825 \(1\.00\)0\.776 \(0\.370\)0\.768 \(0\.491\)Amazon Nova Premier0\.083 \(–\)0\.126 \(–\)0\.148 \(–\)0\.188 \(–\)0\.192 \(–\)0\.220 \(–\)0\.178 \(–\)0\.106 \(–\)0\.257 \(–\)Qwen 3 32B0\.086 \(0\.759\)0\.134 \(0\.931\)0\.163 \(0\.517\)0\.260 \(1\.00\)0\.290 \(1\.00\)0\.313 \(1\.00\)0\.208 \(1\.00\)0\.192 \(0\.897\)0\.126 \(0\.931\)MMLU\-ProClaude 4\.5 Sonnet0\.921 \(0\.808\)0\.922 \(0\.797\)0\.911 \(0\.770\)0\.945 \(0\.910\)0\.945 \(0\.909\)0\.944 \(0\.909\)0\.900 \(0\.884\)0\.944 \(0\.885\)0\.890 \(0\.602\)DeepSeek R10\.892 \(0\.671\)0\.862 \(0\.568\)0\.878 \(0\.643\)0\.941 \(0\.913\)0\.933 \(0\.901\)0\.930 \(0\.907\)0\.882 \(0\.882\)0\.927 \(0\.796\)0\.870 \(0\.577\)GPT\-OSS 20B0\.706 \(0\.565\)0\.725 \(0\.500\)0\.726 \(0\.541\)0\.869 \(0\.933\)0\.868 \(0\.931\)0\.868 \(0\.931\)0\.830 \(0\.935\)0\.785 \(0\.631\)0\.774 \(0\.573\)Amazon Nova Premier0\.705 \(0\.463\)0\.699 \(0\.531\)0\.690 \(0\.458\)0\.849 \(0\.957\)0\.823 \(0\.944\)0\.844 \(0\.955\)0\.801 \(0\.945\)0\.820 \(0\.775\)0\.811 \(0\.713\)Qwen 3 32B0\.692 \(0\.640\)0\.638 \(0\.501\)0\.671 \(0\.592\)0\.820 \(0\.972\)0\.808 \(0\.968\)0\.817 \(0\.971\)0\.787 \(0\.966\)0\.728 \(0\.686\)0\.742 \(0\.724\)HotpotQAClaude 4\.5 Sonnet0\.663 \(0\.571\)0\.717 \(0\.619\)0\.667 \(0\.552\)0\.681 \(0\.608\)0\.730 \(0\.653\)0\.686 \(0\.598\)0\.839 \(0\.656\)0\.847 \(0\.700\)0\.768 \(0\.576\)DeepSeek R10\.754 \(0\.635\)0\.730 \(0\.552\)0\.760 \(0\.628\)0\.792 \(0\.727\)0\.780 \(0\.699\)0\.796 \(0\.715\)0\.835 \(0\.658\)0\.829 \(0\.708\)0\.852 \(0\.696\)GPT\-OSS 20B0\.759 \(0\.637\)0\.725 \(0\.554\)0\.748 \(0\.603\)0\.819 \(0\.756\)0\.815 \(0\.739\)0\.811 \(0\.743\)0\.813 \(0\.721\)0\.772 \(0\.592\)0\.80 \(0\.707\)Amazon Nova Premier0\.825 \(0\.606\)0\.779 \(0\.526\)0\.800 \(0\.571\)0\.866 \(0\.763\)0\.835 \(0\.734\)0\.862 \(0\.760\)0\.864 \(0\.740\)0\.812 \(0\.647\)0\.830 \(0\.618\)Qwen 3 32B0\.681 \(0\.620\)0\.659 \(0\.561\)0\.680 \(0\.613\)0\.761 \(0\.753\)0\.794 \(0\.770\)0\.782 \(0\.770\)0\.787 \(0\.729\)0\.702 \(0\.630\)0\.707 \(0\.612\)TruthfulQAClaude 4\.5 Sonnet0\.997 \(0\.944\)0\.996 \(0\.926\)0\.994 \(0\.926\)0\.997 \(0\.955\)0\.997 \(0\.964\)0\.996 \(0\.949\)0\.994 \(0\.926\)0\.997 \(0\.949\)0\.979 \(0\.503\)DeepSeek R10\.931 \(0\.604\)0\.897 \(0\.538\)0\.940 \(0\.844\)0\.954 \(0\.886\)0\.957 \(0\.888\)0\.926 \(0\.819\)0\.940 \(0\.844\)0\.949 \(0\.721\)0\.907 \(0\.530\)GPT\-OSS 20B0\.872 \(0\.633\)0\.865 \(0\.675\)0\.916 \(0\.936\)0\.921 \(0\.953\)0\.918 \(0\.952\)0\.909 \(0\.946\)0\.916 \(0\.936\)0\.864 \(0\.608\)0\.856 \(0\.525\)Amazon Nova Premier0\.920 \(0\.616\)0\.948 \(0\.741\)0\.946 \(0\.862\)0\.963 \(0\.931\)0\.946 \(0\.877\)0\.963 \(0\.919\)0\.946 \(0\.862\)0\.953 \(0\.828\)0\.907 \(0\.596\)Qwen 3 32B0\.780 \(0\.553\)0\.752 \(0\.500\)0\.856 \(0\.936\)0\.865 \(0\.957\)0\.857 \(0\.946\)0\.853 \(0\.950\)0\.856 \(0\.936\)0\.822 \(0\.694\)0\.812 \(0\.636\)

Table 5:Selective prediction performance \(Sel\-AUC; AUROC in parentheses\) – Combined StructU\+Self\-ConsU\.Sel\-AUC measures area under the risk–coverage curve \(higher is better\)\. Positive class = robust failure under sampling \(τ=1\.0\\tau\{=\}1\.0\)\.StructU\+Self\-ConsUreports the combined estimator \(within,across,total\) using TrueSkill\+PageRank as the preference backend\. Baselines:Self\-ConsU,VerbalizedU,SemanticU\. Bold = best; underline = second\-best per row\.
### D\.3MMLU\-Pro Domain Breakdown

Table[6](https://arxiv.org/html/2606.17312#A4.T6)reports performance across MMLU\-Pro domains\. In Physics and Math, StructU\-within consistently outperforms other components, aligning with the intuition that these domains admit multiple valid derivations\. In Engineering and Law, the hybrid variants show largest gains, suggesting structural rankings provide scaffolding that improves selective prediction when paired with self\-consistency\.

DomainModelStructU \(Ours\)StructU\+ConsU \(Ours\)BaselineswithinacrosstotalwithinacrosstotalSelf\-ConsUSemanticUChemistryClaude 4\.5 Sonnet0\.928 \(0\.793\)0\.940 \(0\.782\)0\.842 \(0\.305\)0\.945 \(0\.871\)0\.955 \(0\.890\)0\.950 \(0\.892\)0\.935 \(0\.867\)0\.916 \(0\.583\)DeepSeek R10\.921 \(0\.729\)0\.856 \(0\.492\)0\.808 \(0\.328\)0\.959 \(0\.931\)0\.939 \(0\.899\)0\.915 \(0\.864\)0\.937 \(0\.889\)0\.860 \(0\.497\)GPT\-OSS 20B0\.812 \(0\.583\)0\.809 \(0\.493\)0\.840 \(0\.462\)0\.901 \(0\.903\)0\.903 \(0\.904\)0\.929 \(0\.926\)0\.889 \(0\.911\)0\.836 \(0\.538\)Amazon Nova Premier0\.739 \(0\.660\)0\.685 \(0\.451\)0\.701 \(0\.398\)0\.872 \(0\.977\)0\.877 \(0\.980\)0\.886 \(0\.985\)0\.875 \(0\.982\)0\.821 \(0\.734\)Qwen 3 32B0\.742 \(0\.680\)0\.716 \(0\.540\)0\.618 \(0\.344\)0\.840 \(0\.980\)0\.838 \(0\.978\)0\.834 \(0\.976\)0\.839 \(0\.977\)0\.764 \(0\.712\)EngineeringClaude 4\.5 Sonnet0\.854 \(0\.813\)0\.853 \(0\.791\)0\.658 \(0\.194\)0\.913 \(0\.946\)0\.906 \(0\.942\)0\.893 \(0\.925\)0\.899 \(0\.937\)0\.803 \(0\.587\)DeepSeek R10\.829 \(0\.625\)0\.755 \(0\.453\)0\.791 \(0\.455\)0\.950 \(0\.965\)0\.931 \(0\.944\)0\.928 \(0\.943\)0\.943 \(0\.950\)0\.849 \(0\.592\)GPT\-OSS 20B0\.464 \(0\.406\)0\.485 \(0\.363\)0\.675 \(0\.557\)0\.834 \(0\.977\)0\.839 \(0\.979\)0\.847 \(0\.982\)0\.836 \(0\.978\)0\.604 \(0\.516\)Amazon Nova Premier0\.598 \(0\.596\)0\.529 \(0\.383\)0\.597 \(0\.447\)0\.826 \(0\.981\)0\.832 \(0\.989\)0\.844 \(0\.990\)0\.813 \(0\.984\)0\.750 \(0\.729\)Qwen 3 32B0\.683 \(0\.652\)0\.642 \(0\.581\)0\.570 \(0\.376\)0\.804 \(0\.977\)0\.802 \(0\.978\)0\.804 \(0\.978\)0\.799 \(0\.978\)0\.688 \(0\.665\)LawClaude 4\.5 Sonnet0\.848 \(0\.803\)0\.818 \(0\.705\)0\.648 \(0\.268\)0\.876 \(0\.879\)0\.853 \(0\.846\)0\.803 \(0\.790\)0\.832 \(0\.838\)0\.723 \(0\.483\)DeepSeek R10\.717 \(0\.568\)0\.725 \(0\.552\)0\.695 \(0\.464\)0\.788 \(0\.826\)0\.793 \(0\.822\)0\.775 \(0\.809\)0\.772 \(0\.822\)0\.713 \(0\.529\)GPT\-OSS 20B0\.460 \(0\.619\)0\.437 \(0\.535\)0\.422 \(0\.443\)0\.541 \(0\.914\)0\.530 \(0\.908\)0\.530 \(0\.909\)0\.537 \(0\.908\)0\.450 \(0\.540\)Amazon Nova Premier0\.471 \(0\.478\)0\.502 \(0\.516\)0\.542 \(0\.588\)0\.581 \(0\.846\)0\.620 \(0\.869\)0\.643 \(0\.888\)0\.589 \(0\.861\)0\.578 \(0\.599\)Qwen 3 32B0\.454 \(0\.572\)0\.446 \(0\.578\)0\.431 \(0\.499\)0\.571 \(0\.934\)0\.553 \(0\.917\)0\.538 \(0\.910\)0\.554 \(0\.922\)0\.463 \(0\.606\)MathClaude 4\.5 Sonnet0\.963 \(0\.839\)0\.956 \(0\.787\)0\.860 \(0\.243\)0\.974 \(0\.927\)0\.973 \(0\.923\)0\.959 \(0\.868\)0\.950 \(0\.888\)0\.948 \(0\.650\)DeepSeek R10\.952 \(0\.711\)0\.945 \(0\.658\)0\.879 \(0\.353\)0\.968 \(0\.926\)0\.965 \(0\.930\)0\.950 \(0\.886\)0\.945 \(0\.892\)0\.918 \(0\.556\)GPT\-OSS 20B0\.880 \(0\.567\)0\.891 \(0\.524\)0\.907 \(0\.479\)0\.960 \(0\.940\)0\.962 \(0\.956\)0\.966 \(0\.962\)0\.961 \(0\.944\)0\.904 \(0\.548\)Amazon Nova Premier0\.796 \(0\.668\)0\.779 \(0\.550\)0\.762 \(0\.441\)0\.899 \(0\.961\)0\.896 \(0\.968\)0\.894 \(0\.957\)0\.878 \(0\.958\)0\.849 \(0\.689\)Qwen 3 32B0\.815 \(0\.653\)0\.812 \(0\.637\)0\.726 \(0\.442\)0\.915 \(0\.981\)0\.921 \(0\.995\)0\.915 \(0\.988\)0\.894 \(0\.982\)0\.796 \(0\.577\)PhysicsClaude 4\.5 Sonnet0\.955 \(0\.822\)0\.956 \(0\.825\)0\.832 \(0\.207\)0\.967 \(0\.910\)0\.969 \(0\.922\)0\.958 \(0\.891\)0\.965 \(0\.893\)0\.919 \(0\.577\)DeepSeek R10\.921 \(0\.649\)0\.918 \(0\.585\)0\.890 \(0\.470\)0\.959 \(0\.923\)0\.962 \(0\.925\)0\.960 \(0\.925\)0\.963 \(0\.907\)0\.895 \(0\.542\)GPT\-OSS 20B0\.789 \(0\.530\)0\.800 \(0\.480\)0\.811 \(0\.420\)0\.907 \(0\.933\)0\.921 \(0\.943\)0\.935 \(0\.958\)0\.930 \(0\.943\)0\.813 \(0\.484\)Amazon Nova Premier0\.787 \(0\.691\)0\.705 \(0\.454\)0\.721 \(0\.413\)0\.887 \(0\.964\)0\.893 \(0\.968\)0\.901 \(0\.973\)0\.889 \(0\.965\)0\.833 \(0\.742\)Qwen 3 32B0\.793 \(0\.750\)0\.739 \(0\.599\)0\.676 \(0\.405\)0\.874 \(0\.979\)0\.875 \(0\.979\)0\.875 \(0\.977\)0\.879 \(0\.976\)0\.813 \(0\.758\)

Table 6:Selective prediction \(Sel\-AUC; AUROC in parentheses\)\.Higher is better; positive class = robust failure \(τ=1\.0\\tau\{=\}1\.0\)\.StructUreports structural uncertainty \(within,across,total\), selecting the best variant among Bradley–Terry\+PageRank and TrueSkill\+PageRank\.StructU\+Self\-ConsUreports the combined estimators\. Baselines:Self\-ConsU,SemanticU\. Bold = best; underline = second\-best per row\.
### D\.4Robustness to Correctness Thresholds

Table[7](https://arxiv.org/html/2606.17312#A4.T7)reports AUROC under three correctness thresholdsτ∈\{1\.0,0\.8,0\.6\}\\tau\\in\\\{1\.0,0\.8,0\.6\\\}\. All uncertainty signals exhibit modest degradation as the threshold is relaxed, confirming that separability is not an artifact of strict labeling but remains stable under looser correctness definitions\.

DatasetModelSelf\-ConsUStructU\_within \(Bradley–Terry\+PageRank\)StructU\+Self\-ConsU\_withinτ=1\.0\\tau\{=\}1\.0τ=0\.8\\tau\{=\}0\.8τ=0\.6\\tau\{=\}0\.6τ=1\.0\\tau\{=\}1\.0τ=0\.8\\tau\{=\}0\.8τ=0\.6\\tau\{=\}0\.6τ=1\.0\\tau\{=\}1\.0τ=0\.8\\tau\{=\}0\.8τ=0\.6\\tau\{=\}0\.6Math\-SynthClaude 4\.5 Sonnet0\.9780\.9780\.8460\.9860\.9050\.7730\.9920\.9770\.846DeepSeek R10\.8890\.8890\.7980\.9020\.8530\.7700\.9310\.9160\.798GPT\-OSS 20B0\.9180\.9180\.7920\.7750\.7790\.7100\.9550\.9200\.792Amazon Nova Premier0\.9860\.9860\.9110\.9380\.9120\.8460\.9970\.9870\.911Qwen 3 32B0\.9900\.9900\.8620\.8500\.7660\.7100\.9980\.9910\.862MATH\-500Claude 4\.5 Sonnet0\.8180\.8110\.7750\.8140\.7340\.6810\.8420\.8070\.760DeepSeek R10\.7650\.7580\.7260\.6310\.5960\.5540\.7680\.7570\.717GPT\-OSS 20B0\.7050\.6790\.6180\.5770\.5340\.4820\.7060\.6860\.618Amazon Nova Premier0\.8760\.8520\.8240\.7020\.7510\.7140\.8740\.8580\.818Qwen 3 32B0\.8540\.8720\.8420\.7670\.7950\.7420\.8860\.8940\.814MMLU\-ProClaude 4\.5 Sonnet0\.8840\.8330\.7830\.8340\.7680\.7280\.9120\.8650\.819DeepSeek R10\.8820\.8130\.7570\.5590\.5020\.4730\.8880\.8180\.760GPT\-OSS 20B0\.9350\.8700\.8120\.4560\.3940\.3800\.9180\.8500\.785Amazon Nova Premier0\.9450\.8460\.7350\.6100\.5060\.4640\.9330\.8380\.723Qwen 3 32B0\.9660\.9040\.7990\.6730\.6600\.5860\.9700\.9110\.796

Table 7:*Robustness to correctness thresholds \(per\-model AUROC\)\.*AUROC for detecting*robust failures under sampling*at thresholdsτ∈\{1\.0,0\.8,0\.6\}\\tau\\in\\\{1\.0,0\.8,0\.6\\\}withN=5N\{=\}5\. A question is labeled incorrect ifp^corr<τ\\hat\{p\}\_\{\\mathrm\{corr\}\}<\\tau\. We report: Self\-ConsU, StructU\_within \(within\-trial from Bradley–Terry\+PageRank\), and the combined StructU\+Self\-ConsU\_within\. Higher is better\.

## Appendix EAdditional Analysis of Structural Uncertainty Signals

### E\.1Structural Collapse Across Additional Models for HotpotQA

![Refer to caption](https://arxiv.org/html/2606.17312v1/images/combined_StructU_within_StructU_accross_4models.png)Figure 10:Structural collapse on factual retrieval across four additional models\.Rows correspond to Amazon Nova Premier, DeepSeek R1, GPT\-OSS 20B, and Qwen 3 32B\. Each row shows the across\-trial uncertainty distribution \(left\), within\-trial uncertainty distribution \(center\), and joint across\-trial–within\-trial scatter \(right\) on Math\-Synth and HotpotQA, conditioned on correctness \(Bradley–Terry \+ PageRank\)\. The “HotpotQA signature”—near\-zero across\-trial uncertainty with near\-maximum within\-trial uncertainty \(log⁡5≈1\.61\\log 5\\approx 1\.61, dotted line\)—is reproduced across all models, confirming that the structural collapse is a task\-level phenomenon independent of model capability\.Figure[10](https://arxiv.org/html/2606.17312#A5.F10)extends the structural collapse analysis to Amazon Nova Premier, DeepSeek R1, GPT\-OSS 20B, and Qwen 3 32B\. The “HotpotQA signature”—near\-zero across\-trial uncertainty with near\-maximum within\-trial uncertainty \(log⁡5≈1\.61\\log 5\\approx 1\.61\)—reproduces identically across all models, confirming that factual retrieval elicits prompt\-invariant reasoning chains rendering preference graphs uninformative regardless of model capability\. In contrast, Math\-Synth retains clear across\-trial separation between correct and incorrect questions for all models, though the dynamic range narrows for weaker models \(Amazon Nova Premier, Qwen 3 32B\)\.

This universality aligns with Table[1](https://arxiv.org/html/2606.17312#S4.T1): on HotpotQA, StructU\+Self\-ConsU fails to improve over Self\-ConsU alone for DeepSeek R1 \(0\.796 vs 0\.835 Sel\-AUC\) and Claude 4\.5 \(0\.742 vs 0\.839\), confirming degenerate preference graphs introduce noise rather than signal\.

### E\.2Qualitative Assessment of Structural Collapse

To provide mechanistic evidence for the quantitative findings in Section[4\.2](https://arxiv.org/html/2606.17312#S4.SS2), we analyze four examples from Claude 4\.5 Sonnet withSelf\-ConsU=0\\text\{Self\-ConsU\}=0\(unanimous agreement\), isolating structural signals where dispersion\-based methods are uninformative\. Table[8](https://arxiv.org/html/2606.17312#A5.T8)summarizes key quantities\.

DatasetCorrect?Self\-ConsUStructUacross\\text\{StructU\}\_\{\\text\{across\}\}StructUwithin\\text\{StructU\}\_\{\\text\{within\}\}πmax/πmin\\pi\_\{\\max\}/\\pi\_\{\\min\}CVmax\\text\{CV\}\_\{\\max\}DiagnosisMath\-Synth✗0\.00\.0351\.5491\.730\.497Signal firesMath\-Synth✓0\.00\.0011\.6071\.090\.047Appropriately quietHotpotQA✗0\.0<<0\.0011\.6091\.060\.013CollapsedHotpotQA✓0\.0<<0\.0011\.6081\.080\.015CollapsedTable 8:Summary of qualitative examples\. All four satisfySelf\-ConsU=0\\text\{Self\-ConsU\}=0\.StructUacross\\text\{StructU\}\_\{\\text\{across\}\}: across\-trial uncertainty;StructUwithin\\text\{StructU\}\_\{\\text\{within\}\}: within\-trial uncertainty\.πmax/πmin\\pi\_\{\\max\}/\\pi\_\{\\min\}: ratio of largest to smallest mean PageRank score\.CVmax\\text\{CV\}\_\{\\max\}: maximum coefficient of variation of PageRank across spanning tree trials\.
### E\.3Math\-Synth: Structural Diversity Enables Discrimination

#### E\.3\.1Incorrect Example \(Self\-ConsU=0\\text\{Self\-ConsU\}=0,StructUacross=0\.035\\text\{StructU\}\_\{\\text\{across\}\}=0\.035\)

##### Task\.

A 6\-digit synthetic arithmetic problem involving nested negations and multiplication:

−\(−\(−\(−\(−\(−⏟6​outer negations−\(−500\)⏟=500×200\)\)\)\)\)\)−−\(−\(−1\)\)⏟=1\\underbrace\{\-\(\-\(\-\(\-\(\-\(\-\}\_\{6\\text\{ outer negations\}\}\\underbrace\{\-\(\-500\)\}\_\{=500\}\\times 200\\;\)\\;\)\\;\)\\;\)\\;\)\\;\)\\;\\;\\underbrace\{\-\-\(\-\(\-1\)\)\}\_\{=1\}

The left sub\-expression evaluates as−\(−500\)×200=100,000\-\(\-500\)\\times 200=100\{,\}000, then wrapped in6 outer negations\(even count→\\topositive\), yielding\+100,000\+100\{,\}000\. Adding the right part:100,000\+1=100,001100\{,\}000\+1=100\{,\}001\. All five responses unanimously answer−99,999\-99\{,\}999; the ground truth is𝟏𝟎𝟎,𝟎𝟎𝟏\\mathbf\{100\{,\}001\}\.

##### Shared error\.

All responses miscount outer negations as5\(odd→\\rightarrownegative\) instead of6\(even→\\rightarrowpositive\), flipping the sign from\+100,000\+100\{,\}000to−100,000\-100\{,\}000via different error paths \(Table[9](https://arxiv.org/html/2606.17312#A5.T9)\)\.

IDStrategyKey Reasoning Stepsπ¯\\bar\{\\pi\}R1Step\-by\-step w/ self\-check−\(−500\)×200=100,000\-\(\-500\)\\times 200=100\{,\}000→\\;\\rightarrow\\;counts 7 total negation signs→\\;\\rightarrow\\;counts 5 remaining after multiply \(correct: 6\)→\\;\\rightarrow\\;odd→\\rightarrow−100,000\-100\{,\}000\[correct:even→\\rightarrow\+100,000\+100\{,\}000\]→\\;\\rightarrow\\;self\-check re\-derives each step but*repeats same miscount*→\\;\\rightarrow\\;−100,000\+1=−99,999\-100\{,\}000\+1=\-99\{,\}9990\.238R2Think\-aloudMetacognitive narration: “I need to count remaining negations after multiplication”→\\;\\rightarrow\\;identifies 5 remaining \(correct: 6\)→\\;\\rightarrow\\;−100,000\-100\{,\}000\[correct:\+100,000\+100\{,\}000\]→\\;\\rightarrow\\;−100,000\+1=−99,999\-100\{,\}000\+1=\-99\{,\}9990\.215R3SocraticReframes as parity problem:“7 negations total \(odd\) applied to−500×200=−100,000\-500\\times 200=\-100\{,\}000gives−100,000\-100\{,\}000”— conflates inner negation with outer count \[correct:6 outer negations \(even\) applied to\+100,000\+100\{,\}000\]→\\;\\rightarrow\\;−100,000\+1=−99,999\-100\{,\}000\+1=\-99\{,\}9990\.224R4DecompositionSub\-problem 1: left expression→\\;\\rightarrow\\;Sub\-problem 2: right expression→\\;\\rightarrow\\;−100,000\-100\{,\}000\[correct:\+100,000\+100\{,\}000\]\+1=−99,999\+1=\-99\{,\}999\. Most modular structure;*no negation count shown, no verification step*\.0\.138R5Analogical reasoningRelates to simpler cases \(−\(−5\)=5\-\(\-5\)=5,−\(−\(−3\)\)=−3\-\(\-\(\-3\)\)=\-3\)→\\;\\rightarrow\\;parses−500×200=−100,000\-500\\times 200=\-100\{,\}000then applies 6 negations to−100,000\-100\{,\}000\[correct:−\(−500\)×200=\+100,000\-\(\-500\)\\times 200=\+100\{,\}000then 6 negations\]→\\;\\rightarrow\\;initial miscount, self\-corrects counting but retains inner parsing error→\\;\\rightarrow\\;−100,000\+1=−99,999\-100\{,\}000\+1=\-99\{,\}9990\.185Table 9:Reasoning traces for the Math\-Synth incorrect example\. The correct evaluation requires6 outer negations\(even→\\rightarrow\+100,000\+100\{,\}000\), but all responses miscount5\(odd→\\rightarrow−100,000\-100\{,\}000\)\. Each response reaches the same wrong answer via a structurally different error path\. Ground truth:100,001100\{,\}001; unanimous model answer:−99,999\-99\{,\}999\.
##### PageRank dynamics\.

The structural diversity across responses produces a skewed PageRank distribution \(πmax/πmin=1\.73\\pi\_\{\\max\}/\\pi\_\{\\min\}=1\.73\) with substantial instability across spanning tree trials\. Table[10](https://arxiv.org/html/2606.17312#A5.T10)reports the per\-trial PageRank vectors\.

R1R2R3R4R5Trial 00\.2630\.2520\.2430\.0550\.186Trial 10\.2250\.1900\.2120\.2120\.161Trial 20\.2050\.2190\.1890\.1640\.224Trial 30\.2650\.2330\.2550\.0570\.190Trial 40\.2330\.1800\.2220\.2020\.163Mean0\.2380\.2150\.2240\.1380\.185CV0\.0970\.1240\.1030\.4970\.124Table 10:Per\-trial PageRank distributions for the Math\-Synth incorrect example\. Response 4 \(decomposition, no verification\) exhibits extreme instability \(CV=0\.497=0\.497\), fluctuating between0\.0550\.055and0\.2120\.212across trials\.
##### Analysis\.

Responses differ substantively in verification depth \(R1: explicit self\-check; R4: none\), metacognitive structure \(R2: narrated strategy; R5: mid\-derivation correction\), and error pathway \(R1–R4: miscount negations; R5: misparse grouping\)—genuine derivation quality differences, not surface reformulations\.

Table[11](https://arxiv.org/html/2606.17312#A5.T11)shows these differences produce unstable preferences\. Across 20 judgments, 80% of confidence scores are≤\\leq65, indicating near\-indifference\. The R1 vs R2 pair reveals instability: the judge cites ”clarity” to prefer R1 in trials 3–4 but R2 in trials 1 and 5\. R4’s PageRank fluctuates between 0\.055 and 0\.212 \(CV=0\.497\) because ranking depends on which spanning tree edges are sampled\.

PairWinner TrialsConf\.InstabilityR1 vs R2Flips: 3,4≻\\succ1,555–62”Clarity” cited both waysR4 vs R5R5: 3,4 / R4: 165–85R4 rejected \(85\) vs R5, marginal \(55\) vs R1R1 vs R5R1: 2,3,4 stable72R1 self\-check valued over R5 correctionTable 11:Judge preference instability \(Math\-Synth incorrect\)\. Confidence: 50=no preference, 100=certainty\.This instability maps onto the two uncertainty components\.Across\-trial\(StructUacross=0\.035\\text\{StructU\}\_\{\\text\{across\}\}=0\.035\): competing quality criteria \(verification depth, metacognitive clarity, modularity\) produce different winners depending on sampled edges, shifting PageRank vectors between trials\.Within\-trial\(StructUwithin=1\.549\\text\{StructU\}\_\{\\text\{within\}\}=1\.549\): low confidence \(≤\\leq65\) prevents domination, distributing PageRank mass across multiple candidates per trial\. Together, elevatedStructUacross\\text\{StructU\}\_\{\\text\{across\}\}correctly flags unreliability despiteSelf\-ConsU=0\\text\{Self\-ConsU\}=0, while highStructUwithin\\text\{StructU\}\_\{\\text\{within\}\}reflects multiple distinct—though uniformly flawed—reasoning strategies\.

#### E\.3\.2Correct Example \(Self\-ConsU=0\\text\{Self\-ConsU\}=0,StructUacross≈0\.001\\text\{StructU\}\_\{\\text\{across\}\}\\approx 0\.001\)

##### Task\.

A 14\-digit synthetic arithmetic problem involving nested negations and multiplication:

−\(−\(−\(−\(−\(−⏟6​outer negations\(−9 999 999 999 999⏟13​nines×−1\)\+1\)\)\)\)\)\)\\underbrace\{\-\(\-\(\-\(\-\(\-\(\-\}\_\{6\\text\{ outer negations\}\}\(\\underbrace\{\-9\\,999\\,999\\,999\\,999\}\_\{13\\text\{ nines\}\}\\times\-1\)\+1\\;\)\\;\)\\;\)\\;\)\\;\)\\;\)

The inner product evaluates to\(−9 999 999 999 999\)×\(−1\)=9 999 999 999 999\(\{\-9\\,999\\,999\\,999\\,999\}\)\\times\(\{\-1\}\)=9\\,999\\,999\\,999\\,999\. Adding11gives10 000 000 000 00010\\,000\\,000\\,000\\,000\. Six outer negations \(even count→\\tosign unchanged\) yield\+10 000 000 000 000\+10\\,000\\,000\\,000\\,000\. All five responses unanimously answer10 000 000 000 00010\\,000\\,000\\,000\\,000, matching the ground truth\.

##### All responses correct\.

Every response reaches the correct answer through sound reasoning\. The five prompt templates produce identical arithmetic but differ in presentation: R1 adds self\-check with \(verified\) annotations; R2 narrates thinking with even/odd shortcut; R3 frames as Socratic Q&A; R4 decomposes into sub\-problems; R5 draws analogy to−\(−\(−3\)\)\-\(\-\(\-3\)\)\. Because computation is straightforward and all strategies succeed, variation is purely stylistic \(Table[12](https://arxiv.org/html/2606.17312#A5.T12)\)\.

IDStrategyKey Reasoning Stepsπ¯\\bar\{\\pi\}R1Step\-by\-step w/ self\-check\(−9​⋯​9\)×\(−1\)=9​⋯​9\(\-9\\\!\\cdots\\\!9\)\\times\(\-1\)=9\\\!\\cdots\\\!9→\\;\\rightarrow\\;adds 1 to get101310^\{13\}→\\;\\rightarrow\\;counts 6 outer negations \(even→\\rightarrowpositive\)→\\;\\rightarrow\\;applies each negation step\-by\-step \(Steps 3–8\)→\\;\\rightarrow\\;*self\-check*recounts negation signs from original expression, re\-verifies each step with ✓ marks→\\;\\rightarrow\\;10,000,000,000,00010\{,\}000\{,\}000\{,\}000\{,\}0000\.220R2Think\-aloudMetacognitive narration: “I need to work from the innermost parentheses outward”→\\;\\rightarrow\\;same arithmetic→\\;\\rightarrow\\;counts 6 negative signs→\\;\\rightarrow\\;applies each negation→\\;\\rightarrow\\;notes “6 negations = even = positive result”→\\;\\rightarrow\\;10,000,000,000,00010\{,\}000\{,\}000\{,\}000\{,\}0000\.215R3Socratic“What is the problem asking?” … “What method should I use?”→\\;\\rightarrow\\;identifies 6 consecutive negation operations→\\;\\rightarrow\\;applies each negation \(Steps 3–8\)→\\;\\rightarrow\\;“Does my answer make sense? Yes\. 6 negations \(even\)→\\rightarrowpositive”→\\;\\rightarrow\\;10,000,000,000,00010\{,\}000\{,\}000\{,\}000\{,\}0000\.195R4DecompositionSub\-problem 1: inner product→\\;\\rightarrow\\;Sub\-problem 2: add 1→\\;\\rightarrow\\;Sub\-problem 3: apply 6 negations→\\;\\rightarrow\\;“even number of negations \(6\), the result is positive”→\\;\\rightarrow\\;10,000,000,000,00010\{,\}000\{,\}000\{,\}000\{,\}000\. Most modular structure; concise summary paragraph\.0\.210R5Analogical reasoningRelates to simpler cases: “similar to evaluating−\(−\(−3\)\)\-\(\-\(\-3\)\)”→\\;\\rightarrow\\;states−\(−x\)=x\-\(\-x\)=xprinciple→\\;\\rightarrow\\;same step\-by\-step negations→\\;\\rightarrow\\;“6 is even, result is positive”→\\;\\rightarrow\\;10,000,000,000,00010\{,\}000\{,\}000\{,\}000\{,\}0000\.160Table 12:Reasoning traces for the Math\-Synth correct example\. The correct evaluation requires6 outer negations\(even→\\rightarrow\+1013\+10^\{13\}\), and all responses count correctly\. Each response reaches the same right answer via a stylistically different but arithmetically equivalent path\. Ground truth:10,000,000,000,00010\{,\}000\{,\}000\{,\}000\{,\}000; unanimous model answer:10,000,000,000,00010\{,\}000\{,\}000\{,\}000\{,\}000\.
##### PageRank dynamics\.

The purely stylistic variation across responses produces a compressed PageRank distribution \(πmax/πmin=1\.38\\pi\_\{\\max\}/\\pi\_\{\\min\}=1\.38\) with minimal instability across spanning tree trials\. Table[13](https://arxiv.org/html/2606.17312#A5.T13)reports the per\-trial PageRank vectors\.

R1R2R3R4R5Trial 00\.2250\.2400\.1550\.2350\.145Trial 10\.2300\.2200\.1850\.2150\.150Trial 20\.2150\.2100\.2050\.2050\.165Trial 30\.2250\.2150\.1800\.2100\.170Trial 40\.2200\.2100\.2000\.2100\.160Mean0\.2200\.2150\.1950\.2100\.160CV0\.0240\.0540\.0890\.0540\.060Table 13:Per\-trial PageRank distributions for the Math\-Synth correct example\. All responses exhibit low coefficient of variation; compare R4’s CV==0\.054 here with CV==0\.497 in the incorrect example \(Table[10](https://arxiv.org/html/2606.17312#A5.T10)\)\.
##### Analysis\.

Responses differ only in expository format: verification depth \(R1: explicit self\-check; R4: summary paragraph\), pedagogical framing \(R3: Socratic dialogue; R5: simpler analogues\), and narrative style \(R2: thinking\-aloud\)\. Unlike the incorrect example, these presentational differences lack substantive reasoning quality differences—every response counts negations correctly and arrives at the correct answer\.

Table[14](https://arxiv.org/html/2606.17312#A5.T14)shows these stylistic\-only differences produce*stable*preferences\. Across 20 judgments,zero reversalsoccur\. The judge applies consistent tie\-breaking: explicit self\-verification valued over conciseness \(R1≻\\succR2\), directness over pedagogical framing \(R4≻\\succR3\), both over analogical scaffolding \(R5 ranked last\)\. Confidence clusters at 52–62, reflecting genuine discrimination difficulty that is*stable*rather than*variable*\.

PairWinnerConf\.PatternR1 vs R5R1: 1,3,4,555Stable; ”self\-check adds rigor”R1 vs R2R1: 1,2,455Stable; ”verification” over ”narration”R4 vs R5R4: 3,4,552Stable; ”decomposition more focused”Table 14:Judge preference stability \(Math\-Synth correct\)\. Zero reversals\. Confidence: 50=no preference, 100=certainty\.This stability maps onto the uncertainty components\.Across\-trial\(StructUacross≈0\.001\\text\{StructU\}\_\{\\text\{across\}\}\\approx 0\.001\) is near zero because stylistic preferences—however weakly held—are reproducible: the same criterion applied in the same direction every trial \(Table[14](https://arxiv.org/html/2606.17312#A5.T14)\), so PageRank vectors barely shift \(Table[13](https://arxiv.org/html/2606.17312#A5.T13), all CVs≤\\leq0\.089\)\.Within\-trial\(StructUwithin\\text\{StructU\}\_\{\\text\{within\}\}\) remains moderate because low confidence \(52–62\) prevents single\-response domination\. The near\-zeroStructUacross\\text\{StructU\}\_\{\\text\{across\}\}correctly identifies reliability despiteSelf\-ConsU=0\\text\{Self\-ConsU\}=0\. The≈30×\\approx 30\\timesdifference from the incorrect example \(0\.0010\.001vs0\.0350\.035\) demonstrates the core claim: when reliably right, preferences are stable; when reliably wrong, preferences destabilize, even with identical surface agreement\.

### E\.4HotpotQA: Preference Graph Collapse

#### E\.4\.1Incorrect Example \(Self\-ConsU=0\\text\{Self\-ConsU\}=0,StructUacross<0\.001\\text\{StructU\}\_\{\\text\{across\}\}<0\.001\)

##### Task\.

A multi\-hop question from HotpotQA:*“Text Me Merry Christmas” is a song performed by Kristen Bell and a group that originated at what university?*Expected reasoning: \(i\) identify the group asStraight No Chaserfrom retrieved passage \(Document 10\), then \(ii\) locate the group’s university of origin\.Ground truth:Indiana University\. All five responses unanimously state the provided documents lack this information\.

##### Shared failure\.

All responses correctly identify Straight No Chaser from Document 10 \(hop 1\) but fail hop 2 due to missing context describing the group’s origins\. Every response correctly reports the answer cannot be determined—unanimous incorrect agreement\(Self\-ConsU=0\\text\{Self\-ConsU\}=0\) driven by shared retrieval gap, not reasoning error\. Table[15](https://arxiv.org/html/2606.17312#A5.T15)shows identical retrieval chains despite different prompts, with variation limited to surface reformulation\.

IDStrategyKey Reasoning Stepsπ¯\\bar\{\\pi\}R1Step\-by\-step w/ self\-checkIdentifies Doc 10→\\;\\rightarrow\\;extracts “Straight No Chaser and Kristen Bell”→\\;\\rightarrow\\;searches all documents for university origin→\\;\\rightarrow\\;information not found\[correct:Indiana University\]→\\;\\rightarrow\\;*self\-check*re\-verifies Doc 10 is the only relevant source, confirms gap→\\;\\rightarrow\\;abstains0\.202R2Think\-aloud“Let me read through the question carefully”→\\;\\rightarrow\\;identifies Doc 10→\\;\\rightarrow\\;scans Docs 1–9, lists each with one\-line summary \(“not relevant”\)→\\;\\rightarrow\\;information not found\[correct:Indiana University\]→\\;\\rightarrow\\;abstains0\.207R3Socratic“What is the question asking?”→\\;\\rightarrow\\;identifies Doc 10→\\;\\rightarrow\\;“none of the provided documents contain information about which university”→\\;\\rightarrow\\;information not found\[correct:Indiana University\]→\\;\\rightarrow\\;“Does my answer make sense? … the context only confirms they performed the song”→\\;\\rightarrow\\;abstains0\.197R4DecompositionSub\-question 1: identify group→\\;\\rightarrow\\;Sub\-question 2: find university→\\;\\rightarrow\\;checks Docs 1–10→\\;\\rightarrow\\;information not found\[correct:Indiana University\]→\\;\\rightarrow\\;“cannot answer based solely on provided documents”→\\;\\rightarrow\\;abstains0\.196R5Analogical reasoningFrames as two\-step retrieval pattern: “Entity X associated with Entity Y, find attribute of Y”→\\;\\rightarrow\\;identifies Doc 10→\\;\\rightarrow\\;searches all documents→\\;\\rightarrow\\;information not found\[correct:Indiana University\]→\\;\\rightarrow\\;abstains0\.198Table 15:Reasoning traces for the HotpotQA incorrect example\. All five responses correctly identifyStraight No Chaseras the group \(hop 1\) but fail hop 2 due to missing context\. Each response reaches the same abstention via a stylistically different but substantively identical path\. Ground truth: Indiana University; unanimous model answer:*cannot be determined*\.
##### PageRank dynamics\.

Because responses are*substantively identical*—each performs the same successful first hop and failed second hop—the judge has even less basis for discrimination than in Math\-Synth correct\. Table[16](https://arxiv.org/html/2606.17312#A5.T16)confirms collapse: all trials produce near\-identical, near\-uniform distributions with maximum deviation from1/N=0\.2001/N=0\.200of just0\.0080\.008\.

R1R2R3R4R5Trial 00\.2020\.2080\.1960\.1960\.196Trial 10\.2010\.2070\.2010\.1950\.195Trial 20\.2010\.2070\.1950\.1950\.201Trial 30\.2020\.2080\.1960\.1960\.196Trial 40\.2010\.2070\.1950\.1950\.201Mean0\.2020\.2070\.1970\.1960\.198CV0\.0030\.0030\.0110\.0030\.013Table 16:PageRank distributions across five responses for HotpotQA incorrect trials\. Near\-uniform distributions \(max deviation from1/N=0\.2001/N=0\.200of just0\.0080\.008\) confirm full rank collapse\.
##### Analysis\.

Unlike Math\-Synth incorrect—where responses reached the same wrong*answer*via structurally different*error paths*—here responses share both answer \(abstention\) and reasoning outcome \(successful hop 1, failed hop 2\)\. Variation is purely expository: R1 adds self\-check, R2 lists documents, R3 uses Socratic Q&A, R4 decomposes, and R5 frames as retrieval pattern\.

Table[17](https://arxiv.org/html/2606.17312#A5.T17)shows the judge finds essentially nothing to discriminate\. Across∼35\{\\sim\}35judgments,one reversaloccurs \(R3 vs R4, iteration 4, conf=52\)\. Confidence: 80% at 52, R4 vs R5 at literal 50 \(coin\-flip\) all appearances\. Only outlier: R2 vs R5 at 62 citing R2’s ”explicit document\-by\-document listing”—the sole substantive distinction\.

PairWinnerConf\.PatternR2 vs R1R2: all trials52Stable; doc\-by\-doc listing preferredR4 vs R5Tie: all trials50Literal coin\-flip; ”essentially a tie”R3 vs R4R4: 2,5 / R3: 452–55Only reversalTable 17:Judge preferences \(HotpotQA incorrect\)\. One reversal across∼35\{\\sim\}35comparisons\. Confidence: 50=no preference\.This near\-total indifference maps onto uncertainty components\. The collapse mechanism: factual retrieval over a fixed document set is deterministic\. The model scans keywords, identifies documents, locates answer or does not\. Different prompts cannot induce different retrieval strategies—reasoning chains are determined by document structure, not prompt framing\. Consequently, pairwise judgments find nothing to discriminate, PageRank converges to near\-uniformity\.Across\-trial\(StructUacross<0\.001\\text\{StructU\}\_\{\\text\{across\}\}<0\.001\) is near zero because all trials agree on uniformity—genuinely nothing to rank\.Within\-trial\(StructUwithin≈log⁡5=1\.609\\text\{StructU\}\_\{\\text\{within\}\}\\approx\\log 5=1\.609\) reaches theoretical maximum, reflecting flat PageRank where no response dominates\.

##### Contrast with Math\-Synth incorrect\.

Both haveSelf\-ConsU=0\\text\{Self\-ConsU\}=0\(unanimous wrong answer\), yet structural profiles diverge sharply\. Math\-Synth involves*endogenous*error \(negation miscounting\) where different strategies produce detectably different error paths, yielding preference instability andStructUacross=0\.035\\text\{StructU\}\_\{\\text\{across\}\}=0\.035\. HotpotQA involves*exogenous*failure \(missing context\) where no reasoning diversity can compensate for absent evidence, producing substantively identical responses andStructUacross<0\.001\\text\{StructU\}\_\{\\text\{across\}\}<0\.001\. StructU distinguishes these unanimous failure regimes—one flagged unreliable \(0\.035\), the other low\-uncertainty \(¡0\.001\)—but cannot detect failures leaving no trace in preference structure\.

#### E\.4\.2Correct Example \(Self\-ConsU=0\\text\{Self\-ConsU\}=0,StructUacross<0\.001\\text\{StructU\}\_\{\\text\{across\}\}<0\.001\)

##### Task\.

A HotpotQA question:*What creature of American folklore gained notoriety in 1964?*Retrieved context discusses several folklore creatures \(Teakettler, Hidebehind, Chessie\) but none mention 1964\. All responses correctly identify this gap and abstain\.

##### Observation\.

Every response executes identical retrieval: scan all documents for ”1964” and folklore creatures → identify Documents 1, 5, 8 as partially relevant \(creatures but no 1964\) → note closest match is Chessie \(1977/1980s sightings\) → conclude information absent → abstain\. Variation is surface\-level only: R1 adds self\-check; R2 lists documents; R3 uses Socratic framing; R4 decomposes; R5 casts as date\-retrieval pattern\.

The uncertainty profile is statistically indistinguishable from the incorrect example:StructUacross<0\.001\\text\{StructU\}\_\{\\text\{across\}\}<0\.001,StructUwithin=1\.608\\text\{StructU\}\_\{\\text\{within\}\}=1\.608,πmax/πmin=1\.08\\pi\_\{\\max\}/\\pi\_\{\\min\}=1\.08, all CVs¡0\.015\. Table[18](https://arxiv.org/html/2606.17312#A5.T18)shows near\-uniform PageRank frozen across trials—virtually identical to the incorrect example \(Table[16](https://arxiv.org/html/2606.17312#A5.T16)\)\.

R1R2R3R4R5Trial 00\.2030\.2070\.1980\.1980\.194Trial 10\.2020\.2060\.1990\.1970\.196Trial 20\.2040\.2060\.1970\.1980\.195Trial 30\.2030\.2070\.1980\.1970\.195Trial 40\.2020\.2060\.1980\.1980\.196Mean0\.2030\.2060\.1980\.1980\.195CV0\.0040\.0020\.0030\.0030\.004Table 18:Per\-trial PageRank distributions for the HotpotQA correct example\. Distributions are near\-uniform and frozen across trials, yieldingStructUacross<0\.001\\text\{StructU\}\_\{\\text\{across\}\}<0\.001—statistically indistinguishable from the incorrect example \(Table[16](https://arxiv.org/html/2606.17312#A5.T16)\)\.
##### Significance\.

Identical collapse on correct and incorrect examples demonstrates this is a*task structure*property, not error status\. Retrieval over fixed documents produces prompt\-invariant chains regardless of outcome\. Different prompts cannot induce different retrieval strategies—chains are determined by document structure, not prompt framing\.

This represents aboundary conditionfor structural uncertainty\. On Math\-Synth, StructU successfully separated correct from incorrect unanimous agreement \(0\.0010\.001vs0\.0350\.035,30×30\\timesdifference\) because different prompts induced genuinely different reasoning strategies the judge could differentially rank\. On HotpotQA retrieval, StructU produces indistinguishable values \(¡0\.001 both cases\) because deterministic retrieval suppresses the reasoning diversity self\-preference requires\. The preference graph collapses in both cases, rendering StructU structurally uninformative—not because the method is flawed, but because the task affords no structural variation to exploit\. This limitation is shared with Self\-ConsU, which also reports zero in both cases, highlighting that uncertainty quantification methods relying on response diversity are fundamentally constrained when reasoning is deterministic given input context\.

### E\.5Summary and Implications

The qualitative evidence supports three conclusions:

##### \(1\) Structural uncertainty detects errors invisible to self\-consistency\.

On Math\-Synth, the incorrect example exhibits30×30\\timeshigher across\-trial uncertainty than the correct example, despite both havingSelf\-ConsU=0\\text\{Self\-ConsU\}=0\. The mechanism is that diverse prompt templates elicit structurally distinct reasoning strategies on mathematical tasks, and the model’s inability to stably rank these strategies when all are flawed produces the across\-trial uncertainty signal\.

##### \(2\) Preference graph collapse explains the HotpotQA limitation\.

On factual retrieval, different prompt templates cannot elicit different reasoning paths because the retrieval process is determined by the document set\. The resulting identical reasoning chains produce near\-uniform, stable PageRank distributions \(StructUacross≈0\\text\{StructU\}\_\{\\text\{across\}\}\\approx 0\), eliminating the structural signal regardless of correctness\.

##### \(3\) The collapse signature is itself diagnostic\.

Near\-zero across\-trial uncertainty combined with near\-maximum within\-trial uncertainty \(StructUacross≈0\\text\{StructU\}\_\{\\text\{across\}\}\\approx 0,StructUwithin≈log⁡N\\text\{StructU\}\_\{\\text\{within\}\}\\approx\\log N\) constitutes a detectable signature indicating that the model lacks a coherent internal quality criterion for the task\. This signature can inform practitioners about when to rely on structural versus dispersion\-based uncertainty methods: when it is detected, self\-preference signals are uninformative and alternative estimators should be preferred\.

### E\.6Prompt Templates

#### E\.6\.1Response Generation Prompts

We employ five distinct prompt templates to induce diverse reasoning patterns across candidate responses\. This diversity is essential for meaningful pairwise comparisons, as it ensures that differences in solution quality reflect substantive reasoning variations rather than superficial stylistic differences\.

`Prompt 1: Step\-by\-step with self\-check Prompt 2: Think\-aloud decision process Prompt 3: Socratic guiding questions Prompt 4: Decomposition into sub\-problems Prompt 5: Analogical reasoning`

`E\.6\.2 Pairwise Comparison Judge Prompt The following prompt elicits pairwise preference judgments with confidence scores\. The structured output format \(<reasoning\>, , \) enables consistent parsing across all trials and models\. Pairwise Judge Prompt: Choose the better response with confidence E\.6\.3 Verbalized Uncertainty Baseline Prompt For the verbalized uncertainty baseline , we directly elicit the model’s self\-assessed confidence\. This prompt produces a single response with an explicit confidence score, which we compare against structural and self\-consistency baselines\. Verifier Prompt: JSON\-only solution check`