Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles

# Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles
Source: [https://arxiv.org/html/2607.20768](https://arxiv.org/html/2607.20768)
###### Abstract

Majority voting over LLMs is widely assumed to benefit from diversity, and diversity measures are used to choose which models to combine\. We ask whether five such measures track diversity or mainly re\-express capability, auditing them as predictors of majority\-vote gain over the best member across 31,900 subsets of 30 LLMs on MMLU\-Pro \(29 on TruthfulQA\) under explicit capability controls\. Three findings emerge\. First, latent complementarity is ubiquitous: oracle gain is positive in 100% of subsets, yet simple voting beats the strongest member in only 9\.98% of all canonical size\-3 subsets \(18\.71% with held\-out best selection\); the pooled size\-2–4 rate is 1\.27%, partly reflecting deterministic even\-size voting behavior\. Second, a joint\-correctness proxy \(strict diversity\) is nearly collinear with one minus mean accuracy \(size\-3 Spearmanρ=\+0\.991\\rho=\+0\.991/\+0\.988\+0\.988\); raw diversity–gain associations are strongly capability\-entangled and, with one exception, unstable under control\. Third, three linear contingency\-table statistics are algebraically non\-separable; after capability control, the empirically stable remainder is a modest residual pairwise co\-failure association in which more shared error corresponds to lower gain\. This direction is robust, but its magnitude is configuration\-dependent\. Joint raw\-space linear regressions treating strict diversity, disagreement, and double\-fault as independent predictors are rank\-deficient by construction\.

Are Diversity Metrics Measuring Diversity? A Capability\-Controlled Audit of Majority\-Vote Gain in LLM Ensembles

Donghwan KimAidentyx Inc\., San Jose, CA, USAdavid\.kim@aidentyx\.com

## 1Introduction

Majority voting is one of the canonical aggregation rules in ensemble learning\(Dietterich,[2000](https://arxiv.org/html/2607.20768#bib.bib10)\), and its appeal rests on a classical intuition: if base predictors fail on different inputs, meaning they are*diverse*, a vote can outvote individual errors and exceed any single member\(Kuncheva and Whitaker,[2003](https://arxiv.org/html/2607.20768#bib.bib17)\)\. The rule is equally central in LLM practice, from self\-consistency decoding\(Wang et al\.,[2023](https://arxiv.org/html/2607.20768#bib.bib27)\)to multi\-agent sampling\-and\-voting\(Li et al\.,[2024](https://arxiv.org/html/2607.20768#bib.bib19)\), and a family of diversity measures is used, explicitly or implicitly, to reason about which models to combine\.

Two facts complicate this for modern LLMs\. LLM errors are strongly correlated, and more accurate models are, if anything,*more*correlated\(Kim et al\.,[2025](https://arxiv.org/html/2607.20768#bib.bib15); Goel et al\.,[2025](https://arxiv.org/html/2607.20768#bib.bib12)\)\. Recent work therefore balances competence and diversity through adaptive filtering\(Cohen et al\.,[2026](https://arxiv.org/html/2607.20768#bib.bib9)\), quality\-matched pools and co\-failure ceilings\(Chen,[2026](https://arxiv.org/html/2607.20768#bib.bib5)\), or accuracy\-adjusted dependence measures\(Ali,[2026](https://arxiv.org/html/2607.20768#bib.bib2)\)\. We complement this prescriptive and concurrent work with a measurement question:*for modern LLM pools, how do five diversity\-related measures behave before and after explicit capability control?*

We answer this by treating realized majority\-vote gain, defined as the vote’s accuracy minus that of the best individual member, as the dependent variable and auditing five diversity\-related measures as its predictors under explicit capability controls\. Beating the strongest available member is a deliberately stringent criterion\. It is not the only operationally relevant baseline: cost, latency, and model\-selection uncertainty all matter in deployment\(Chen et al\.,[2024a](https://arxiv.org/html/2607.20768#bib.bib6)\)\. But if an ensemble cannot outperform its strongest constituent on accuracy, diversity alone does not justify aggregation on accuracy grounds, and our held\-out selection analysis addresses the case where the strongest member is not known in advance\. Majority vote is the canonical rule being reasoned about when diversity is invoked; we audit that classical intuition rather than learned aggregators, routers, or judges\. Our findings are diagnostic and deflationary: several commonly used diversity\-related statistics are strongly entangled with capability, most severely for the three linearly coupled contingency\-table statistics, namely strict diversity, disagreement, and double\-fault\. After adjustment, the most directionally stable remainder is a modest pairwise co\-failure association, while other residual associations are specification\-dependent\.

Our contributions are:

1. 1\.A capability\-controlled audit\.We audit five diversity\-related measures as predictors of majority\-vote gain across 31,900 subsets of 30 modern LLMs on MMLU\-Pro, reproduced with 29 models on TruthfulQA, under six linear capability\-control specifications \(best; best\+mean; and four additional specifications\), together with nonlinear and matched analyses, model\-level resampling robustness analysis, and slice\- and threshold\-robustness checks\.
2. 2\.Entanglement with capability\.We show that several pairwise diversity\-related statistics are strongly entangled with member capability; in particular, our joint\-correctness proxy \(“strict diversity”\) is nearly collinear with one minus mean accuracy \(size\-3 Spearmanρ=\+0\.991\\rho=\+0\.991/\+0\.988\+0\.988\), helping explain why several counter\-intuitive raw diversity–gain associations attenuate or become specification\-dependent under capability control\.
3. 3\.A modest residual pairwise co\-failure axis, and algebraic non\-separability\.Strict diversity, disagreement, and double\-fault are algebraically non\-separable \(strict==disagreement\+\+double\-fault;1−Acc¯=1\-\\overline\{\\mathrm\{Acc\}\}=double\-fault\+12\+\\tfrac\{1\}\{2\}disagreement\), so raw\-space linear control of mean capability forces a one\-dimensional residual axis \(r=−1\.000r=\-1\.000, slope−1/2\-1/2\) and joint raw\-space linear regressions of the three are rank\-deficient\. That*raw\-space collapse is algebraic*; the*empirical*finding is that the corresponding rank\-space pairwise co\-failure signal remains associated with majority\-vote gain: more shared error corresponds to lower gain\. This direction is robust, but its magnitude is conditional\.

We are explicit about what we do not claim\. We do not claim diversity never helps, that any single statistic is uniquely “correct,” or that our residual signal is large\. The claims are measurement\-level and associational, not causal, and their magnitudes are conditional on slice and roster composition \(Section 5\.6, Limitations\)\.

## 2Related Work

2\.1 Classical ensemble diversity\.Ensemble methods classically combine base predictors by \(weighted\) voting\(Dietterich,[2000](https://arxiv.org/html/2607.20768#bib.bib10)\), with diversity long treated as a key ingredient and an explicit target of ensemble construction\(Brown et al\.,[2005](https://arxiv.org/html/2607.20768#bib.bib4)\)\. The intuition that ensemble gains reflect both member accuracy and disagreement traces back to the ambiguity decomposition of neural network ensembles\(Krogh and Vedelsby,[1995](https://arxiv.org/html/2607.20768#bib.bib16)\)\. Ensemble\-pruning work explicitly optimizes a trade\-off between classifier strength and divergence\(Zhang et al\.,[2006](https://arxiv.org/html/2607.20768#bib.bib30)\), and unified theory warns against maximizing diversity without managing the bias/variance/diversity trade\-off\(Wood et al\.,[2023](https://arxiv.org/html/2607.20768#bib.bib29)\)\. Yet measuring diversity has never been straightforward:Kuncheva and Whitaker \([2003](https://arxiv.org/html/2607.20768#bib.bib17)\)catalogued diversity measures via a pairwise contingency table and found their relationship to ensemble accuracy weak and inconsistent on real problems\. Their catalogue defines disagreement and double\-fault \(their Eqs\. 6–7\), but not\(b\+c\+d\)/N\(b\+c\+d\)/Nor Jaccard error overlap; strict diversity is our name for the complement of joint correctness\. We revisit their question for modern LLM ensembles, with realized*gain over the best member*as the target and explicit capability controls\.

2\.2 Voting and self\-consistency in LLMs\.Voting\-style aggregation is widely used with LLMs: self\-consistency decoding samples multiple reasoning paths and selects the modal answer\(Wang et al\.,[2023](https://arxiv.org/html/2607.20768#bib.bib27)\), extended to free\-form generation by universal self\-consistency\(Chen et al\.,[2024b](https://arxiv.org/html/2607.20768#bib.bib7)\), and multi\-agent studies report gains from sampling\-and\-voting as the number of instantiated agents grows\(Li et al\.,[2024](https://arxiv.org/html/2607.20768#bib.bib19); Du et al\.,[2024](https://arxiv.org/html/2607.20768#bib.bib11)\)\. These results motivate majority voting as an object of study, but they largely concern single\-model self\-sampling gains over an agent’s own baseline, not when voting over a*heterogeneous*pool beats the strongest available member\. Condorcet\-style guarantees require stronger conditions than is sometimes stated: independence of classifier outputs alone is not enough without suitable independence of their errors\(Vardeman and Morris,[2013](https://arxiv.org/html/2607.20768#bib.bib25)\)\. In financial\-sentiment experiments, heterogeneous NLP/LLM voting produced only marginal improvement, which the authors attribute to insufficient independence\(Lefort et al\.,[2024](https://arxiv.org/html/2607.20768#bib.bib18)\)\. This is task\-specific preprint evidence, not a general theorem about LLMs\. Our question is the measurement version: when majority\-vote gain over the best member is the target, do diversity\-related measures carry information beyond member capability?

2\.3 Correlated errors and model similarity\.Kim et al\. \([2025](https://arxiv.org/html/2607.20768#bib.bib15)\)show LLM errors are correlated and that stronger models are more correlated;Goel et al\. \([2025](https://arxiv.org/html/2607.20768#bib.bib12)\)find stronger models make more similar mistakes\. These works document the*empirical pattern*underlying our capability confound, in which accurate models agree and fail together, but study error correlation itself, not majority\-vote gain, and do not quantify how strongly standard diversity measures re\-express capability\. Our diagnosis builds on this pattern and makes its consequence for diversity measurement precise\.

2\.4 LLM ensemble selection, quality control, and aggregation\.Prescriptive work designs aggregation and selection rules\(Chen et al\.,[2025](https://arxiv.org/html/2607.20768#bib.bib8)\), including ranking/fusion\(Jiang et al\.,[2023](https://arxiv.org/html/2607.20768#bib.bib13); Si et al\.,[2023](https://arxiv.org/html/2607.20768#bib.bib22)\), layered proposer–aggregators\(Wang et al\.,[2025](https://arxiv.org/html/2607.20768#bib.bib26); Li et al\.,[2026](https://arxiv.org/html/2607.20768#bib.bib20)\), focal\-diversity pruning\(Tekin et al\.,[2024](https://arxiv.org/html/2607.20768#bib.bib23)\), competence filtering\(Cohen et al\.,[2026](https://arxiv.org/html/2607.20768#bib.bib9)\), higher\-order aggregation\(Ai et al\.,[2026](https://arxiv.org/html/2607.20768#bib.bib1)\), error\-distribution shaping\(Baek et al\.,[2026](https://arxiv.org/html/2607.20768#bib.bib3)\), and complementarity\- or information\-theoretic selection\(Zhang et al\.,[2026](https://arxiv.org/html/2607.20768#bib.bib31); Turkmen et al\.,[2026](https://arxiv.org/html/2607.20768#bib.bib24)\)\. Concurrent work on quality matching, all\-member co\-failure ceilings\(Chen,[2026](https://arxiv.org/html/2607.20768#bib.bib5)\), and accuracy\-adjusted dependence\(Ali,[2026](https://arxiv.org/html/2607.20768#bib.bib2)\)reinforces the need to separate dependence from capability\. Our focus is complementary and diagnostic: unweighted size 2–4 majority subsets, five diversity\-related measures under nested controls, and the algebraic non\-separability of the three linear contingency\-table statistics\.

## 3Preliminaries

### 3\.1Contingency\-table measures, two identities, and focal diversity

For a pair of models onNNitems, the standard2×22\\times 2contingency table\(Kuncheva and Whitaker,[2003](https://arxiv.org/html/2607.20768#bib.bib17)\)assigns each item to one cell by whether each model is correct:

We select five measures to span joint correctness, split errors, shared errors, normalized error overlap, and a recent LLM\-specific subset criterion\. The audit is not intended as an exhaustive catalogue of classical diversity indices\.

In particular, the Q\-statistic, the correlation coefficient, and kappa are functions of the same2×22\\times 2cell counts, andKuncheva and Whitaker \([2003](https://arxiv.org/html/2607.20768#bib.bib17)\)report that these pairwise measures are strongly intercorrelated; we therefore select representatives spanning distinct structural roles rather than near\-duplicates\.

We audit four contingency\-table measures, extended to subsets of sizekkby averaging over member pairs:

- •Strict diversity\(our term; not a named measure in Kuncheva and Whitaker’s catalogue\):SD=\(b\+c\+d\)/N\\mathrm\{SD\}=\(b\+c\+d\)/N, the complement of joint correctness\.
- •Disagreement:\(b\+c\)/N\(b\+c\)/N
- •Double\-fault \(co\-failure\):d/Nd/N
- •Mean pairwise Jaccard error\-set similarity:d/\(b\+c\+d\)d/\(b\+c\+d\), the standard Jaccard index on model error sets, averaged over pairs \(0 when both error sets are empty\)\. The mean of pairwise ratios is close to, but not identical with, the pooled ratio of sums \(max divergence 0\.089\); we retain the pairwise\-mean definition throughout\.

Two exact identities frame the analysis\. First,

strict=disagreement\+double\-fault,\\displaystyle=\\text\{disagreement\}\+\\text\{double\-fault\},max⁡\|ϵ\|\\displaystyle\\max\|\\epsilon\|<1\.9×10−16,\\displaystyle<9\\times 0^\{\-16\},whereϵ\\epsilonis the numerical verification error\. Thus, strict, disagreement, and double\-fault are linearly dependent, both pairwise and after pairwise averaging\. At the pair level, Jaccard is the ratiodouble​\-​fault/strict\\mathrm\{double\\text\{\-\}fault\}/\\mathrm\{strict\}\. Because the subset statistic averages pairwise Jaccard ratios, however, it is not algebraically determined by the subset\-level averaged double\-fault and strict measures \(the maximum observed divergence from their ratio is 0\.089\)\. Second, withAcc¯=\(2​a\+b\+c\)/\(2​N\)\\overline\{\\mathrm\{Acc\}\}=\(2a\+b\+c\)/\(2N\)the pair mean accuracy,

1−Acc¯=DoubleFault\+12​Disagreement,1\-\\overline\{\\mathrm\{Acc\}\}=\\mathrm\{DoubleFault\}\+\\tfrac\{1\}\{2\}\\mathrm\{Disagreement\},an exact identity preserved under pairwise averaging to the subset level\. In raw \(unranked\) measure space, any linear residualization that includes mean member accuracy therefore forces

DoubleFaultres=−12​Disagreementres,\\mathrm\{DoubleFault\}\_\{\\mathrm\{res\}\}=\-\\tfrac\{1\}\{2\}\\mathrm\{Disagreement\}\_\{\\mathrm\{res\}\},exactly \(Pearsonr=−1r=\-1, slope−1/2\-1/2\)\. This identity need not be preserved after the marginal rank transformations used for partial Spearman correlations, so the controlled magnitudes in Table 3 need not be symmetric\. Raw\-space one\-dimensionality after mean control is thus*algebraic*; whether the corresponding rank\-space association with majority\-vote gain is robust is*empirical*\(Section 5\.4–Section 5\.5\)\.

Separately, we audit focal diversity\(Tekin et al\.,[2024](https://arxiv.org/html/2607.20768#bib.bib23), Eq\. 10\), a subset\-level measure that is*not*a member of this algebraic family\. For a subset ofkkmodels, letFqF\_\{q\}be the number of members that fail on itemqq\. For each focal memberiithat fails at least once, we compute

P1,i\\displaystyle P\_\{1,i\}=𝔼q:i​fails​\[Fqk\],\\displaystyle=\\mathbb\{E\}\_\{q:i\\text\{ fails\}\}\\\!\\left\[\\frac\{F\_\{q\}\}\{k\}\\right\],P2,i\\displaystyle P\_\{2,i\}=𝔼q:i​fails​\[Fq​\(Fq−1\)k​\(k−1\)\],\\displaystyle=\\mathbb\{E\}\_\{q:i\\text\{ fails\}\}\\\!\\left\[\\frac\{F\_\{q\}\(F\_\{q\}\-1\)\}\{k\(k\-1\)\}\\right\],ρi\\displaystyle\\rho\_\{i\}=1−P2,iP1,i,\\displaystyle=1\-\\frac\{P\_\{2,i\}\}\{P\_\{1,i\}\},and define subset focal diversity as the mean ofρi\\rho\_\{i\}over eligible focal members \(undefined focal terms are skipped\)\. Any attenuation we report for focal is therefore an empirical finding, not an algebraic consequence of the two identities above\.

### 3\.2Majority vote, gain, and an oracle upper bound

For subsetSSof sizekk: the pipeline’s majority vote counts an item correct iff the number of correct members strictly exceedsk/2k/2\(a correct\-count rule; exact even\-size splits count as incorrect\)\. Section 5\.1 reports robustness under an answer\-level plurality rule \(most\-voted answer wins; the two rules are provably identical at sizes 2–3 and differ only at size 4\)\. WithAccMV​\(S\)\\mathrm\{Acc\}\_\{\\mathrm\{MV\}\}\(S\)andAccbest​\(S\)\\mathrm\{Acc\}\_\{\\mathrm\{best\}\}\(S\)the best member’s accuracy on the same item set:

- •Gain:g​\(S\)=AccMV​\(S\)−Accbest​\(S\)g\(S\)=\\mathrm\{Acc\}\_\{\\mathrm\{MV\}\}\(S\)\-\\mathrm\{Acc\}\_\{\\mathrm\{best\}\}\(S\)\.
- •Oracle upper bound:the gain of a hypothetical aggregator that is correct whenever*any*member is\. This is not a real aggregator and is used only as an upper bound on latent complementarity\.

Under the correct\-count rule, size\-2 majority vote can never exceed the best member \(MV correct requires both correct\), so size 2 is excluded from gain\-rate headlines\. Unless stated otherwise, “best” is selected in\-sample; Section 5\.1 quantifies held\-out selection\. All associations are between a measure andg​\(S\)g\(S\)across subsets\.

## 4Experimental Setup

![Refer to caption](https://arxiv.org/html/2607.20768v1/fig1_pipeline_overview.png)Figure 1\. Capability\-controlled audit pipeline\. Top: benchmark\-specific routes, fixed filters, and common\-parse slices\. Bottom: exhaustive size\-2–4 subsets, per\-subset outcomes/diversity, capability controls, and robustness checks\.Models and fixed filters\.We ran 30 benchmark\-specific model routes via OpenRouter\. Primary MMLU\-Pro inference was collected in April 2026, with a July 2026 retry confined to previously unparsed responses; TruthfulQA was collected in the same July pipeline\. Provider selection followed OpenRouter’s default routing and was not pinned; provider metadata beyond the requested route ID was not recorded\. The canonical inclusion rule uses parse rate≥\\geq0\.90 and full\-500 accuracy in \[0\.40, 0\.92\]\. The accuracy band was fixed in the April analysis pipeline and retained unchanged across MMLU\-Pro, TruthfulQA, and the July parsing retry; its endpoints are a pragmatic construction choice, not a literature\-derived or formally preregistered standard\. The retry revisited only previously unparsed responses \(larger token budgets, a reasoning\-field fallback, simplified answer prompts\)\. After retry, all 30 MMLU\-Pro models pass both filters, so the accuracy band is non\-binding there\. We form all size 2–4 subsets:435\+4,060\+27,405=31,900435\+4\{,\}060\+27\{,\}405=31\{,\}900\. Full OpenRouter IDs, parse rates, full\-sample accuracies, and the one benchmark\-specific route\-name discrepancy appear in Appendix A\.1\.

Evaluation protocol\.We use zero\-shot multiple\-choice prompts that request a final option label\. Generation and parsing details, including the exact benchmark prompt templates and retry protocol, are provided in Appendix A\.2\.

The common parsed slice\.Items are sampled uniformly at random \(no category stratification; seed = 42\) from the MMLU\-Pro test split\(Wang et al\.,[2024](https://arxiv.org/html/2607.20768#bib.bib28)\)\(12,032 items\); the sampled category distribution does not differ significantly from the full split \(χ2\\chi^\{2\}\(13\) = 19\.9, p = 0\.10\)\. The larger distributional shift documented below therefore arises primarily after the parse filter rather than from obvious category imbalance in the initial sample: restricting to the intersection of items successfully parsed for all 30 models after the benchmark\-specific parsing and retry pipeline reduces the item set from 500 to 356 questions, and this filtering is not neutral\. Table 1 computes each item’s accuracy and disagreement over the models that parsed that item, then averages across items; unparsed responses are excluded from these descriptive per\-item denominators rather than scored as incorrect\.

Table 1\. Effect of common\-parse filtering on the MMLU\-Pro evaluation slice\.Domain composition differs between retained and dropped items \(χ2​\(13\)=61\.65\\chi^\{2\}\(13\)=61\.65,p=2\.7×10−8p=2\.7\\times 10^\{\-8\}\)\.

Headline numbers are computed on this slice; Section 5\.6 shows that the direction of the core capability\-entanglement and co\-failure results is unchanged under per\-subset item denominators \(median 461 items at size 3 and 452 at size 4\) and a less\-filtered 451\-item slice, with residual caveats in Limitations\.

Pipeline audit and release plan\.Gain computations are sensitive to denominator consistency \(ensemble and best\-member accuracies must use the same item set\)\. We audited every gain\-computing script in the codebase and regenerated all reported numbers from a single consistent pipeline; regenerated values match the analysis files to rounding \(max \|diff\|≤\\leq5×\\times10−510^\{\-5\}over all 31,900 rows\), and the gain\>0 base rate \(1\.27%\) and best\-member accuracies reproduce exactly\. The planned public release includes the audit and generation scripts plus the derived binary correctness and parse\-status matrices \(model×\\timesitem\), item IDs, and the model roster, which suffice to reproduce the core results even if redistribution constraints prevent releasing raw model responses; raw prediction caches remain conditional on redistribution review\.

Cross\-benchmark reproduction and threshold sensitivity\.We repeat the core analysis on TruthfulQA\(Lin et al\.,[2022](https://arxiv.org/html/2607.20768#bib.bib21)\)\(29 models; 338 common items; 27,811 subsets\)\.qwen/qwen3\.6\-plusalone is excluded by the historical 0\.92 upper bound \(full\-500 accuracy 0\.922\)\. In a post\-hoc sensitivity analysis whose protocol was fixed before execution, we remove the upper cutoff and admit all 30 models on the same 338\-item denominator: gain\>0 = 0\.98% \(canonical 1\.09%\); size\-3 controlled strict/disagreement/double\-fault = \+0\.397/\+0\.313/−\-0\.570 \(canonical \+0\.378/\+0\.297/−\-0\.553\); the double\-fault model\-resampling interval remains entirely negative\. The lower 0\.40 bound is non\-binding in both retried datasets\.

Robustness under subset dependence \(design choice\)\.The 31,900 subsets reuse the same 30 models and are highly overlapping; they are not independent observations, and subset\-level p\-values would be inflated\. We therefore do not report subset\-level significance\. Our evidentiary standard throughout is: direction, effect size, model\-level resampling intervals \(resampling models; B = 1,000; procedure in Appendix B\), and leave\-one\-model\-out stability\. Because the resampling procedure deduplicates drawn models rather than reweighting by draw multiplicity, we describe its percentile intervals as robustness intervals, not conventional cluster\-bootstrap confidence intervals \(Appendix B\)\.

Capability controls\.Six linear rank\-space specifications: \(i\) best\-member accuracy; \(ii\) best \+ mean member accuracy; and four additional sets: best\+minimum, best\+median, best\+standard deviation, and the full sorted member\-accuracy vector \(Section 5\.5\)\. Separate robustness estimators use nonlinear decile indicators for best\+mean and matching on the discrete best\-member accuracy with mean tolerance 0\.01 and minimum stratum size 8\. Benchmark item sampling uses seed 42; robustness analyses use the seeds reported with their protocols in Appendices B–C\.

## 5Results

*Estimator convention\. Table 3 reports Spearman correlations: the Raw column over the full subset population; control columns are partial Spearman \(linear residualization in rank space on the stated covariates\)\. Size\-4 counterparts are in Appendix Table C1\. Additional estimators are in Appendix C \(Figure C1, Table C2\)\.*

### 5\.1Majority voting rarely beats the best member

Oracle gain is positive in 100% of subsets on both benchmarks, yet realized majority\-vote gain is typically negative\. Under the canonical correct\-count rule with in\-sample best selection, the most interpretable rate is at size 3, where the vote beats the strongest member in 9\.98% of all canonical size\-3 ensembles \(Figure 2\); pooled over all size\-2–4 subsets the rate is 1\.27%, mixing that genuine rarity with structural zeros \(size 2 is identically 0% under the correct\-count rule; size 4 is 0% largely because exact 2–2 ties count as incorrect\)\. Selecting the best member on one half of items and evaluating gain on the other \(20 seeds\) raises the rates \(Table 2\)\.

Held\-out values are mean±\\pmSD over 20 seeds\. Size 2 is omitted: positive gain is structurally impossible under the correct\-count rule \(0\.00% in both rows\)\.

Table 2\. Percentage of ensembles with positive majority\-vote gain under in\-sample and held\-out best\-member selection\.Excluding size\-4 tie items \(recomputing best on the same reduced set\) yields 8\.50% at size 4; an answer\-plurality rule, identical at sizes 2–3 and differing only at size 4, yields 4\.52%\. TruthfulQA agrees \(1\.09%, or 0\.98% with no upper accuracy cutoff\)\. The qualitative conclusion is unchanged: latent complementarity is always present, yet simple voting converts it into a win in only 9\.98% of all canonical size\-3 subsets\. This gap between latent and realized gain is not specific to unweighted voting: in closed\-loop table recognition, iteration produces better candidates that a reference\-free LLM judge largely fails to select\(Kim,[2026](https://arxiv.org/html/2607.20768#bib.bib14)\)\.

![Refer to caption](https://arxiv.org/html/2607.20768v1/fig2_oracle_vs_realized.png)Figure 2\. Oracle opportunity versus realized majority\-vote gain, size 3\. Most points lie below y = 0 despite strictly positive oracle gain; 9\.98% lie above zero \(dashed y = x\)\. Sizes 2/4 appear in Table 2 only, as deterministic majority/tie behavior suppresses their rates\.
### 5\.2Raw measure–gain correlations are misleading

Table 3 \(“Raw”, size 3\) reports raw Spearman correlations with gain\. The pattern contradicts the classical intuition: strict diversity and disagreement are negatively associated with gain, while mean pairwise Jaccard error\-set similarity is positively associated\. Taken at face value this would suggest overlap\-based proxies carry the*wrong sign*\. The next subsections show that this reading is strongly shaped by capability\.

### 5\.3Capability explains most measure variation

Strict diversity is empirically near\-collinear with capability in this setting:

strict≈1−mean member accuracy,ρMMLU\-Pro=\+0\.991,ρTQA=\+0\.988\.\\begin\{gathered\}\\text\{strict\}\\approx 1\-\\text\{mean member accuracy\},\\\\ \\rho\_\{\\text\{MMLU\-Pro\}\}=\+0\.991,\\quad\\rho\_\{\\text\{TQA\}\}=\+0\.988\.\\end\{gathered\}
We stress which part of this is empirical\. That strict diversity equals one minus the*joint\-correct*rate is algebra; that the joint\-correct rate is*this*collinear with mean accuracy is not\. Under the same rank\-space best\+mean projection used for partial Spearman, only 1\.1% of MMLU\-Pro and 1\.5% of TruthfulQA strict\-diversity variance remains\. Under independent member errors the quantities can diverge substantially; the observed near\-collinearity is an empirical property of modern LLM pools, consistent with their strongly correlated errors\(Kim et al\.,[2025](https://arxiv.org/html/2607.20768#bib.bib15)\)\. Its practical consequence is that this joint\-correctness proxy leaves very little variation independent of capability; its controlled association with majority\-vote gain is correspondingly specification\-sensitive \(Figure 3\)\.

Figure 4 generalizes the strict\-diversity case across all five measures\. Ranked best and mean member accuracy explain 98\.9%/98\.5% of strict, 92\.2%/88\.5% of disagreement, and 85\.7%/83\.8% of double\-fault variation on MMLU\-Pro/TruthfulQA, respectively\. Entanglement is substantial but not uniform: the corresponding fractions are 56\.4%/42\.7% for Jaccard and 45\.2%/23\.0% for focal diversity\. We therefore do not claim that all five are interchangeable capability proxies\.

Under capability control, most of the apparent raw diversity–gain associations do not survive as robust standalone effects: they attenuate, reverse, or become specification\-dependent \(Figure 5\)\. The pairwise co\-failure association is the notable exception, remaining negative across specifications \(−\-0\.432 linear;−\-0\.474 nonlinear\)\. Strict diversity flips to \+0\.339 under linear best\+mean control, shrinks to \+0\.092 under nonlinear control, and vanishes in the 16\-model pool \(−\-0\.006\)\. Focal diversity, though not algebraically tied to this family, empirically follows the same pattern of attenuation and specification dependence \(\+0\.049 after control at size 3;−\-0\.138 at size 4\)\.

This sign instability is informative: a coefficient that reverses under a linear control and nearly disappears under a nonlinear one is consistent with residual capability confounding or specification sensitivity, rather than a robust standalone diversity effect\.

Table 3\. Spearman association of five measures with majority\-vote gain, ensemble size 3 \(n = 4,060\), under increasing capability control \(control columns “\| best” and “\| best\+mean”: partial Spearman, linear rank\-space residualization\)\. Robustness estimates and intervals in Appendix C \(Figure C1, Table C2\)\.†\\daggerLinear\-control positive residuals in the full pool only \(see Section 5\.3, Section 5\.6\)\. Size 2 excluded for strict \(there strict = 1−\-Acc\_MV, a tautology\)\.Size\-4 associations \(Appendix Table C1\) show the same qualitative pattern: raw signs reverse or attenuate under best\+mean control, and double\-fault remains negative \(−\-0\.380\)\.

![Refer to caption](https://arxiv.org/html/2607.20768v1/fig3_strict_vs_meanacc.png)Figure 3\. Strict diversity against one minus mean member accuracy, size 3, on MMLU\-Pro \(left\) and TruthfulQA \(right\), OLS fits, shared axes\.![Refer to caption](https://arxiv.org/html/2607.20768v1/fig4_capability_entanglement.png)Figure 4\. Capability entanglement across the five audited measures \(size 3\)\. Points show the descriptive in\-sample fraction of each measure’s rank variance accounted for by an OLS projection on ranked best and mean member accuracy, the same control space used for partial Spearman rather than a held\-out predictiveR2R^\{2\}\. Entanglement is strongest for strict diversity, disagreement, and double\-fault, and weaker for Jaccard and focal diversity\.![Refer to caption](https://arxiv.org/html/2607.20768v1/fig4_control_dotplot.png)Figure 5\. Capability control changes apparent diversity–gain associations \(size 3; values from Table 3\): raw vs\. partial Spearman controlling best and best\+mean\. Only double\-fault stays negative throughout\.
### 5\.4A residual pairwise co\-failure association remains

The contingency\-table statistics do not provide several independent linear signals\. From the exact identity in Section 3\.1,1−Acc¯=DoubleFault\+12​Disagreement1\-\\overline\{\\mathrm\{Acc\}\}=\\mathrm\{DoubleFault\}\+\\tfrac\{1\}\{2\}\\mathrm\{Disagreement\}, any linear residualization of the raw \(unranked\) measures that includes mean member accuracy*forces*

DoubleFaultres=−12​Disagreementres,\\mathrm\{DoubleFault\}\_\{\\mathrm\{res\}\}=\-\\tfrac\{1\}\{2\}\\mathrm\{Disagreement\}\_\{\\mathrm\{res\}\},exactly, hence Pearsonr=−1\.000r=\-1\.000with slope−1/2\-1/2\(max numerical error∼3×10−16\\sim 3\\times 10^\{\-16\}; Appendix B\.1\)\. This raw\-space one\-dimensionality is algebraically inevitable; it is not an empirical discovery\. Table 3 and Figure 6 instead use marginal ranks followed by linear residualization\. Rank transformation does not preserve the raw linear identity, so the partial\-Spearman magnitudes for disagreement and double\-fault need not be symmetric \(\+0\.292 vs−\-0\.432\)\. What is empirical is that rank\-residual pairwise co\-failure remains negatively associated with majority\-vote gain: more shared error, lower gain \(Figure 6\)\. A raw\-space joint regression of gain on disagreement, double\-fault, mean is rank\-deficient by construction\.

![Refer to caption](https://arxiv.org/html/2607.20768v1/fig5_partial_residual.png)Figure 6\. Residual pairwise co\-failure association \(size 3, MMLU\-Pro\): rank residuals of gain vs\. double\-fault after best\+mean control, with decile\-binned means \(TruthfulQA:ρ=−0\.553\\rho=\-0\.553, Appendix Figure C2\)\.
### 5\.5Robustness and boundary conditions

The residual pairwise co\-failure association is directionally robust but bounded: more shared error is associated with less gain \(forest plot in Appendix Figure C1; full battery in Appendix Table C2: nonlinear and matched controls, model\-level resampling intervals, leave\-one\-model\-out, per\-subset denominators, 451\-item slice, TruthfulQA\)\.

Robust in direction\.Negative under all six linear control specifications, nonlinear and matched/stratified controls, plurality voting, per\-subset denominators, a less\-filtered slice, TruthfulQA, and removal of the upper accuracy cutoff\. Model\-level resampling intervals exclude zero at size 3 and on TruthfulQA; at size 4 the MMLU\-Pro interval crosses zero while leave\-one\-model\-out estimates remain uniformly negative\.

Conditional in magnitude\.Point estimates span approximately−\-0\.18 to−\-0\.57 across slice, roster, benchmark, and threshold configurations\. Under a non\-circular difficulty split \(difficulty from 15 held\-out models; analysis on the other 15\), the association is strong on easy/medium items \(≈\\approx−\-0\.75 to−\-0\.86\) but much weaker on hard items \(−\-0\.18 to−\-0\.22\), where gain variation is limited \(Appendix Figure C3\); the medium band is small \(36–42 items\) due to ties in definer accuracy\. Magnitude is pool\-sensitive \(near zero in a restricted Chinese\-origin pool\), and held\-out prediction of rare gain\>0 events remains weak \(size\-3 pairwise co\-failure AUC 0\.597\)\. This is a diagnosis of shortfall, not a recipe for winning ensembles\. The*pairwise*co\-failure component carries the association; a strict all\-members\-wrong rate is much weaker after control, distinguishing our residual axis from the all\-member co\-failure ceiling of concurrent work\(Chen,[2026](https://arxiv.org/html/2607.20768#bib.bib5)\)\.

### 5\.6Is the parsed slice driving the results?

A natural objection is that diversity matters most on hard items that a common\-parse filter tends to drop\. Three cache\-only checks argue otherwise\. \(i\)Per\-subset denominators\(each subset on its own members’ parsed intersection; median 461/452 items at sizes 3/4\) leave results essentially unchanged: strict↔\\leftrightarrow\(1−\-mean\)ρ=\+0\.991\\rho=\+0\.991; double\-fault−\-0\.418/−\-0\.358; size\-3 gain\>0 = 8\.05%\. \(ii\) Aless\-filtered 16\-model / 451\-item sliceremoves most slice bias and replicates all three core directions \(strict↔\\leftrightarrow\(1−\-mean\)≈\\approx\+0\.996; controlled strict vanishes; double\-fault−\-0\.236/−\-0\.195\)\. \(iii\)Dropped itemsare harder and show higher disagreement, and 63\.2% fall in the hard band, where the controlled pairwise co\-failure association is substantially weaker \(approximately−\-0\.18 to−\-0\.22\)\. Thus, the common\-parse filter disproportionately removes items from a regime in which this particular signal is less informative\.

## 6Discussion

Raw correlations that seem to say “diversity hurts and overlap helps” are strongly shaped by capability: the audited measures are entangled with member accuracy and, for strict diversity, nearly collinear with it\. This is a*measurement\-level*finding, convergent with competence filtering\(Cohen et al\.,[2026](https://arxiv.org/html/2607.20768#bib.bib9)\), quality matching / all\-member co\-failure ceilings\(Chen,[2026](https://arxiv.org/html/2607.20768#bib.bib5)\), and accuracy\-adjusted dependence\(Ali,[2026](https://arxiv.org/html/2607.20768#bib.bib2)\)\.

Intervention work such as D\-FUSEr shows that*reshaping*error distributions can still improve majority vote\(Baek et al\.,[2026](https://arxiv.org/html/2607.20768#bib.bib3)\); our narrower claim is that observational diversity statistics over heterogeneous pools are substantially capability\-entangled, especially for the linearly coupled contingency\-table trio\. Progress may therefore require error\-shaping interventions or capability\-controlled measurement rather than raw diversity scores alone\.

The negative direction of the residual co\-failure association is itself unsurprising under majority voting: errors shared by multiple members directly reduce the cases a vote can recover\. Our contribution is therefore not the sign alone, but the measurement diagnosis that, after capability adjustment, this pairwise co\-failure direction is the only contingency\-table component that remains directionally stable, while several commonly interpreted diversity signals do not provide separable information\.

These results should not be read as evidence that error diversity is unimportant for aggregation; rather, the audited observational statistics do not cleanly isolate ensemble\-relevant error structure from member capability\.

Practical recommendations\.\(i\) Evaluate against the strongest member; \(ii\) control for capability level and spread, including nonlinear specifications, before crediting diversity; \(iii\) treat contingency\-table measures as algebraically coupled; \(iv\) validate selection on held\-out items \(Section 5\.1\)\.

## 7Conclusion

Across 31,900 MMLU\-Pro subsets and a 29\-model TruthfulQA reproduction, the audited measures are substantially but unevenly entangled with capability, most severely for the three linearly coupled statistics: strict diversity, disagreement, and double\-fault \(84–99% of rank variance explained\)\. Strict diversity is nearly collinear with one minus mean accuracy, whereas Jaccard and focal diversity retain substantially more residual variation\. Controlled measure–gain associations are specification\-dependent, with pairwise co\-failure providing the most directionally stable residual association: more shared error is associated with lower majority\-vote gain, although the magnitude is modest and configuration\-dependent\.

## 8Limitations

1. 1\.Slice conditionality \(mitigated, not eliminated\)\.Retained items are easier and show higher agreement than dropped ones \(Section 4\)\. Section 5\.6 shows the*direction*of the core capability\-entanglement and co\-failure results is unchanged under per\-subset denominators and a less\-filtered 451\-item slice; however, effect*magnitudes*attenuate on less\-filtered configurations and should not be extrapolated\. Separately, the 356\-item set introduces item\-sampling noise at the subset level, but this noise attenuates rather than inflates the reported association \(split\-half correction≈\\approx−\-0\.53 vs\. reported−\-0\.43\), and subsampling curves are stable from 150 items onward \(Section 5\.5\)\.
2. 2\.Modest, configuration\-dependent effect\.Point estimates span approximately−\-0\.18 to−\-0\.57 across slice, roster, benchmark, and threshold configurations; the MMLU size\-4 resampling interval includes zero \(leave\-one\-model\-out remains uniformly negative\), the association weakens sharply on hard items, and it is near zero in a restricted Chinese\-origin pool\. The linear\-control positive residuals of strict/disagreement are roster\-dependent and should not be interpreted as a robust reversal\.
3. 3\.Thin residual\.Because strict≈\\approx1−\-mean, only 1\.1%/1\.5% of its rank variance survives best\+mean control on MMLU\-Pro/TruthfulQA; estimates on this residual are inherently less stable \(mitigated via nonlinear controls and model\-level resampling intervals, but a structural limit\)\.
4. 4\.Voting\-rule and baseline conditionality\.Headline rates use a conservative correct\-count vote \(even\-size ties incorrect\) and in\-sample best selection; we report plurality\-rule and held\-out alternatives \(Section 5\.1\), under which the qualitative conclusions are unchanged\. The 1\.27% all\-subset rate mixes structural size\-2 zeros and size\-4 tie effects with the more informative size\-3 rate \(9\.98%\)\.
5. 5\.Correlational, overlapping, exploratory\.All analyses are correlational; subsets reuse 30 models \(model\-level resampling robustness analysis throughout; Appendix B\); the analysis was exploratory and iteratively refined\. The exact accuracy\-band endpoints were not formally preregistered or literature\-derived\. The no\-upper\-cutoff analysis was post hoc, with its protocol fixed before execution, and establishes local robustness to one boundary model rather than universal threshold invariance\.
6. 6\.Prompt\-regime and parsing conditionality\.All models were evaluated under a common zero\-shot chain\-of\-thought instruction\. Different prompting or reasoning regimes may alter both member capability and error dependence, so the measured diversity–capability relationships should not be assumed prompt\-invariant\. Initial parsing rules also differed slightly across collection batches; the common\-slice and alternative\-denominator analyses mitigate but do not eliminate parser\-induced missingness\. The retry also re\-queried previously unparsed cases with progressively simplified prompts, so the final correctness matrix mixes a small number of recovered responses with the initial prompting regime\. An initial\-response\-only sensitivity check is not reported: the pre\-retry intersection of items parsed by all 30 models is only 18 items, below the 150\-item stability floor used elsewhere \(Section 5\.5\)\.
7. 7\.Concurrent work and scope\.Concurrent preprints study quality\-matched pools, all\-model co\-failure ceilings, and accuracy\-adjusted pair dependence\(Chen,[2026](https://arxiv.org/html/2607.20768#bib.bib5); Ali,[2026](https://arxiv.org/html/2607.20768#bib.bib2)\)\. Our scope is narrower and diagnostic: unweighted majority voting, multiple\-choice benchmarks, and five audited measures\.

## References

- Ai et al\. \(2026\)Rui Ai, Yuqi Pan, David Simchi\-Levi, Milind Tambe, and Haifeng Xu\. 2026\.[Beyond majority voting: LLM aggregation by leveraging higher\-order information](https://arxiv.org/abs/2510.01499)\.In*Forty\-Third International Conference on Machine Learning*\.
- Ali \(2026\)Junade Ali\. 2026\.[Quantifying diversity of thought: A predictive law of weighted LLM ensemble lift](https://arxiv.org/abs/2607.17384)\.*Preprint*, arXiv:2607\.17384\.
- Baek et al\. \(2026\)David Baek, Andrew Estornell, Yichi Zhang, Muhammad Faaiz Taufiq, Jean\-Francois Ton, Jie Mei, and Tao Wang\. 2026\.[D\-FUSEr: Diverse failure, unified success via error\-distribution shaping in LLM reasoning](https://icml.cc/virtual/2026/poster/63783)\.In*Forty\-Third International Conference on Machine Learning*\.OpenReview: To2O1ed5cV\.
- Brown et al\. \(2005\)Gavin Brown, Jeremy Wyatt, Rachel Harris, and Xin Yao\. 2005\.[Diversity creation methods: a survey and categorisation](https://doi.org/10.1016/j.inffus.2004.04.004)\.*Information Fusion*, 6\(1\):5–20\.
- Chen \(2026\)Josef Chen\. 2026\.[When does combining language models help? a co\-failure ceiling on routing, voting, and mixture\-of\-agents across 67 frontier models](https://arxiv.org/abs/2606.27288)\.*Preprint*, arXiv:2606\.27288\.
- Chen et al\. \(2024a\)Lingjiao Chen, Matei Zaharia, and James Zou\. 2024a\.[FrugalGPT: How to use large language models while reducing cost and improving performance](https://arxiv.org/abs/2305.05176)\.*Transactions on Machine Learning Research*\.
- Chen et al\. \(2024b\)Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou\. 2024b\.[Universal self\-consistency for large language models](https://openreview.net/forum?id=LjsjHF7nAN)\.In*ICML 2024 Workshop on In\-Context Learning*\.Earlier version titled “Universal Self\-Consistency for Large Language Model Generation”, arXiv:2311\.17311, 2023\.
- Chen et al\. \(2025\)Zhijun Chen, Xiaodong Lu, Jingzheng Li, Pengpeng Chen, Zhuoran Li, Kai Sun, Yuankai Luo, Qianren Mao, Ming Li, Likang Xiao, Dingqi Yang, Xiao Huang, Yikun Ban, Hailong Sun, and Philip S\. Yu\. 2025\.[Harnessing multiple large language models: A survey on LLM ensemble](https://arxiv.org/abs/2502.18036)\.*Preprint*, arXiv:2502\.18036\.Version 6, updated 22 April 2026\.
- Cohen et al\. \(2026\)Seffi Cohen, Nurit Cohen Inger, Niv Goldshlager, Bracha Shapira, and Lior Rokach\. 2026\.[DFPE: A diverse fingerprint ensemble for enhancing LLM performance](https://doi.org/10.18653/v1/2026.findings-eacl.282)\.In*Findings of the Association for Computational Linguistics: EACL 2026*, pages 5326–5336, Rabat, Morocco\. Association for Computational Linguistics\.
- Dietterich \(2000\)Thomas G\. Dietterich\. 2000\.[Ensemble methods in machine learning](https://doi.org/10.1007/3-540-45014-9_1)\.In*Multiple Classifier Systems*, volume 1857 of*Lecture Notes in Computer Science*, pages 1–15, Berlin, Heidelberg\. Springer\.
- Du et al\. \(2024\)Yilun Du, Shuang Li, Antonio Torralba, Joshua B\. Tenenbaum, and Igor Mordatch\. 2024\.[Improving factuality and reasoning in language models through multiagent debate](https://proceedings.mlr.press/v235/du24e.html)\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 11733–11763\. PMLR\.
- Goel et al\. \(2025\)Shashwat Goel, Joschka Strüber, Ilze Amanda Auzina, Karuna K Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, and Jonas Geiping\. 2025\.[Great models think alike and this undermines AI oversight](https://arxiv.org/abs/2502.04313)\.In*Proceedings of the 42nd International Conference on Machine Learning*, volume 267 of*Proceedings of Machine Learning Research*, pages 19621–19678\. PMLR\.
- Jiang et al\. \(2023\)Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin\. 2023\.[LLM\-Blender: Ensembling large language models with pairwise ranking and generative fusion](https://doi.org/10.18653/v1/2023.acl-long.792)\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 14165–14178, Toronto, Canada\. Association for Computational Linguistics\.
- Kim \(2026\)Donghwan Kim\. 2026\.[LLM\-as\-a\-judge scores are unreliable optimization signals in closed\-loop table recognition](https://arxiv.org/abs/2607.13347)\.*Preprint*, arXiv:2607\.13347\.
- Kim et al\. \(2025\)Elliot Myunghoon Kim, Avi Garg, Kenny Peng, and Nikhil Garg\. 2025\.[Correlated errors in large language models](https://arxiv.org/abs/2506.07962)\.In*Proceedings of the 42nd International Conference on Machine Learning*, volume 267 of*Proceedings of Machine Learning Research*, pages 30038–30066\. PMLR\.
- Krogh and Vedelsby \(1995\)Anders Krogh and Jesper Vedelsby\. 1995\.[Neural network ensembles, cross validation, and active learning](https://proceedings.neurips.cc/paper/1994/hash/b8c37e33defde51cf91e1e03e51657da-Abstract.html)\.In*Advances in Neural Information Processing Systems 7*, pages 231–238\. MIT Press\.Proceedings of the 1994 conference\.
- Kuncheva and Whitaker \(2003\)Ludmila I\. Kuncheva and Christopher J\. Whitaker\. 2003\.[Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy](https://doi.org/10.1023/A:1022859003006)\.*Machine Learning*, 51\(2\):181–207\.
- Lefort et al\. \(2024\)Baptiste Lefort, Eric Benhamou, Jean\-Jacques Ohana, Béatrice Guez, David Saltiel, and Thomas Jacquot\. 2024\.[Examining independence in ensemble sentiment analysis: A study on the limits of large language models using the condorcet jury theorem](https://doi.org/10.48550/arXiv.2409.00094)\.*Preprint*, arXiv:2409\.00094\.
- Li et al\. \(2024\)Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye\. 2024\.[More agents is all you need](https://arxiv.org/abs/2402.05120)\.*Transactions on Machine Learning Research*\.
- Li et al\. \(2026\)Wenzhe Li, Yong Lin, Mengzhou Xia, and Chi Jin\. 2026\.[Rethinking mixture\-of\-agents: Is mixing different large language models beneficial?](https://arxiv.org/abs/2502.00674)*Transactions on Machine Learning Research*\.
- Lin et al\. \(2022\)Stephanie Lin, Jacob Hilton, and Owain Evans\. 2022\.[TruthfulQA: Measuring how models mimic human falsehoods](https://doi.org/10.18653/v1/2022.acl-long.229)\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 3214–3252, Dublin, Ireland\. Association for Computational Linguistics\.
- Si et al\. \(2023\)Chenglei Si, Weijia Shi, Chen Zhao, Luke Zettlemoyer, and Jordan Boyd\-Graber\. 2023\.[Getting MoRE out of mixture of language model reasoning experts](https://doi.org/10.18653/v1/2023.findings-emnlp.552)\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 8234–8249, Singapore\. Association for Computational Linguistics\.
- Tekin et al\. \(2024\)Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang, Sihao Hu, and Ling Liu\. 2024\.[LLM\-TOPLA: Efficient LLM ensemble by maximising diversity](https://doi.org/10.18653/v1/2024.findings-emnlp.698)\.In*Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 11951–11966, Miami, Florida, USA\. Association for Computational Linguistics\.
- Turkmen et al\. \(2026\)Yigit Turkmen, Baturalp Buyukates, and Melih Bastopcu\. 2026\.[Don’t always pick the highest\-performing model: An information theoretic view of LLM ensemble selection](https://arxiv.org/abs/2602.08003)\.*Preprint*, arXiv:2602\.08003\.
- Vardeman and Morris \(2013\)Stephen B\. Vardeman and Max D\. Morris\. 2013\.[Majority voting by independent classifiers can increase error rates](https://doi.org/10.1080/00031305.2013.778788)\.*The American Statistician*, 67\(2\):94–96\.
- Wang et al\. \(2025\)Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou\. 2025\.[Mixture\-of\-agents enhances large language model capabilities](https://arxiv.org/abs/2406.04692)\.In*International Conference on Learning Representations*\.Spotlight\.
- Wang et al\. \(2023\)Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V\. Le, Ed H\. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou\. 2023\.[Self\-consistency improves chain of thought reasoning in language models](https://arxiv.org/abs/2203.11171)\.In*International Conference on Learning Representations*\.
- Wang et al\. \(2024\)Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen\. 2024\.[MMLU\-Pro: A more robust and challenging multi\-task language understanding benchmark](https://arxiv.org/abs/2406.01574)\.In*Advances in Neural Information Processing Systems 37: Datasets and Benchmarks Track*\.
- Wood et al\. \(2023\)Danny Wood, Tingting Mu, Andrew M\. Webb, Henry W\. J\. Reeve, Mikel Luján, and Gavin Brown\. 2023\.[A unified theory of diversity in ensemble learning](https://jmlr.org/papers/v24/23-0041.html)\.*Journal of Machine Learning Research*, 24\(359\):1–49\.
- Zhang et al\. \(2006\)Yi Zhang, Samuel Burer, and W\. Nick Street\. 2006\.[Ensemble pruning via semi\-definite programming](https://jmlr.org/papers/v7/zhang06a.html)\.*Journal of Machine Learning Research*, 7\(48\):1315–1338\.
- Zhang et al\. \(2026\)Yichi Zhang, Kevin Lu, Yuang Zhang, Jie Gao, Lirong Xia, and Fang\-Yi Yu\. 2026\.[Mixture of complementary agents for robust LLM ensemble](https://arxiv.org/abs/2605.24048)\.*Preprint*, arXiv:2605\.24048\.Associated ICLR 2026 submission withdrawn\.

## Appendix AModel roster and parsing audit

### A\.1Model roster and full\-sample filtering statistics

Parse rate and accuracy use the full 500\-item sample\. Full\-500 accuracy is the number of correct predictions divided by 500; unparsed responses therefore contribute zero to accuracy\. Canonical inclusion requires parse rate≥\\geq0\.90 and full\-500 accuracy in \[0\.40, 0\.92\]\.

MMLU\-Pro:30/30 routes pass the canonical filter\.

Table A1\. MMLU\-Pro model roster and full\-sample filtering statistics\.TruthfulQA:29/30 routes pass the canonical filter\.

Table A2\. TruthfulQA model roster and full\-sample filtering statistics\.Route\-provenance note\.The benchmark files each contain 30 aliases, but the recorded Haiku alias/slug differs: MMLU\-Pro usesclaude\_haiku→\\rightarrowanthropic/claude\-haiku\-4\.5, whereas TruthfulQA usesclaude\_haiku45→\\rightarrowanthropic/claude\-haiku\-4\-5\. The table preserves the recorded benchmark\-specific identifiers and does not silently assume that differently written routes are identical\. On TruthfulQA,qwen36\_pluspasses parsing but is excluded only because its full\-500 accuracy is 92\.2%, above the historical 92% upper bound\.

### A\.2Prompting, inference, and parsing protocol

Benchmarks and sampling\.MMLU\-Pro is loaded fromTIGER\-Lab/MMLU\-Pro\(test split; 12,032 items\)\. We sample 500 row indices uniformly without replacement using Python’srandom\.sampleafter setting seed 42; no category stratification is used\. TruthfulQA is loaded fromtruthfulqa/truthful\_qa\(multiple\_choiceconfiguration, validation split\), from which 500 items are sampled by the same procedure and seed\. We use themc1\_targetschoices and the unique choice labeled 1 as the TruthfulQA answer\. The released sampled\-question files store the original dataset row index, local question ID, question text, options, and answer label; all 500 TruthfulQA mappings were verified against the Hugging Face validation\-row order\.

Initial prompt templates\.Both benchmarks use zero\-shot chain\-of\-thought multiple\-choice prompts with no demonstrations\.No system prompt was used:every request contained one user message only\. In the templates below,questionis replaced by the benchmark question andlettered\_optionsby one option per line inA\. option text,B\. option text, … format\. Individual benchmark questions are not part of the template\.

MMLU\-Pro:

> Answer the following multiple choice question\. Think step by step, then provide your final answer\. At the very end, write your answer in this exact format: ANSWER: X \(where X is the letter of the correct option\) Question: \{question\} Options: \{lettered\_options\} Let’s think step by step:

TruthfulQA:

> Answer the following question truthfully\. Think step by step, then provide your final answer\. At the very end, write your answer in this exact format: ANSWER: X \(where X is the letter of the correct option\) Question: \{question\} Options: \{lettered\_options\} Let’s think step by step:

Table A3\. Inference and parsing settings\.The five MMLU\-Pro routes assigned 8,192 initial output tokens weredeepseek\_r1,kimi\_k2,gemini25\_pro,gemini25\_flash, andgemini25\_flash\_lite\.

Initial parser\.

Matching ignores case\. Labels outside the available option range are rejected\. Patterns run in order, using the last match from each\.

MMLU\-Pro, original eight\-route batch\.In order:ANSWER: X,answer: X,The answer is X,Final answer: X, and a response consisting only of one label\.

MMLU\-Pro, later scheduler\.First remove complete<think\>\.\.\.</think\>blocks; then checkANSWER: X,answer: X, boxed labels, boldAnswer: X, bold labels,The \[correct\] answer is X, the Chinese prefixes encoded by U\+9009/U\+62E9 \(“selection”\), U\+7B54/U\+6848 \(“answer”\), or U\+6B63/U\+786E/U\+7B54/U\+6848 \(“correct answer”\), a line beginning withX\.orX\), and a response consisting only of one label\.

TruthfulQA\.First remove complete<think\>\.\.\.</think\>blocks; then checkANSWER: X,answer: X, boxed labels, bold labels,The \[correct\] answer is X, and a response consisting only of one label\. If no valid label is found, the record is stored with no prediction andparse\_success=False; it is not scored as a parsed prediction\.

Exact retry prompts and sequence\.The retry pass uses the following three benchmark\-agnostic templates, in order, for both benchmarks\. Options in Retry 1–2 useA\. option text; Retry 3 uses\(A\) option text\.

Retry 1:

> Answer the following multiple choice question\. Think step by step, then provide your final answer\. At the very end, write your answer in this exact format: ANSWER: X \(where X is the letter of the correct option\) Question: \{question\} Options: \{lettered\_options\} Let’s think step by step:

Retry 2:

> Question: \{question\} \{lettered\_options\} Which option is correct? Reply with just the letter\. Your answer \(one letter only\):

Retry 3:

> Select the correct answer\. \{question\} \{parenthesized\_options\} Answer with only one letter like: A

For each retry prompt, we first parse responsecontentusing this ordered cascade:ANSWER: X,answer: X,Answer: X, boldX,The answer is X,correct answer is X, a standaloneX, and a final valid option letter\. If content parsing fails, the same parser is applied to the providerreasoningfield\. A successful parse stops the sequence immediately; otherwise the next prompt is tried\. After all three prompts fail, the entry remains unparsed\.

Retry scope and preservation guarantee\.Retries were applied only to previously unparsed responses \(parse\_success=False\) from routes below the parse\-rate threshold\. Already parsed predictions were never regenerated, re\-queried, or replaced\. The merge changes only a targeted unparsed record after a successful retry and carries every previously parsed record forward unchanged\. The releasedretry\_parsing\.pyimplements this target selection, three\-prompt schedule, parser fallback, and merge logic and provides a no\-API\-\-dry\-runaudit\.

Voting rules\.The canonical correct\-count rule marks an item correct only when more than half the subset members are correct; exact even\-size splits are incorrect\. The answer\-level plurality robustness check requires the correct option to be the unique most\-voted answer; any tie for the highest answer count is marked incorrect\. These rules are identical for subset sizes 2–3 and can differ at size 4\.

## Appendix BModel\-level resampling procedure

Subsets heavily overlap because they reuse the same 30 models\. We therefore avoid subset\-level p\-values and use model\-level resampling for robustness intervals on partial Spearman associations \(B = 1,000; NumPy seed 0, using one RNG stream across sizes 3 and 4 within each benchmark\)\. Pseudocode:

1. 1\.Draw a multiset of model indices of sizeMMuniformly with replacement\.
2. 2\.Deduplicate to a setUUof unique drawn models\. If\|U\|<8\|U\|<8, discard the replicate\.
3. 3\.Retain only subsets whose members are all inUU\(subsets that use any non\-drawn model are dropped\)\. If fewer than 50 subsets remain, discard the replicate\.
4. 4\.On the retained subsets, recompute the partial Spearman of double\-fault with gain given best, mean \(rank\-space residualization, same as the main tables\)\.
5. 5\.The 95% interval is the 2\.5th–97\.5th percentile of valid replicate statistics\.

Because step 2 deduplicates draws, duplicate multiplicities do not reweight the statistic; the procedure is a random\-subpool perturbation of the model roster rather than a conventional cluster bootstrap, and we accordingly report its percentile intervals as model\-level resampling robustness intervals, not bootstrap confidence intervals\. Leave\-one\-model\-out estimates drop all subsets containing a held\-out model and recompute the same partial association\.

### B\.1Residual identity verification

Numerically, using the raw \(unranked\) measures,max⁡\|double​\-​faultres\+12​disagreeres\|\\max\|\\mathrm\{double\\text\{\-\}fault\}\_\{\\mathrm\{res\}\}\+\\tfrac\{1\}\{2\}\\mathrm\{disagree\}\_\{\\mathrm\{res\}\}\|is at machine precision \(∼3×10−16\\sim 3\\times 10^\{\-16\}\) after linearly removing best and mean member accuracy at size 3, matching the algebraic prediction of Section 3\.1: the residuals are perfectly collinear \(Pearsonr=−1\.000r=\-1\.000\) with slope−1/2\-1/2\. Table 3 and Figure 6 use rank\-space residualization instead; the empirical content of Section 5\.4 is the resulting association with majority\-vote gain, not the raw\-space collinearity itself\.

## Appendix CFull robustness battery

This appendix collects the size\-4 counterpart to main\-text Table 3 and the full robustness battery for the controlled double\-fault association with majority\-vote gain\.

Figure C1 summarizes point estimates \(and, where available, model\-level resampling intervals and leave\-one\-model\-out ranges\) across control, denominator, slice, and benchmark specifications at sizes 3 and 4\. Figure C2 is the TruthfulQA partial\-residual counterpart to main\-text Figure 6\. Figure C3 shows the non\-circular difficulty boundary condition without aggregating its two model splits into an uncertainty interval\. Figure C4 decomposes the gain\>0 rate by ensemble size and voting or selection rule\.

Table C1 is the size\-4 analogue of main\-text Table 3: Spearman associations of the five measures with majority\-vote gain under the Raw→\\to\|\|best→\\to\|\|best\+mean control ladder \(n=27,405n=27\{,\}405\)\. Table C2 lists the numerical values behind Figure C1 and the remaining robustness checks referenced in Sections 5\.4–5\.6 \(alternative controls, resampling intervals, leave\-one\-model\-out, denominators, slice, and TruthfulQA\)\.

![[Uncaptioned image]](https://arxiv.org/html/2607.20768v1/fig6_forest.png)

Figure C1\. Controlled double\-fault association across specifications \(size 3 circles, size 4 triangles\); bars show the MMLU\-Pro model\-resampling interval and LOMO range\.

![[Uncaptioned image]](https://arxiv.org/html/2607.20768v1/figC2_partial_residual_tqa.png)

Figure C2\. TruthfulQA partial\-residual plot \(size 3;ρ=−0\.553\\rho=\-0\.553\), counterpart to Figure 6\.

Table C1\. Size\-4 associations \(counterpart to main\-text Table 3\), ensemble size 4 \(n = 27,405\)\.Table C2\. Full robustness of the co\-failure \(double\-fault\) association under best\+mean control unless noted\.At size 4 under answer\-plurality voting, the Raw / \|best / \|best\+mean double\-fault associations are−\-0\.377 /−\-0\.781 /−\-0\.456; sizes 2–3 plurality coincides with the correct\-count rule and therefore matches Table 3\.

### C\.1Additional robustness definitions

Table C3\. Protocols and numerical results for additional robustness checks cited in Sections 5\.5 and 8\.![Refer to caption](https://arxiv.org/html/2607.20768v1/figC3_difficulty_cofailure.png)

Figure C3\. Difficulty\-conditioned controlled pairwise co\-failure association on MMLU\-Pro, size 3\. For each of the two reported split seeds \(0 and 1\), 15 models define item\-difficulty terciles and the disjoint 15 form 455 evaluated subsets; points are seed\-specific partial Spearman coefficients after rank\-space best\+mean control\. Ties yield unequal easy/medium/hard bands: n = 190/36/130 items for seed 0 and 194/42/120 for seed 1\. Lines connect bands within a split only; with two splits, no uncertainty interval is estimated\.![Refer to caption](https://arxiv.org/html/2607.20768v1/figC4_gainrate_decomposition.png)

Figure C4\. Decomposition of the gain\>0 rate by ensemble size and rule\. The pooled 1\.27% mixes structural size\-2 zeros and size\-4 tie effects with the size\-3 rate\. Held\-out best selection raises the observed win rate at sizes 3–4, while size 2 remains structurally zero\.Control and slice estimators\.The six linear control specifications in Table C2 are \(best\), \(best, mean\), \(best, minimum\), \(best, median\), \(best, standard deviation\), and the full sorted member\-accuracy vector; all use the rank\-space partial\-Spearman procedure defined in Section 5\. The nonlinear specification replaces continuous rank controls for best and mean with decile indicators\. The matched estimator exactly matches the discrete best\-member accuracy, bins mean accuracy at width 0\.01, requires at least eight subsets per stratum, computes within\-stratum Spearman correlations, and averages them weighted by stratum size\. Matching on best\+mean retains 3,551/4,060 size\-3 subsets across 134 strata \(87\.5%\) and 27,137/27,405 size\-4 subsets across 230 strata \(99\.0%\)\. Per\-subset\-denominator results recompute every subset on the intersection parsed by its own members\. The 451\-item slice greedily removes low\-parse routes until a common set of at least 450 items is available, leaving 16 models and 451 items\. TruthfulQA repeats the same filtering, subset construction, and estimator on its 338\-item common denominator\. The no\-upper\-cutoff intervals come from the separately initialized deterministic RNGs in the pre\-fixed threshold\-sensitivity protocol\.

Held\-out best selection\.For each of 20 seeds \(0–19\), the 356 common items are randomly permuted and split into halves of 178 items each \(A/B; remainder none\)\. Within each subset, the comparison member is the model with maximum accuracy on split A; exact accuracy ties are broken by the lowest model index in the filtered roster order \(numpy\.argmax\)\. Majority\-vote accuracy and the selected member’s accuracy are then evaluated exclusively on split B\. The reported 18\.71% \(±\\pm3\.70\) is the mean \(±\\pmSD\) over seeds of the fraction of size\-3 subsets with positive held\-out gain\.

The difficulty result above is deliberately non\-circular: the models defining item difficulty never enter the evaluated subsets\. Because ties in definer accuracy make the three item bands unequal, we report the two seed\-specific coefficients rather than pooling them\. The held\-out AUC is a predictive diagnostic, not a capability\-controlled effect estimate\. The attenuation\-corrected value is also approximate: the classical correction applies directly to a raw correlation, whereas applying it to the partial coefficient additionally treats the best and mean controls as measured with negligible error\. We therefore use it only to assess whether item noise plausibly attenuates, rather than creates, the observed association\.

Oracle\-gain capture ratio\.For completeness, define this descriptive ratio asg​\(S\)/goracle​\(S\)g\(S\)/g\_\{\\mathrm\{oracle\}\}\(S\)on subsets with positive oracle gain\. Its MMLU\-Pro mean across subset\-level ratios is−\-131\.7%, while the pooled ratio∑Sg​\(S\)/∑Sgoracle​\(S\)\\sum\_\{S\}g\(S\)/\\sum\_\{S\}g\_\{\\mathrm\{oracle\}\}\(S\)is−\-120\.6%\. These negative values reflect typical underperformance relative to the best member, but the per\-subset ratio can be unstable when oracle gain is small; we therefore do not use it as a headline result\.