Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps

# Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps
Source: [https://arxiv.org/html/2608.06940](https://arxiv.org/html/2608.06940)
\(Preprint — August 2026\)

###### Abstract

LLM judge panels are a standard evaluation tool, but prior work reports highly correlated panel errors: nine judges provide roughly the effective information of two independent ones, and aggregation closes only a small fraction of the gap\. A natural remedy – a signal from a different evidence source, e\.g\. executing a test suite – produced no distinguishable change in the panel’s effective\-vote count at scale \(−0\.04\-0\.04, 95% CI\[−0\.10,\+0\.02\]\[\-0\.10,\+0\.02\]\)\. Aggregate dependence and conditional decision utility are different questions\. Elementary majority arithmetic fixes the affected set for single\-ballot substitution: only decisions with a one\-vote margin can change\. The empirical question is whether panel error rates rise and useful substitutions concentrate there\. They do: the entire accuracy gain concentrates on these*pivotal*queries, where it is large \(\+10\.4 to \+23\.3pp across three headline configurations\), and is exactly zero elsewhere\. We confirm the pattern across three code benchmarks and four panel sizes \(a 9\-judge extension and 56 dependent subsampling checks, gain \+6\.5 to \+16\.1pp\)\. On HumanEval\+/MBPP\+, a majority\-side replacement rule raises overall accuracy from 82\.44% to 85\.62% while invoking the signal on 16\.2% of queries; signal\-only remains stronger at 87\.60%\. Thus population\-level dependence diagnostics and margin\-stratified utility are complementary, and the affected\-set characterization yields a call\-reduction rule for any specified single\-ballot substitution policy\.

## 1Introduction

Multi\-model LLM judge panels are increasingly used to evaluate model outputs at scale, on the intuition that pooling several independent judgments should be more reliable than trusting any single one\.Kohli \([2026](https://arxiv.org/html/2608.06940#bib.bib1)\)recently complicated this intuition: across nine frontier judges from seven model families, panel errors are so correlated that the panel carries roughly the statistical information of two truly independent judges, and even an oracle\-calibrated aggregation rule closes at most about eleven percent of the resulting reliability gap\. Their result suggests that simply adding more LLM judges can offer diminishing returns when the models share blind spots\.

A natural response is to add a vote derived from another procedure\. In code generation, a partial unit\-test execution can supply an imperfect pass/fail ballot\. Following[Kohli](https://arxiv.org/html/2608.06940#bib.bib1)’s effective\-votes methodology, seven LLM judges blindly score candidate solutions and the verification signal replaces one judge\. A pilot suggested an effective\-vote increase, but its 98%\-accurate signal had only five errors from one solver and could not support a general conclusion\. At ten times the scale, a less accurate signal produced no distinguishable change in effective votes \(95% CI crossing zero\), although panel accuracy still improved\.

The improvement becomes interpretable after stratifying queries by vote margin\. When at least five of seven judges agree, single\-ballot substitution changes zero decisions and gains 0\.0000 accuracy\. On 4–3 votes, the same signal improves accuracy by more than ten percentage points\. The boundary follows from majority arithmetic: replacing one voter can change the vote total by at most one, so it can flip the outcome only when the original tally is one vote from a tie\. We call such a query*pivotal*\. The empirical claim is not that most panel errors occur there – only 199 of 569 errors do in the main setting – but that panel error*rates*rise as the margin narrows and all decision changes and accuracy gains from single\-ballot substitution are confined there\.

This has two consequences\. First, an aggregate statistic like the effective\-vote count averages a population that is 84% unaffected by substitution and 16% strongly affected; the average can therefore obscure a reliable conditional effect\. Second, once a substitution rule has been specified, verification can be skipped outside the pivotal region without changing that rule’s predictions\. This reduces verifier\-call frequency to 12–27% in our experiments\. It does not establish a total\-system cost advantage, and universal execution is already inexpensive for the test\-based signal studied here\. Figure[1](https://arxiv.org/html/2608.06940#S1.F1)summarizes this aggregate\-to\-pivotal\-to\-gating logic\.

Our contributions are: \(1\) we separate a structural statement from an empirical one: elementary majority\-vote arithmetic identifies the exact affected set for single\-ballot substitution \(Propositions 1–2\), while measurements across three code benchmarks and four panel sizes show that error rates rise and substitution gains concentrate there; \(2\) using the sameneffn\_\{\\mathrm\{eff\}\}statistic asKohli \([2026](https://arxiv.org/html/2608.06940#bib.bib1)\), we show that population\-level dependence and margin\-conditional substitution utility are complementary diagnostics on the same data \(Table[4](https://arxiv.org/html/2608.06940#S5.T4)\); and \(3\) we evaluate margin gating with uniformly random, fixed, and majority\-side replacement rules, showing that gating preserves each rule’s universal\-substitution predictions while the replacement rule materially affects accuracy \(Table[7](https://arxiv.org/html/2608.06940#S6.T7)\)\. A secondary, exploratory analysis associates part of the pivotal\-region error pattern with panel composition \(Section 5\)\.

![Refer to caption](https://arxiv.org/html/2608.06940v1/figures/fig0_story_overview_selected_v3.png)Figure 1:Overview of the aggregate blind spot and pivotal\-vote mechanism\. On HumanEval\+/MBPP\+ withk=7k\{=\}7, random replacement gains\+11\.2\+11\.2pp within the pivotal region; majority\-side gating invokes the verifier on 16\.2% of queries and improves overall accuracy from 82\.44% to 85\.62%\. Icons illustrate the mechanism; experiments span three code benchmarks andk∈\{3,5,7,9\}k\\in\\\{3,5,7,9\\\}\.
## 2Related Work

### Correlated errors in LLM judge panels\.

Kohli \([2026](https://arxiv.org/html/2608.06940#bib.bib1)\)is the closest prior work: it introduces the effective\-votes framework we build on,neff=k/\(1\+\(k−1\)​ϕ¯\)n\_\{\\mathrm\{eff\}\}=k/\(1\+\(k\-1\)\\bar\{\\phi\}\), a Kish\-style effective sample size over the mean pairwise correlationϕ¯\\bar\{\\phi\}of judges’ \(or judge\-plus\-signal’s\) binary error vectors for panel sizekk; we use it exactly as defined there\. Their diagnosis – correlated judges, not weak aggregation, is the bottleneck – motivates our search for a signal drawn from a different source of evidence, but their framework, as we show, is not designed to detect value concentrated in a small, identifiable query subset\.Kuaiet al\.\([2026](https://arxiv.org/html/2608.06940#bib.bib2)\)audit behavioral entanglement among LLMs and reweight verifier ensembles, but only among LLMs, not non\-LLM signals orneffn\_\{\\mathrm\{eff\}\}itself\.Kwoket al\.\([2026](https://arxiv.org/html/2608.06940#bib.bib3)\)optimize a single verifier’s scoring granularity, a complementary but orthogonal problem to multi\-judge panels\.Vergaet al\.\([2024](https://arxiv.org/html/2608.06940#bib.bib14)\)propose PoLL, a panel of diverse smaller LLM judges cheaper and less self\-biased than one strong judge, motivating multi\-judge panels as we assume here, but without conditioning aggregation on vote margin\.

### Margin\-gated verification as engineering practice\.

Akinfaderin and Diallo \([2026](https://arxiv.org/html/2608.06940#bib.bib4)\)’s SemEval system already invokes a symbolic \(Z3\) verifier when a five\-model ensemble splits 3–2\. Thus narrow\-margin verification is not our algorithmic invention\. We instead isolate the aggregation\-rule boundary, test the full margin sweep, distinguish aggregate dependence from conditional utility, and compare replacement rules on substantially more panel decisions\. These contributions position our work as a systematic empirical and methodological analysis of the heuristic rather than its first use\.

### Selective computation and disagreement\-based routing\.

Our margin gate instantiates the older idea of spending a costly resource where cheap signals indicate it is needed\.Seunget al\.\([1992](https://arxiv.org/html/2608.06940#bib.bib11)\)select informative points by committee disagreement; selective prediction \(Geifman and El\-Yaniv,[2017](https://arxiv.org/html/2608.06940#bib.bib12)\) abstains on low\-confidence inputs; learning\-to\-defer \(Mozannar and Sontag,[2020](https://arxiv.org/html/2608.06940#bib.bib13)\) routes decisions to an expert; and FrugalGPT \(Chenet al\.,[2024](https://arxiv.org/html/2608.06940#bib.bib16)\) uses cost\-aware model cascades\. Our narrower contribution is to derive the exact affected set for single\-ballot substitution under equal\-weight majority aggregation and to measure the resulting conditional utility\.

### Theoretical framing\.

Work on multi\-judge and multi\-agent voting for LLMs is overwhelmingly framed in terms of the Condorcet Jury Theorem, which concerns whether independent, better\-than\-chance voters make majority voting reliable in aggregate \(Condorcet,[1785](https://arxiv.org/html/2608.06940#bib.bib6); used explicitly byKohli,[2026](https://arxiv.org/html/2608.06940#bib.bib1)and by several majority\-voting\-harm analyses in adjacent work\)\. We instead draw on the notion of a*pivotal*or decisive voter, formalized in social choice theory by the Banzhaf power index \(Banzhaf,[1965](https://arxiv.org/html/2608.06940#bib.bib5)\), which measures how often a single voter’s ballot determines the outcome of a vote\. To our knowledge this framing has not previously been applied to LLM judge or ensemble panels; it is a natural fit for our question because it concerns*which*votes can be changed by a single ballot, rather than whether voters are independent on average\.

### Verifiable\-domain benchmarks\.

We evaluate on HumanEval\+\(Liuet al\.,[2023](https://arxiv.org/html/2608.06940#bib.bib9)\), an extended\-test\-case version of HumanEval\(Chenet al\.,[2021](https://arxiv.org/html/2608.06940#bib.bib7)\); MBPP\+\(Liuet al\.,[2023](https://arxiv.org/html/2608.06940#bib.bib9)\), similarly derived from MBPP\(Austinet al\.,[2021](https://arxiv.org/html/2608.06940#bib.bib8)\); and LiveCodeBench\(Jainet al\.,[2025](https://arxiv.org/html/2608.06940#bib.bib10)\), a contamination\-resistant benchmark of recent competitive\-programming problems that we use as a harder setting than HumanEval\+/MBPP\+\.

## 3The Pivotal\-Vote Mechanism

### Setup\.

A panel ofkkjudges \(herek∈\{3,5,7,9\}k\\in\\\{3,5,7,9\\\}\) blindly scores a candidate solution as correct or incorrect; the panel’s decision is the majority vote\. For queryii, letsis\_\{i\}be the number of judges voting “correct\.” We measure the panel’s disagreement with the*margin*mi=\|2​si−k\|m\_\{i\}=\|2s\_\{i\}\-k\|:mi=km\_\{i\}=kmeans unanimity, and for oddkkthe smallest possible valuemi=1m\_\{i\}=1means the vote is decided by a single ballot \(e\.g\.⌈k/2⌉\\lceil k/2\\rceilvs\.⌊k/2⌋\\lfloor k/2\\rfloor\)\. We call a query withmi=1m\_\{i\}=1*pivotal*\.

### Why value is structurally confined to pivotal queries\.

> Proposition 1\.*Let akk\-judge panel \(kkodd\) predict the label for queryiiby unweighted majority vote, withsis\_\{i\}of thekkvotes cast “correct\.” There exists a choice of which single judge’s vote to replace with a verification signal’s vote \(correct or incorrect\) that changes the panel’s predicted label if and only ifmi=\|2​si−k\|=1m\_\{i\}=\|2s\_\{i\}\-k\|=1; whenmi≥3m\_\{i\}\\geq 3, no choice of replaced judge or replacement value can change the predicted label\.*

*Proof\.*Letsi′s\_\{i\}^\{\\prime\}be the vote total after replacement\. Since only one of thekkballots changed,si′∈\{si−1,si,si\+1\}s\_\{i\}^\{\\prime\}\\in\\\{s\_\{i\}\-1,s\_\{i\},s\_\{i\}\+1\\\}\. The panel predicts the label CORRECT iff the tally exceedsk/2k/2\. Becausekkis odd, the decision boundary lies strictly between the two integers⌊k/2⌋\\lfloor k/2\\rfloorand⌈k/2⌉=⌊k/2⌋\+1\\lceil k/2\\rceil=\\lfloor k/2\\rfloor\+1\. A change of at most one vote can movesis\_\{i\}across this boundary only ifsi∈\{⌊k/2⌋,⌈k/2⌉\}s\_\{i\}\\in\\\{\\lfloor k/2\\rfloor,\\lceil k/2\\rceil\\\}, i\.e\.mi=\|2​si−k\|=1m\_\{i\}=\|2s\_\{i\}\-k\|=1, and in that case replacing a vote on the majority side with the opposite value does cross it\. For anysis\_\{i\}withmi≥3m\_\{i\}\\geq 3,sis\_\{i\}is at least two votes from the boundary andsi′s\_\{i\}^\{\\prime\}cannot cross it regardless of which judge is replaced or what value replaces it\.■\\blacksquareNote this bounds the label the panel*predicts*; whether that prediction is*accurate*against ground truth is a separate, empirical question we test directly below\.

This is a property of majority voting under single\-ballot substitution, not a claim requiring statistical evidence: it holds by construction for*any*panel and*any*replacement signal, regardless of how accurate that signal is\. It immediately implies that the*expected accuracy gain from substitution is exactly zero on every non\-pivotal query*, since the panel’s output – and hence its correctness – cannot change there\.

### Even panels: an asymmetric extension\.

All our experiments use oddkk, where the vote total alone decides the outcome\. Evenkkrequires a tie\-breaking convention; we consider the natural one where ties default to “incorrect” \(decision=correct=\\mathrm\{correct\}iffsi\>k/2s\_\{i\}\>k/2\), and derive the analogous structure\.

> Proposition 2\.*Let akk\-judge panel \(kkeven\) decide queryiiby majority vote with ties broken as “incorrect\.” Suppose exactly one judge’s vote is replaced\. Then the panel’s decision can change if and only ifsi∈\{k/2,k/2\+1\}s\_\{i\}\\in\\\{k/2,\\ k/2\+1\\\}\.*

*Proof\.*As before,si′∈\{si−1,si,si\+1\}s\_\{i\}^\{\\prime\}\\in\\\{s\_\{i\}\-1,s\_\{i\},s\_\{i\}\+1\\\}, and decision=correct=\\mathrm\{correct\}iff the tally is≥k/2\+1\\geq k/2\+1\. The boundary lies between the integersk/2k/2\(incorrect\) andk/2\+1k/2\+1\(correct\), so a±1\\pm 1change crosses it only ifsi∈\{k/2,k/2\+1\}s\_\{i\}\\in\\\{k/2,k/2\+1\\\}\.■\\blacksquare

Unlike the odd case, this set is*not*symmetric in the marginmi=\|2​si−k\|m\_\{i\}=\|2s\_\{i\}\-k\|:si=k/2s\_\{i\}=k/2\(a tie\) hasmi=0m\_\{i\}=0, andsi=k/2\+1s\_\{i\}=k/2\+1\(the narrowest possible “correct” majority\) hasmi=2m\_\{i\}=2– butsi=k/2−1s\_\{i\}=k/2\-1, the narrowest possible*incorrect*majority, also hasmi=2m\_\{i\}=2and is*not*pivotal under this convention \(a further−1\-1vote keeps it below the boundary\)\. Margin alone is thus insufficient to identify pivotal queries oncekkis even; the*signed*tally relative tok/2k/2is needed, with the two sides of a given margin behaving asymmetrically\. We validate this directly in Section 5 by re\-analyzing the existing 7\-judge data as all\(76\)=7\\binom\{7\}\{6\}=7six\-judge \(even\) subsets, with no new data collection: the theory predicts a structural zero specifically on the incorrect\-leaning bare majority \(si=2s\_\{i\}=2fork=6k=6\), and nowhere else does the odd\-panel intuition of “same margin, same status” hold\.

### What requires evidence\.

The arithmetic identifies where a single replacement can affect decisions; it does not say how many errors occur there or whether an imperfect signal improves them\. We therefore test two empirical questions: whether panel error rate increases asmi→1m\_\{i\}\\to 1, and whether substitution atmi=1m\_\{i\}=1improves accuracy\.

### Verification signal and its asymmetry\.

For problemppwith assertion setTpT\_\{p\}and coveragecc, letSc​\(p\)⊆TpS\_\{c\}\(p\)\\subseteq T\_\{p\}contain the firstq=max⁡\(1,min⁡\(\|Tp\|,round​\(c​\|Tp\|\)\)\)q=\\max\(1,\\min\(\|T\_\{p\}\|,\\mathrm\{round\}\(c\|T\_\{p\}\|\)\)\)assertions in source order\. The signal isVc​\(x\)=𝟙​\[⋀t∈Sc​\(p\)Pass​\(x,t\)\]V\_\{c\}\(x\)=\\mathbb\{1\}\[\\bigwedge\_\{t\\in S\_\{c\}\(p\)\}\\mathrm\{Pass\}\(x,t\)\]\. We usec∈\{5,10,20,50\}%c\\in\\\{5,10,20,50\\\}\\%in the main sweep and 13 levels from 2–80% in Section 5\. Because the full\-suite pass/fail label usesTpT\_\{p\}, this signal is a nested proxy, not an independent measurement: a full\-suite pass necessarily passes the subset, so the proxy has no false rejections relative to that operational label and errs only by accepting bugs outsideSc​\(p\)S\_\{c\}\(p\)\. This designed one\-sided error structure is potentially complementary to the panel’s false\-rejection skew and limits generalization to other verifiers \(Section 7\)\. Replacing source\-order selection with uniformly random assertion subsets over 3,115 candidates with usable code and parseable assertions gives87\.10±0\.12%87\.10\\pm 0\.12\\%signal accuracy and\+10\.99±0\.07\+10\.99\\pm 0\.07pp pivotal gain across five seeds, close to the source\-order results \(Supplementary C\)\.

Aggregate correlation gives a mixed diagnosis\. On LiveCodeBench, mean signal–judgeϕ\\phi\(−0\.04\-0\.04\) is below mean judge–judgeϕ\\phi\(0\.260\.26\)\. On HumanEval\+/MBPP\+, signal–judgeϕ\\phi\(0\.310\.31\) is not lower than judge–judgeϕ\\phi\(0\.280\.28\)\. The pooled statistic therefore does not establish independence; Section 5 instead measures the different question of margin\-conditional decision changes\.

A matched\-accuracy permutation control further rules out an unqualified independence account \(Supplementary A\)\. The real signal’s pivotal gain \(\+11\.2\+11\.2pp\) is below the mean for 2,000 synthetic signals with identical overall accuracy but randomly permuted error locations \(\+14\.6\+14\.6pp; 95% range\[13\.1,16\.0\]\[13\.1,16\.0\]pp\)\. Both panel accuracy \(62\.1% pivotal vs\. 82\.4% overall\) and signal accuracy \(81\.7% vs\. 87\.6%\) decline on pivotal queries\. Our evidence therefore supports a descriptive mechanism – high margin\-conditional signal accuracy, arising partly from the nested proxy’s one\-sided errors and lower sensitivity to query difficulty – but does not isolate statistical independence as its cause\.

### A worked example\.

On HumanEval problem 83 \(countnn\-digit integers starting or ending with 1\), aclaude\-opus\-4\-7solution special\-casesn=1n\{=\}1\(returning 1\) and otherwise returns the closed form18\*\(10\*\*\(n\-2\)\)forn\>1n\{\>\}1– correct by inclusion\-exclusion \(10n−1\+9×10n−2−10n−2=18×10n−210^\{n\-1\}\+9\\times 10^\{n\-2\}\-10^\{n\-2\}=18\\times 10^\{n\-2\}\) – and it passes the full test suite\. Our panel splits 4–3 against it \(qwen3\-max\-previewoffers its own mistaken derivation; three judges correctly validate it\)\. Our 20%\-coverage signal, which does not reason about the formula, returnsCORRECTand flips the panel: an elaborate, confident,*incorrect*derivation is exactly as persuasive to a plausibility\-reasoning judge as a correct one – a failure mode execution is immune to by construction\.

## 4Experimental Setup

We construct candidates with 7 solver models, execute them in a sandboxed subprocess against the full test suite \(the operational pass/fail label\) and a limited subset \(the signal\), and have 3–9 judges – disjoint from the solver roster – score each candidate without execution results\. Judges state the verdict on the first response line\. Judge calls use temperature 0; solver calls use temperature 0 on HumanEval\+/MBPP\+ and 0\.2 on LiveCodeBench\. Calls use fixed prompts and a single provider gateway in July 2026\. Table[1](https://arxiv.org/html/2608.06940#S4.T1)reports exact model IDs; prompts, raw outputs, and data are in the supplementary code/data archive\.

The 463 HumanEval\+/MBPP\+ problems are 163 of HumanEval\+’s 164 \(HumanEval/32 excluded a priori for a documented evalplus special\-oracle bug unrelated to any candidate’s correctness\) plus a fixed, seeded random sample of 300 of MBPP\+’s 378 \(oversampled and filtered for unparseable assertions before any candidate was generated\) – both decisions made before any results were observed, not a post\-hoc filter\.

We test four panel configurations\. \(1\)HumanEval\+/MBPP\+,k=7k=7: 463 problems×\\times7 solvers, yielding 3,241 candidates with a recorded full\-suite outcome and all 7 verdicts\. The random\-assertion ablation uses 3,115 of them; the remaining 126 lack usable extracted code or a parseable positive assertion count but remain valid full\-suite failures for the primary analysis\. \(2\)HumanEval\+/MBPP\+,k=9k=9: the same candidates plus two judges from companies not otherwise in the judge roster\. \(3\)k∈\{3,5\}k\\in\\\{3,5\\\}: all 21 five\-judge and 35 three\-judge subsets of the original panel\. These reuse candidates and verdicts and are dependent composition checks, not independent replications\. \(4\)LiveCodeBench,k=7k=7: 60 LeetCode\-style problems \(median 14 tests\)×\\times7 solvers, with a 70% full\-suite pass rate\. HumanEval\+/MBPP\+ is the discovery set; the other configurations are post\-discovery replication checks rather than preregistered confirmatory tests\.

For each configuration we compare panel\-only accuracy with a policy that samples the replaced judge uniformly\. Its expected accuracy is computed exactly by averaging allkkleave\-one\-out substitutions, separately for pivotal \(m=1m=1\) and non\-pivotal \(m≥3m\\geq 3\) queries\. Section 6 additionally compares fixed and majority\-side rules\. We report 95% CIs from 10,000 task\-level bootstrap resamples\.

Table 1:Model roster\. Judges are disjoint from solvers\. Judges 8–9 extend thek=7k\{=\}7panel rather than replacing members and were not selected for accuracy\.
## 5Results

Table 2:Expected accuracy gain under uniformly random single\-judge replacement at 20% test coverage, computed exactly by averaging all replacement choices\. Gains are stratified into pivotal \(m=1m\{=\}1\) and non\-pivotal \(m≥3m\{\\geq\}3\) queries; brackets are 95% task\-bootstrap CIs\.†\\dagger: two newly added, provider\-diverse judges, not a resample\.Table[2](https://arxiv.org/html/2608.06940#S5.T2)summarizes the three headline configurations; Figure[2](https://arxiv.org/html/2608.06940#S5.F2)shows that panel error*rate*rises as the vote margin narrows\. In every configuration, non\-pivotal gain is exactly zero, while pivotal gain is 10–23pp with a bootstrap CI excluding zero \(P​\(gain≤0\)<0\.001P\(\\text\{gain\}\\leq 0\)<0\.001\)\. This does not mean that most errors are pivotal: in the main setting, 199 errors are pivotal and 370 are non\-pivotal\. A margin\-threshold sweep onk=7k\{=\}7HumanEval\+/MBPP\+ gives the same \+1\.82pp overall gain for every thresholdt≥1t\\geq 1, because calls outsidem=1m=1cannot alter a single\-substitution decision\.

Thek∈\{3,5\}k\\in\\\{3,5\\\}subsampling check \(56 dependent judge subsets\) reproduces the same pattern: non\-pivotal gain is exactly zero and pivotal gain is positive in every subset \(Table[3](https://arxiv.org/html/2608.06940#S5.T3)\)\. On LiveCodeBench, pivotal gain is \+23\.3pp versus \+11\.2pp on HumanEval\+/MBPP\+; this comparison is descriptive because benchmark difficulty and signal accuracy both differ\.

Table 3:Dependent composition analysis over every\(75\)\\binom\{7\}\{5\}and\(73\)\\binom\{7\}\{3\}judge subset from the samek=7k\{=\}7panel and candidate set\. Pivotal gain is positive and non\-pivotal gain is zero in all 56 subsets; these are not independent replications\.### Aggregate and conditional diagnostics\.

Table[4](https://arxiv.org/html/2608.06940#S5.T4)compares the pooled effective\-vote count with margin\-stratified utility on the samek=7k\{=\}7HumanEval\+/MBPP\+ data\. The pooledneffn\_\{\\mathrm\{eff\}\}change is not distinguishable from zero \(−0\.04\-0\.04, 95% CI\[−0\.10,\+0\.02\]\[\-0\.10,\+0\.02\]\), whereas uniformly random replacement improves overall accuracy by \+1\.82pp\. Stratification locates that average: \+11\.2pp on the 16\.2% pivotal subset and exactly zero elsewhere\. The diagnostics answer different questions – population\-level error dependence versus conditional decision utility\.

Table 4:The pooled effective\-vote diagnostic \(neffn\_\{\\mathrm\{eff\}\},Kohli,[2026](https://arxiv.org/html/2608.06940#bib.bib1)\) and margin\-conditional substitution utility on the same data\. Accuracy rows use uniformly random replacement\.k=7k\{=\}7, HumanEval\+/MBPP\+, 20% test coverage\.
### Cross\-benchmark breakdown\.

Withink=7k\{=\}7, HumanEval\+ \(n=236n\{=\}236pivotal,905905non\-pivotal\) gains\+16\.95\+16\.95pp/0\.00000\.0000; MBPP\+ \(n=289n\{=\}289/18111811\) gains\+6\.52\+6\.52pp/0\.00000\.0000\. The benchmarks differ in magnitude but share the structural zero outside the pivotal region\.

### Signal\-accuracy sweep\.

How does the pivotal\-region gain scale with signal accuracy? We re\-execute all 3,241 candidates locally at 13 test\-suite granularities \(2–80% of assertions\), yielding signal accuracies 87\.4–88\.7%\. Figure[3](https://arxiv.org/html/2608.06940#S5.F3)shows the gain rises monotonically over this narrow range \(\+11\.10\+11\.10pp to\+11\.76\+11\.76pp, every CI excluding zero\) – too narrow a span to establish a general dose\-response alone\. We separately plot the LiveCodeBench point \(99\.3% accuracy,\+23\.3\+23\.3pp\) for reference only,*not*as evidence the trend extends across benchmarks\. A coarse 2% signal already recovers most of the 80%\-coverage gain, suggesting gating can also be cheap along test\-suite depth; lower signal accuracies \(below 87%\) are future work \(Section 7\)\.

### Even panels validate the predicted asymmetry\.

We test Proposition 2 by re\-deriving all\(76\)=7\\binom\{7\}\{6\}=7six\-judge \(even\) subsets of the original panel, without additional data collection\. Table[6](https://arxiv.org/html/2608.06940#S5.T6)reports the gain in each region, aggregated across all 7 subsets\. The tie region \(s=3s\{=\}3\) shows the largest gain we test \(\+30\.1\+30\.1pp\); the bare\-*correct*\-leaning majority \(s=4s\{=\}4, predicted pivotal\) shows a smaller, consistently positive gain; critically, the bare\-*incorrect*\-leaning majority \(s=2s\{=\}2, identical\|mi\|=2\|m\_\{i\}\|\{=\}2tos=4s\{=\}4\) shows a gain of*exactly*0\.00000\.0000in all 7 subsets, exactly as Proposition 2 predicts\. This follows directly from the tie\-breaking convention specified in the proposition; we are not aware of prior work stating this asymmetry for LLM judge panels, though it is an arithmetic consequence rather than a surprising empirical discovery in itself\.

### Why panel errors skew toward false rejection near the margin\.

Pivotal\-region errors are not evenly split: the false\-negative rate \(correct code voted incorrect\) rises from 7\.5% on non\-pivotal queries to 33\.7% on pivotal queries \(4\.5×4\.5\\times\), while the false\-positive rate rises only1\.4×1\.4\\times\(39\.1% to 53\.6%\)\. Two qualitative hypotheses do*not*explain this \(Supplementary D\): no hedging\-language difference between error types, and an LLM classification of 100 sampled wrong verdicts finds both equally likely to cite a specific, checkable claim\.

Table 5:Judges’ reject rate, accuracy, and Matthews correlation coefficient \(MCC\), a base\-rate\-robust discrimination metric\. The full\-suite operational label is 80\.4% correct\.Part of this is compositional rather than purely psychological\. Table[5](https://arxiv.org/html/2608.06940#S5.T5)helps separate a mechanical effect from a genuine one\. Raw accuracy’s correlation with reject rate \(r=−0\.99r\{=\}\{\-\}0\.99\) is inflated by the 80\.4% base rate: leniency is rewarded almost by construction\. Base\-rate\-invariant balanced accuracy correlates with reject rate far more weakly \(r=−0\.61r\{=\}\{\-\}0\.61\), confirming part of the gap is this artifact\. But MCC, also base\-rate\-robust, still correlates with reject rate atr=−0\.85r\{=\}\{\-\}0\.85and with panel accuracy across all 28 real subsets atr=0\.86/0\.85r\{=\}0\.86/0\.85\(vs\.r=0\.94/0\.97r\{=\}0\.94/0\.97for raw accuracy; Supplementary D\) – a genuine, smaller discrimination gap survives alongside the mechanical one\. This panel includes more strict, lower\-MCC judges than lenient ones, so its aggregate error where votes are close enough for individual idiosyncrasy to decide skews toward their shared error, false rejection – consistent with a compositional explanation for the correlated\-error problem motivating this paper \(Kohli,[2026](https://arxiv.org/html/2608.06940#bib.bib1)\), paralleling annotator calibration in crowdsourcing \(Dawid and Skene,[1979](https://arxiv.org/html/2608.06940#bib.bib15)\), though with only 7 judges we do not claim an isolated causal mechanism\. A cheap pre\-deployment check – confusion matrix and MCC on a calibration sample – can flag both effects before they distort the votes that matter most\.

Table 6:Accuracy gain across all\(76\)=7\\binom\{7\}\{6\}=7six\-judge subsets, by region\. Despites=2s\{=\}2ands=4s\{=\}4sharing the same\|mi\|=2\|m\_\{i\}\|=2, onlys=4s\{=\}4is pivotal – confirming Proposition 2’s asymmetry prediction\.![Refer to caption](https://arxiv.org/html/2608.06940v1/figures/fig1_margin_error_rate.png)Figure 2:Panel error rate by normalized vote margin; bars are 95% task\-bootstrap CIs\. Counts from narrowest to widest raw margin are HE\+/MBPP\+k=7k\{=\}7: 525/740/1026/950;k=9k\{=\}9: 380/516/769/894/682; LiveCodeBenchk=7k\{=\}7: 113/102/66/139\.![Refer to caption](https://arxiv.org/html/2608.06940v1/figures/fig3_granularity_sweep.png)Figure 3:Observed pivotal\-vote gain over a narrow signal\-accuracy range\. Blue circles: controlled HumanEval\+/MBPP\+ sweep with 95% problem\-cluster bootstrap CIs\. The unconnected LiveCodeBench point is cross\-benchmark context, not evidence of a shared dose\-response curve\.

## 6Discussion

### Aggregate independence statistics do not characterize conditional utility\.

The effective\-votes metric ofKohli \([2026](https://arxiv.org/html/2608.06940#bib.bib1)\)diagnoses correlated errors, averaged over the full query population; it was not designed to detect margin\-conditional decision utility, and Table[4](https://arxiv.org/html/2608.06940#S5.T4)shows directly that it does not\. When a signal’s value is concentrated in a minority subset \(12–27% here\) and exactly zero elsewhere, the average can look statistically insignificant even though the underlying effect is large – without the statistic being wrong for the question it was built to answer\. We suspect this generalizes: any intervention whose value is conditional on an observable query property can look ineffective under a non\-stratifying aggregate statistic\. We recommend reporting effects conditional on panel disagreement alongside any aggregate independence statistic\.

### Margin gating is separate from replacement choice\.

Algorithm[1](https://arxiv.org/html/2608.06940#alg1)accepts a specified replacement ruleRR\. Proposition 1 guarantees that applyingRRonly atm=1m=1produces the same decisions as applying the same single\-ballot rule universally; it does not say that all choices ofRRare equally accurate\. Tables[2](https://arxiv.org/html/2608.06940#S5.T2)–[6](https://arxiv.org/html/2608.06940#S5.T6)report the exact expectation for uniform\-random replacement\. Fixed\-index accuracy ranges from 82\.6–85\.4%, while a majority\-side rule reaches 85\.62% \(Table[7](https://arxiv.org/html/2608.06940#S6.T7)\)\. Thus gating identifies*when*to call the signal; the replacement rule determines how strongly that signal influences a pivotal vote\.

Algorithm 1Margin\-Gated Verification \(oddkk\)1:Input:query

xx, odd\-sized

kk\-judge panel

J=\(j1,…,jk\)J=\(j\_\{1\},\\dots,j\_\{k\}\), signal

VV, specified single\-ballot replacement rule

RR
2:Collect votes

v1,…,vk←J​\(x\)v\_\{1\},\\dots,v\_\{k\}\\leftarrow J\(x\); let

s←∑lvls\\leftarrow\\sum\_\{l\}v\_\{l\}
3:

m←\|2​s−k\|m\\leftarrow\|2s\-k\|
4:if

m=1m=1then

5:

v∗←V​\(x\)v^\{\*\}\\leftarrow V\(x\);

r←R​\(v1,…,vk\)r\\leftarrow R\(v\_\{1\},\\dots,v\_\{k\}\)
6:

vr←v∗v\_\{r\}\\leftarrow v^\{\*\}\{

RR: fixed, uniform\-random, or majority\-side\}

7:return

𝟙​\[∑lvl\>k/2\]\\mathbb\{1\}\[\\sum\_\{l\}v\_\{l\}\>k/2\]
8:else

9:return

𝟙​\[s\>k/2\]\\mathbb\{1\}\[s\>k/2\]\{

V​\(x\)V\(x\)provably cannot change this output \(Prop\. 1\)\}

10:endif

### Alternative deployment policies\.

Table[7](https://arxiv.org/html/2608.06940#S6.T7)separates query gating from replacement choice\. Uniform\-random replacement improves panel accuracy from 82\.44% to 84\.26%; majority\-side replacement reaches 85\.62% \(\+3\.18pp, 95% CI \[2\.10, 4\.26\]\) at the same 16\.2% call rate\. On a pivotal vote, this rule lets the signal overturn the current majority when they disagree; it is decision\-equivalent to giving the signal two votes universally, but avoids non\-pivotal calls\. Signal\-only \(87\.60%\) and the best judge \(87\.16%\) remain stronger, so the deployment claim is conditional on retaining a panel\. Retention can be an external constraint rather than an accuracy choice – for example, an audit or governance protocol may require multi\-provider verdicts, or a deployment may need tolerance to one judge’s outage or model drift\. We do not test those benefits; our policy claim applies only after such a constraint has been imposed\.

Table 7:Policy comparison,k=7k\{=\}7HumanEval\+/MBPP\+\. Gating preserves a specified substitution rule at6×6\\timesfewer calls\. Replacement choice matters: majority\-side exceeds uniform replacement, while signal\-only and the best judge remain stronger\.Sandboxed execution takes 49ms/call vs\. 5\.72s for a judge call \(Supplementary C\)\. Universal verification is therefore already cheap in our setup; gating matters economically when the verifier is expensive, not for this specific signal\.

## 7Limitations

The signal is a nested subset of the full\-suite operational label, giving it one\-sided errors: it can miss uncovered bugs but cannot reject a full\-suite pass\. Its complementarity with the panel’s false\-rejection skew may therefore not transfer to independent, symmetric, or abstaining verifiers\. Signal accuracy is high \(87–99%\), and the controlled sweep spans only 87\.4–88\.7%; behavior below 87% remains unknown\. Signal\-only and the best judge outperform every panel policy tested, so gating is relevant only when a panel must be retained\. Replacement\-policy comparisons are post\-hoc on the discovery data and need held\-out validation\. LiveCodeBench has 113 pivotal observations; all benchmarks are code tasks\. API aliases may drift, and partly overlapping providers introduce a measured same\-provider effect\. Weighted panels, non\-binary signals, and non\-code domains remain open\.

## 8Reproducibility and Ethics

No human subjects or personal data are involved\. Untrusted candidate code runs under a danger\-pattern scan, timeout, and process isolation\. Section 4 reports model IDs, sample sizes, and bootstrap procedures\. CIs capture task\-sampling rather than call\-level uncertainty: 3 of 15 sampled pivotal query–judge groups produced at least one different verdict across five repeated calls \(Supplementary C\)\.

## 9Conclusion

For single\-ballot substitution in an unweighted\-majority panel, only pivotal decisions can change\. Across three code benchmarks and four panel sizes, substitution gain is concentrated there and zero elsewhere\. Aggregate dependence and margin\-conditional utility therefore answer different questions\. When retaining a panel, gate a specified replacement rule on pivotal votes; in our main setting, majority\-side replacement is stronger than uniform replacement, although simpler signal\-only and single\-judge policies remain stronger still\.

## References

- A\. Akinfaderin and N\. Diallo \(2026\)FregeLogic at semeval 2026 task 11: a hybrid neuro\-symbolic architecture for content\-robust syllogistic validity prediction\.External Links:2604\.18328Cited by:[§2](https://arxiv.org/html/2608.06940#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. Sutton \(2021\)Program synthesis with large language models\.External Links:2108\.07732Cited by:[§2](https://arxiv.org/html/2608.06940#S2.SS0.SSS0.Px5.p1.1)\.
- J\. F\. Banzhaf \(1965\)Weighted voting doesn’t work: a mathematical analysis\.Rutgers Law Review19,pp\. 317–343\.Cited by:[§2](https://arxiv.org/html/2608.06940#S2.SS0.SSS0.Px4.p1.1)\.
- L\. Chen, M\. Zaharia, and J\. Zou \(2024\)FrugalGPT: how to use large language models while reducing cost and improving performance\.Transactions on Machine Learning Research\.Cited by:[§2](https://arxiv.org/html/2608.06940#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. Zaremba \(2021\)Evaluating large language models trained on code\.External Links:2107\.03374Cited by:[§2](https://arxiv.org/html/2608.06940#S2.SS0.SSS0.Px5.p1.1)\.
- M\. d\. Condorcet \(1785\)Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix\.Note:Paris: Imprimerie RoyaleCited by:[§2](https://arxiv.org/html/2608.06940#S2.SS0.SSS0.Px4.p1.1)\.
- A\. P\. Dawid and A\. M\. Skene \(1979\)Maximum likelihood estimation of observer error\-rates using the EM algorithm\.Journal of the Royal Statistical Society: Series C \(Applied Statistics\)28\(1\),pp\. 20–28\.Cited by:[§5](https://arxiv.org/html/2608.06940#S5.SS0.SSS0.Px5.p2.5)\.
- Y\. Geifman and R\. El\-Yaniv \(2017\)Selective classification for deep neural networks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.30\.External Links:1705\.08500Cited by:[§2](https://arxiv.org/html/2608.06940#S2.SS0.SSS0.Px3.p1.1)\.
- N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica \(2025\)LiveCodeBench: holistic and contamination free evaluation of large language models for code\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2608.06940#S2.SS0.SSS0.Px5.p1.1)\.
- G\. Kohli \(2026\)Nine judges, two effective votes: correlated errors undermine llm evaluation panels\.External Links:2605\.29800Cited by:[§1](https://arxiv.org/html/2608.06940#S1.p1.1),[§1](https://arxiv.org/html/2608.06940#S1.p2.1),[§1](https://arxiv.org/html/2608.06940#S1.p5.1),[§2](https://arxiv.org/html/2608.06940#S2.SS0.SSS0.Px1.p1.4),[§2](https://arxiv.org/html/2608.06940#S2.SS0.SSS0.Px4.p1.1),[§5](https://arxiv.org/html/2608.06940#S5.SS0.SSS0.Px5.p2.5),[Table 4](https://arxiv.org/html/2608.06940#S5.T4),[§6](https://arxiv.org/html/2608.06940#S6.SS0.SSS0.Px1.p1.1)\.
- C\. Kuai, J\. Jiang, Z\. Zhu, H\. Wang, K\. Wu, Z\. Li, Y\. Zhang, C\. Liu, Z\. Tu, Z\. Fan, and Y\. Zhou \(2026\)How independent are large language models? a statistical framework for auditing behavioral entanglement and reweighting verifier ensembles\.External Links:2604\.07650Cited by:[§2](https://arxiv.org/html/2608.06940#S2.SS0.SSS0.Px1.p1.4)\.
- J\. Kwok, S\. Li, P\. Atreya, Y\. Liu, Y\. Jiang, C\. Finn, M\. Pavone, I\. Stoica, and A\. Mirhoseini \(2026\)LLM\-as\-a\-verifier: a general\-purpose verification framework\.External Links:2607\.05391Cited by:[§2](https://arxiv.org/html/2608.06940#S2.SS0.SSS0.Px1.p1.4)\.
- J\. Liu, C\. S\. Xia, Y\. Wang, and L\. Zhang \(2023\)Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2608.06940#S2.SS0.SSS0.Px5.p1.1)\.
- H\. Mozannar and D\. Sontag \(2020\)Consistent estimators for learning to defer to an expert\.InProceedings of the 37th International Conference on Machine Learning \(ICML\),External Links:2006\.01862Cited by:[§2](https://arxiv.org/html/2608.06940#S2.SS0.SSS0.Px3.p1.1)\.
- H\. S\. Seung, M\. Opper, and H\. Sompolinsky \(1992\)Query by committee\.InProceedings of the Fifth Annual Workshop on Computational Learning Theory \(COLT\),pp\. 287–294\.Cited by:[§2](https://arxiv.org/html/2608.06940#S2.SS0.SSS0.Px3.p1.1)\.
- P\. Verga, S\. Hofstatter, S\. Althammer, Y\. Su, A\. Piktus, A\. Arkhangorodsky, M\. Xu, N\. White, and P\. Lewis \(2024\)Replacing judges with juries: evaluating llm generations with a panel of diverse models\.External Links:2404\.18796Cited by:[§2](https://arxiv.org/html/2608.06940#S2.SS0.SSS0.Px1.p1.4)\.