Oracle Gap and Signal Fidelity: A Fixed-Pool Diagnostic for Test-Time Collaboration

# Oracle Gap and Signal Fidelity: A Fixed-Pool Diagnostic for Test-Time Collaboration
Source: [https://arxiv.org/html/2607.17531](https://arxiv.org/html/2607.17531)
###### Abstract

Test\-time collaboration, including self\-consistency, best\-of\-NNselection, critic models, and verifier pipelines, is often credited with broadly improving LLM reasoning, yet its gains are uneven and sometimes negative\. We ask when training\-free collaboration should be expected to help\. For a fixed candidate pool, we decompose a selector or verifier’s net gain into measurable factors: recoverable mass, verification\-signal coverage, conditional selection quality, and harm to already\-correct outputs\. This reframes collaboration as a candidate\-selection problem rather than as an intrinsic property of a multi\-agent topology\. Across LiveCodeBench, MATH Level\-5 hard subjects, and GPQA\-Diamond, gains are bounded first by the oracle gap and then by signal fidelity, which we measure directly as candidate\-level agreement between verifier verdicts and official labels\. On LiveCodeBench, a public\-test verifier \(MCC 0\.825\) gains\+8\.14\+8\.14percentage points \(pp\) over a first\-sample baseline; a generated\-test verifier \(MCC 0\.248\) improves by\+2\.70\+2\.70pp and is not statistically distinguishable from an LLM selector, but operates at near\-zero harm versus the selector’s 4\.69% harm rate\. On MATH, a symbolic answer\-equivalence selector beats self\-consistency by\+4\.67\+4\.67pp, while LLM selectors are negative\. On GPQA\-Diamond, recoverable mass is only 3\.03% and 87\.54% of candidate pools are answer\-identical; a weaker model’s pools shrink both further, suggesting that oracle gap is a joint property of task, model, and sampling configuration\. Our framework yields a practical pre\-deployment diagnostic: estimate the oracle gap, then measure coverage, signal fidelity, and harm before investing in collaboration\.

## 1Introduction

Test\-time collaboration is now a common recipe for improving large language model reasoning: sampling multiple candidates and voting by self\-consistency, asking a critic to inspect an answer, filtering code with tests, or routing an input to a stronger worker\. These mechanisms are often grouped under one intuition: more computation, more agents, or more judging should improve reliability\.

Empirically, this intuition is incomplete\. A critic may fix some failures while damaging answers that were already correct\. Self\-consistency is meaningful for math problems with normalized final answers, but exact majority voting over code strings is mostly a fallback rule rather than a real selector\. Public tests can be an extremely strong signal for code, while self\-generated tests can inherit the model’s own blind spots\. A second model may add diversity, or it may simply choose among nearly identical candidates, right or wrong\. In short, collaboration is not a property of the topology alone: its value depends on the candidate pool, the task’s evaluation structure, and the fidelity of the signal used to choose among candidates\.

This paper asks:*when should training\-free collaboration be expected to help?*We refer to the resulting fixed\-pool diagnostic framework as*OracleGap*\. We study the selector/verifier setting: for each problem, a fixed candidate pool is generated, and a mechanism chooses one candidate without further training\. This covers best\-of\-NNselection, LLM selectors, generated\-test filtering, public\-test filtering, and symbolic answer\-equivalence selection, and deliberately separates selection from repair: a repair mechanism can create new candidates, whereas a selector can only capture the improvement already present in the pool\.

\(a\) Fixed\-pool diagnostic and gain decompositionInputxxy1y\_\{1\}y2y\_\{2\}y3y\_\{3\}y4y\_\{4\}y5y\_\{5\}Fixed poolC​\(x\)C\(x\),k=5k=5referenceVerification signalcoverage: signal defined?fidelity: verdict vs\. official labelSelecty^\\hat\{y\}fallback→y1\\to y\_\{1\}Outcome accountingfixed\(\+\)\(\+\)/ harmed\(−\)\(\-\)/ unchanged\(0\)\(0\)ref\. wrong\+\+correct alternative⇒\\Rightarrowrecoverable;oracle gapofficial\-label audit onlygain=P​\(R∧D\)​q−P​\(C∧D\)​h\\mathrm\{gain\}=P\(R\\wedge D\)\\,q\-P\(C\\wedge D\)\\,hRR: recoverable,CC: reference correct,DD: signal defined\(b\) Pre\-deployment workflow1\. Estimateoracle gap2\. Auditcoverage and fidelity3\. Comparecapture against harm4\. Deployonly if net\-positivesmall gap⇒\\Rightarrowstop*GPQA: 3\.03 pp*low coverage/fidelity limits capture*L4\-gen: 35\.8% active**MCC 0\.248*harm can erase recovered gains*GPQA L1: net−1\.68\-1\.68pp*high\-fidelity signals pay*L4\-public:\+8\.14\+8\.14pp**zero harm*

Figure 1:OracleGap overview\. \(a\) Official labels expose the audit\-only oracle gap and measure the coverage and fidelity of a deployable signal; selection then produces fixes, harms, or no changes\. \(b\) The pre\-deployment workflow stops when recoverable mass is small and deploys only signals whose captured gains exceed harm\.Our central claim is that selection gains are bounded first by the*oracle gap*and then by the*coverage, fidelity, quality, and harm*of the verification signal\. We make this concrete with a fixed\-pool decomposition:

gain=P​\(recoverable∧defined\)​q−P​\(reference​correct∧defined\)​h,\\begin\{split\}\\mathrm\{gain\}=\{\}&P\(\\mathrm\{recoverable\}\\wedge\\mathrm\{defined\}\)\\,q\\\\ &\-P\(\\mathrm\{reference\\ correct\}\\wedge\\mathrm\{defined\}\)\\,h,\\end\{split\}\(1\)where*recoverable*means the reference output is wrong but some candidate in the pool is correct;qqis conditional selection success on recoverable examples with a defined signal; andhhis how often the mechanism changes a reference\-correct example into an incorrect one among examples with a defined signal\. We also measure verifier*fidelity*directly as candidate\-level agreement between verifier verdicts and official labels, linking the decomposition to imperfect\-verifier theory\.

We evaluate on LiveCodeBench \(execution signals available, generated tests imperfect\), MATH Level\-5 hard subjects \(answers normalizable or symbolically checkable\), GPQA\-Diamond \(a low\-recoverable, low\-diversity boundary\), and HumanEval\+ \(a saturated code anchor\)\(Jainet al\.,[2024](https://arxiv.org/html/2607.17531#bib.bib9); Hendryckset al\.,[2021](https://arxiv.org/html/2607.17531#bib.bib10); Reinet al\.,[2024](https://arxiv.org/html/2607.17531#bib.bib11); Liuet al\.,[2023](https://arxiv.org/html/2607.17531#bib.bib12)\)\.

Figure[1](https://arxiv.org/html/2607.17531#S1.F1)summarizes the fixed\-pool accounting and turns it into a pre\-deployment decision workflow\.

Our contributions are:

1. 1\.A training\-free decomposition of selector/verifier gain into recoverable mass, signal coverage, conditional quality, and harm, making collaboration gains measurable at the task level rather than attributed to agent count or topology\.
2. 2\.A three\-seed LiveCodeBench selector ladder comparing a first\-sample baseline, LLM selectors, a generated\-test verifier, a public\-test verifier, and an any\-of\-kkoracle upper bound on a unified task set with hierarchical confidence intervals\.
3. 3\.Evidence that verifier fidelity is an empirical bottleneck \(Figure[2](https://arxiv.org/html/2607.17531#S4.F2)\): a public\-test verifier with high candidate\-level fidelity captures much more of the oracle gap than a generated\-test verifier with low fidelity, while the latter has near\-zero harm\.
4. 4\.Cross\-task boundary evidence: on MATH, a symbolic answer\-equivalence selector improves over self\-consistency while natural\-language LLM selectors are negative; on GPQA\-Diamond, the candidate pool has only a 3\.03pp oracle gap with high answer homogeneity, and LLM selectors are net\-negative on that pool\.
5. 5\.A routing\-η\\etaextension \(supplementary\) showing that worker routing and candidate selection obey different constraints and should not be conflated under a single notion of collaboration\.

Together, these results yield a practical diagnostic: before deploying a critic, verifier, or multi\-agent selector, first estimate the oracle gap of the candidate pool, then measure whether the available signal has enough coverage, fidelity, and low harm to justify the extra computation\.

## 2Related Work

### 2\.1Test\-Time Selection and Execution\-Based Filtering

Self\-consistency and best\-of\-NNsampling improve reasoning when aggregation or selection is reliable\(Wanget al\.,[2023](https://arxiv.org/html/2607.17531#bib.bib1); Brownet al\.,[2024](https://arxiv.org/html/2607.17531#bib.bib13); Snellet al\.,[2024](https://arxiv.org/html/2607.17531#bib.bib14)\)\. For code, CodeT filters candidates with generated tests\(Chenet al\.,[2022](https://arxiv.org/html/2607.17531#bib.bib2)\), AlphaCode combines sampling with test filtering and clustering\(Liet al\.,[2022](https://arxiv.org/html/2607.17531#bib.bib3)\), and MBR\-Exec selects by behavioral agreement\(Shiet al\.,[2022](https://arxiv.org/html/2607.17531#bib.bib4)\)\. These are direct predecessors of our generated\- and public\-test mechanisms\.

Rather than introduce another selector, we fix candidate pools and decompose realized gain into recoverable mass, signal coverage, conditional quality, and harm\. This diagnostic layer directly compares natural\-language, generated\-test, public\-test, and symbolic\-equivalence mechanisms that are usually evaluated separately\.

### 2\.2Trained and Imperfect Verifiers

Verifier\-based selection includes trained outcome and process verifiers\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.17531#bib.bib5); Uesatoet al\.,[2022](https://arxiv.org/html/2607.17531#bib.bib6); Lightmanet al\.,[2024](https://arxiv.org/html/2607.17531#bib.bib7)\)\. Multi\-Agent Verification introduces BoN\-MAV\(Lifshitzet al\.,[2025](https://arxiv.org/html/2607.17531#bib.bib15)\), while newer general\-purpose and multi\-sequence verifiers improve granularity, calibration, and budget\-aware ranking\(Kwoket al\.,[2026](https://arxiv.org/html/2607.17531#bib.bib21); Kimet al\.,[2026](https://arxiv.org/html/2607.17531#bib.bib22)\)\. These improve the verifier; our training\-free setting asks when an available signal converts candidate diversity into net gain at all\.

A complementary theoretical line derives best\-of\-NNand rejection\-sampling behavior from verifier ROC geometry\(Dorneret al\.,[2025](https://arxiv.org/html/2607.17531#bib.bib16)\), shows that false positives impose accuracy ceilings and can bend scaling curves downward\(Stroeblet al\.,[2024](https://arxiv.org/html/2607.17531#bib.bib17)\), and establishes asymptotic advantages for verifier\-based scaling\(Setluret al\.,[2025](https://arxiv.org/html/2607.17531#bib.bib18)\)\. Compute\-matched verification is not uniformly optimal\(Singhiet al\.,[2025](https://arxiv.org/html/2607.17531#bib.bib19)\);Venkteshet al\.\([2025](https://arxiv.org/html/2607.17531#bib.bib20)\)survey the broader design landscape\.

Most directly,Luet al\.\([2025](https://arxiv.org/html/2607.17531#bib.bib25)\)systematically study when solution verification pays off across self\-, within\-family, and cross\-family solver–verifier pairs\. They find stronger cross\-family verification as solver–verifier similarity decreases; our L3 instead uses a smaller cross\-family selector, so the L1–L3 gap reflects capability rather than family similarity, which our capture/harm decomposition makes explicit\.

We make verifier quality observable through candidate\-level agreement with official labels, including accuracy, error rates, and MCC\. Separating fidelity from coverage and harm captures verifiers that are accurate but rarely active, or conservative and low\-harm yet miss many correct candidates, complementing imperfect\-verifier theory with cross\-domain empirical accounting\.

### 2\.3When Multi\-Agent Collaboration Helps

Recent work directly challenges the assumption that increasing agent count or interaction rounds is intrinsically beneficial\. Diversity\-based analysis shows that homogeneous agents saturate because their outputs are correlated, whereas heterogeneous agents provide complementary effective channels\(Yanget al\.,[2026](https://arxiv.org/html/2607.17531#bib.bib23)\)\. The closest same\-question work finds that entropy dynamics vary with task and coordination topology; reducing interaction\-induced uncertainty therefore need not by itself establish genuine improvement\(Zhaoet al\.,[2026](https://arxiv.org/html/2607.17531#bib.bib24)\)\. Uncertainty decomposition in multi\-agent debate similarly distinguishes epistemic gain from aleatoric cost\(Qiaoet al\.,[2026](https://arxiv.org/html/2607.17531#bib.bib26)\)\.

System\-level studies reach compatible conclusions from different angles\. MAST catalogs system\-design issues, inter\-agent misalignment, and task\-verification failures\(Cemriet al\.,[2025](https://arxiv.org/html/2607.17531#bib.bib27)\); controlled scaling studies find capability saturation and topology\-dependent error propagation\(Kimet al\.,[2025](https://arxiv.org/html/2607.17531#bib.bib28)\); and matched\-token experiments show that single\-agent reasoning can outperform multi\-agent systems when total thinking compute is held fixed\(Tran and Kiela,[2026](https://arxiv.org/html/2607.17531#bib.bib29)\)\. Earlier work also finds that self\-correction is unreliable without external feedback and that debate can improve some factual and reasoning tasks\(Huanget al\.,[2024](https://arxiv.org/html/2607.17531#bib.bib8); Duet al\.,[2023](https://arxiv.org/html/2607.17531#bib.bib33)\)\.

Our scope is narrower and more mechanistic than these topology\-, entropy\-, or system\-level analyses\. We ask whether a concrete selector can convert a*fixed*candidate pool into a better output\. GPQA\-Diamond provides a boundary case: mostk=5k=5pools are answer\-identical and the oracle gap is only 3\.03pp, leaving little room for any selector\. LiveCodeBench and MATH provide the complementary case: recoverable candidates exist, but gains depend on whether the selection signal is faithful and low\-harm\. Thus our oracle\-gap and signal\-fidelity framework is complementary to diversity and entropy: it quantifies the improvement space and the mechanism’s ability to capture it\.

### 2\.4Generated Tests and Label\-Free Fidelity Estimation

Generated\-test methods are especially close to our deployable code verifier\. CoSPlay co\-evolves code candidates and self\-generated unit tests without ground\-truth tests\(Huet al\.,[2026](https://arxiv.org/html/2607.17531#bib.bib30)\)\. UTGen learns error\-revealing inputs and expected outputs, while UTDebug spends test\-time compute to validate feedback and avoid overfitting\(Prasadet al\.,[2025](https://arxiv.org/html/2607.17531#bib.bib31)\)\. UTRL trains test and code generators adversarially so that generated tests better discriminate faulty programs\(Leeet al\.,[2026](https://arxiv.org/html/2607.17531#bib.bib32)\)\. These approaches address the limitation surfaced by our results: self\-generated tests may be low\-harm yet low\-fidelity because they inherit the generator’s blind spots or assign incorrect expected outputs\.

Our contribution is complementary\. We use official labels to audit generated\-test verdicts directly, identifying fidelity and activation as separate bottlenecks\. This creates a bridge to label\-free reliability estimation: future generated\-test systems should be evaluated not only by final pass@1, but also by whether their internal confidence or reliability scores predict official\-label fidelity\.

### 2\.5Positioning Summary

Together, these literatures cover candidate generation, verifier training, imperfect\-verifier theory, multi\-agent scaling, and methods for improving generated tests\. Our contribution is the missing fixed\-pool measurement layer across benchmarks with different decidability profiles: before attributing gains to collaboration or topology, measure whether recoverable answers exist and whether the deployed signal has enough coverage, fidelity, quality, and low harm to recover them\.

## 3Method and Experimental Setup

### 3\.1Problem Setup

For each inputxx, a model produces a fixed candidate pool

C​\(x\)=\{y1,y2,…,yk\}\.C\(x\)=\\\{y\_\{1\},y\_\{2\},\\ldots,y\_\{k\}\\\}\.\(2\)The first candidatey1y\_\{1\}is the first\-sample baseline\. A training\-free collaboration mechanism then selects one candidate or supplies a verification signal over the pool\. This setup covers best\-of\-NNselection, self\-consistency, LLM\-based selectors, generated\-test filtering, public\-test filtering, and symbolic answer\-equivalence selection\.

We distinguish fixed\-pool selection from repair\. A repair mechanism may create a new answer outside the original pool, so its gain depends jointly on detection and generation\. A selector can only capture improvement already present among the sampled candidates\. Fixing the pool isolates the question studied here: how much available oracle space does a concrete selection mechanism convert into realized gain?

### 3\.2Oracle Gap and Recoverable Mass

For each task, the any\-of\-kkoracle succeeds if at least one candidate passes the official evaluator\. Relative to a reference output, a task is*recoverable*when the reference is wrong but some candidate is correct:

recoverable​\(x\)=reference​wrong​\(x\)∧any​@​k​correct​\(x\)\.\\mathrm\{recoverable\}\(x\)=\\mathrm\{reference\\ wrong\}\(x\)\\wedge\\mathrm\{any@\}k\\mathrm\{\\ correct\}\(x\)\.\(3\)The recoverable mass isP​\(recoverable\)P\(\\mathrm\{recoverable\}\)\. The*oracle gap*is the score difference between the any\-of\-kkoracle and the chosen reference\. It is an upper bound on every fixed\-pool selector\. If the gap is small, elaborate collaboration has little absolute room to improve, regardless of selector sophistication\.

### 3\.3Coverage, Quality, Harm, and Fidelity

We decompose gain using three conditional quantities:

coverage\\displaystyle\\mathrm\{coverage\}=P​\(signal​defined\),\\displaystyle=P\(\\mathrm\{signal\\ defined\}\),\(4\)quality\\displaystyle\\mathrm\{quality\}=P​\(selected​passes∣recoverable,defined\),\\displaystyle=P\(\\mathrm\{selected\\ passes\}\\mid\\mathrm\{recoverable\},\\mathrm\{defined\}\),\(5\)harm\\displaystyle\\mathrm\{harm\}=P​\(selected​fails∣reference​passes,defined\)\.\\displaystyle=P\(\\mathrm\{selected\\ fails\}\\mid\\mathrm\{reference\\ passes\},\\mathrm\{defined\}\)\.\(6\)*Signal defined*means the mechanism produced usable evidence: for example, a parseable LLM selector output, executable public tests, generated tests that can be run, or a symbolic answer cluster\. These quantities deliberately live on different slices\. Quality is neither overall accuracy nor one minus harm; it asks whether the mechanism selects a correct candidate among recoverable rows where evidence is available\. Harm asks whether the same mechanism turns a reference\-correct row into an incorrect selected output, again conditional on evidence being defined\. With these definitions, task\-level gain follows Equation[1](https://arxiv.org/html/2607.17531#S1.E1)\.

For deployed mechanisms with a fallback policy, we additionally report*effective capture*and*effective harm*over all recoverable and reference\-correct rows, respectively\. These effective quantities include fallback and no\-op behavior and therefore close the fixed\-minus\-harmed accounting identity\. They must not be confused with the conditional quality and harm terms above\. This distinction is important for conservative verifiers: a method may have low effective harm because it frequently falls back toy1y\_\{1\}, even though its active signal is available on only a small slice\.

We separately measure verifier*fidelity*as a property of the evidence itself: candidate\-level agreement between verifier verdicts and official candidate labels, summarized by accuracy, false\-positive rate, false\-negative rate, and MCC\. Keeping fidelity separate from coverage and effective behavior prevents a high\-fallback mechanism from appearing reliable merely because it seldom changes the reference\.

### 3\.4Selector Ladder

We compare the mechanisms in Table[1](https://arxiv.org/html/2607.17531#S3.T1)on fixedk=5k=5pools\. Self\-consistency and majority voting are aggregation baselines rather than ladder tiers, reported where answers are naturally normalizable \(MATH and GPQA\)\. Verifier implementation depends on the benchmark: executed tests for code, symbolic or normalized\-answer equivalence for MATH, and no natural executable verifier for GPQA, which therefore serves primarily as a low\-recoverable boundary case\.

Table 1:Selector ladder used on fixedk=5k=5pools\.
### 3\.5Critic Actions and Routing Boundary

The word “critic” can denote scientifically different actions\. A verdict critic judges one candidate; a selector critic chooses among candidates; neither creates a new answer\. A repair critic rewrites an answer, and test\-feedback repair rewrites using an external failure signal\. Only the first two are bounded by the fixed\-pool oracle gap\. Our main tables therefore evaluate selector and verifier mechanisms, not repair\.

Worker routing is also distinct from candidate selection: it chooses a worker from input\-level features before candidate evidence exists\. Because routing depends on feature predictability rather than candidate\-level verification, its full definition and results are reported as a supplementary boundary analysis \(Appendix A\.2\), not as a selector\-ladder tier\.

### 3\.6Benchmarks, Models, and Statistical Reporting

Table 2:Benchmarks and their diagnostic roles\.The main model source is Qwen3\.6\-35B\-A3B\-BF16 for LiveCodeBench, MATH, and the GPQA pool\-boundary table; Qwen3\.5\-9B, Qwen2\.5\-14B, DeepSeek\-R1\-Distill\-Qwen\-14B, and Gemma\-3\-4B\-it appear only in follow\-up, robustness, or routing analyses\. To avoid provenance mismatches, every horizontally compared number carries a five\-part provenance tuple\(benchmark,task​set,reference,label​source,model​source\)\(\\mathrm\{benchmark\},\\mathrm\{task\\ set\},\\mathrm\{reference\},\\mathrm\{label\\ source\},\\mathrm\{model\\ source\}\); numbers with different provenance are never merged into one table, and aggregation and confidence\-interval methods are declared per table\. The legacy suffix\_vllmis a run label mapping to Qwen3\.6\-35B\-A3B\-BF16 via the run\-label registry\.

Code tasks use the official LiveCodeBench evaluator; MATH uses trusted regrading or explicitly marked answer\-grounded checks; GPQA compares extracted option letters with dataset labels\. The LiveCodeBench main table uses a three\-seed all\-tier common task set with task\-cluster hierarchical bootstrap confidence intervals \(resampling tasks, keeping seed observations within clusters\)\. Descriptive boundary analyses \(GPQA\) report per\-seed ranges and explicit denominators\. Before running the full ladder we preregistered: \(i\) 8–14% recoverable mass for the Qwen3\.6\-35B pool; \(ii\) public/execution selection beats generated\-test selection; \(iii\) generated\-test selection beats natural\-language selection; \(iv\) cross\-model natural\-language selection does not stably approach execution selection; and \(v\) all mechanism results report coverage, quality, harm, fixed/harmed counts, confidence intervals, and common denominators\. Section[4](https://arxiv.org/html/2607.17531#S4)reports which expectations held\.

## 4Results

### 4\.1LiveCodeBench: Fidelity Explains the Gap between Test\-Based Selectors

Table[3](https://arxiv.org/html/2607.17531#S4.T3)shows the three\-seed ladder on the all\-tier common set \(969/960/959 task\-seed observations per seed; 2,888 total\)\.

Table 3:LiveCodeBench selector ladder on the three\-seed common set\.The oracle any@5 row is not deployable; it measures recoverable mass: 339/2888 first\-sample failures have a correct candidate\. This bound is large enough for selection to matter, but mechanisms capture it unevenly\.

The strongest contrast is between the two execution\-based verifiers:\+8\.14\+8\.14pp \(public tests\) versus\+2\.70\+2\.70pp \(generated tests\)\. We therefore measure fidelity at candidate level\. On the all\-tier common candidate slice \(14,440 candidates\), the public\-test verifier reaches 92\.98% accuracy and MCC 0\.825; the generated\-test verifier reaches 53\.05% and MCC 0\.248 \(same ordering on the full slice\)\. Figure[2](https://arxiv.org/html/2607.17531#S4.F2)visualizes the ladder and the fidelity/risk profile\. The L4\-public/L4\-gen gap is not simply “tests help”; it is a fidelity gap between two test signals\.

![Refer to caption](https://arxiv.org/html/2607.17531v1/figures/figure1_lcb_fidelity_risk.png)Figure 2:LiveCodeBench selector performance and verifier fidelity/risk profile on the common slice\. Public tests capture substantially more of the oracle gap than generated tests; generated tests remain conservative and low\-harm\. The hatched, dashed oracle bar is a non\-deployable upper bound, visually separated from deployable mechanisms\.The natural\-language results are more nuanced\. L1 is numerically higher than L4\-gen by\+0\.80\+0\.80pp, but the direct paired contrast crosses zero \(95% CI\[−0\.38,2\.01\]\[\-0\.38,2\.01\]\): the preregistered prediction that generated tests beat a same\-family LLM selector is not supported\. The two mechanisms occupy different risk points, however: L1 captures 199/339 recoverable cases but harms 98/2089 first\-sample\-correct cases \(4\.69%\); L4\-gen captures 79/339 on the clean decomposition slice with 2/2089 harm \(0\.10%\)\. The low harm of L4\-gen is partly structural: generated tests exist for 2444/3165 = 77\.22% of task\-seed rows, and at least one candidate passes them on only 1132/3165 = 35\.77%; a raw trace audit \(2705 available rows\) records 1736 fallback\-to\-sample0 decisions, including 618 rows with no generated tests, versus 969 active selections\. The conservative fallback protects correct first samples but misses many recoverable cases\.

Table[4](https://arxiv.org/html/2607.17531#S4.T4)instantiates the clean\-slice accounting with the availability audit\. Capture and harm are*effective*quantities after each mechanism’s fallback policy, not the conditional terms of Section[3\.3](https://arxiv.org/html/2607.17531#S3.SS3); the fixed\-minus\-harmed accounting identity closes exactly on the clean slice \(2,887 rows; one task\-seed with inconsistent timeout/harness labels is excluded from the decomposition slice only\)\.

Table 4:Clean\-slice accounting and signal availability\.Signal availability binds, especially for generated tests\. The L4\-gen gap has two separable sources: coverage/activation loss \(rows falling back without active test evidence\) and fidelity loss \(tests disagreeing with official labels when they exist\)\. This is why the framework keeps coverage, fidelity, effective capture, and harm separate\.

Table[5](https://arxiv.org/html/2607.17531#S4.T5)reports the preregistered scorecard; it is mixed, which is exactly why the decomposition is useful\.

Table 5:Preregistered LiveCodeBench scorecard\.Recoverable mass \(8–14%\)Public\>\>generatedGenerated\>\>natural languageL3 not near executionFull reportingPooled 11\.74%Yes\+8\.14\+8\.14vs\.\+2\.70\+2\.70ppYesBeats L3, not L1No; CI overlaps 0\+1\.97\+1\.97vs\.\+8\.14\+8\.14ppYesTables[3](https://arxiv.org/html/2607.17531#S4.T3)–[4](https://arxiv.org/html/2607.17531#S4.T4)Yes
### 4\.2MATH: Symbolic Answer Equivalence Beats Self\-Consistency

MATH offers a different decidable signal: final answers can be normalized or checked for symbolic equivalence\. Table[6](https://arxiv.org/html/2607.17531#S4.T6)reports 750 task\-seed observations \(250 tasks×\\times3 seeds\) under trusted per\-task regrading\.

Table 6:MATH selector results under trusted regrading\.L4\-symbolic should be read carefully: it is not a problem\-grounded verifier that proves candidates from the question\. It clusters final answers bymath\_verifyequivalence and selects from the strongest class—518/750 = 69\.07% of rows use an equivalence key, while 232/750 = 30\.93% fall back to the first sample\. The result shows that symbolic answer equivalence is a stronger aggregation baseline than surface normalized\-answer SC, not that an arbitrary extra verifier beats SC\.

This result does not imply that every explicit math verifier beats self\-consistency\. A separate L4\-grounded artifact, evaluated under its own label source, selects 419/750 correct answers while SC selects 420/750\. It fixes no SC failures and harms one SC success, for−0\.13\-0\.13pp relative to SC under that artifact\. We therefore treat L4\-grounded as evidence that a weak grounding signal may fail to exceed SC, while the trusted L4\-symbolic result is the formal positive result\. Together, the MATH results reinforce the framework’s main distinction: an extra natural\-language judge is not enough, but a high\-fidelity answer\-equivalence signal can convert recoverable mass into gain\.

### 4\.3GPQA\-Diamond: The Oracle Gap Is Too Small to Pay for Harm

GPQA\-Diamond answers normalize to option letters, but correctness is not execution\-checkable\. Table[7](https://arxiv.org/html/2607.17531#S4.T7)reports the three\-seed boundary on the Qwen3\.6\-35B pool\.

Table 7:GPQA\-Diamond boundary on the 35B pool\.*Note:*Across the three seeds, row\-wise accuracy ranges are\[45\.96,48\.99\]\[45\.96,48\.99\],\[43\.94,47\.98\]\[43\.94,47\.98\],\[43\.94,47\.98\]\[43\.94,47\.98\],\[43\.94,46\.97\]\[43\.94,46\.97\], and\[49\.49,52\.53\]\[49\.49,52\.53\], respectively\.

The oracle gap is only 18/594 = 3\.03pp: a perfect selector could add about three points\. The raw pools explain why: 520/594 pools \(87\.54%\) contain five identical answer letters \(mean unique letters 1\.138\)\. SC is 10/594 below the first sample under the pipeline’s tie\-break rule\. LLM selectors are net\-negative, and predictably so: L1 captures 5/18 recoverable cases but harms 15/283 first\-sample\-correct cases \(5\.30% harm; L3: 6/18, 7\.07%\)\. The cost side is roughly first\-sample\-correct mass times harm rate—for L1, 15/594 = 2\.53pp, already exceeding what five recovered cases can compensate\. L1 and SC tie at 273/594 by cancellation, not degeneration: they disagree on 20 observations, each uniquely correct on 10\. A weaker Qwen3\.5\-9B pool shrinks the boundary further \(recoverable 0\.67%, 94\.44% identical pools\): the oracle gap is a joint property of task, model, and sampling configuration, not a benchmark constant\.

### 4\.4Summary

The same diagnostic pattern appears across benchmarks—and signal availability binds across them as well \(35\.77% active generated\-test rows on LiveCodeBench; 69\.07% equivalence\-key rows on MATH\)\. On LiveCodeBench, substantial recoverable mass exists and high\-fidelity public tests capture much of it; generated tests are positive but limited by low fidelity and low activation\. On MATH, symbolic answer equivalence beats SC while LLM selectors are negative for L1 and marginal for L3\. On GPQA\-Diamond, the pool has too little recoverable mass for LLM selector harm to be worthwhile\. Training\-free collaboration is not intrinsically useful because it adds another agent; it is useful when the pool contains recoverable answers and the signal has enough coverage, fidelity, quality, and low harm\.

## 5Discussion and Limitations

### 5\.1Collaboration Gain Is Not a Topology Property

The LiveCodeBench contrast among L4\-public, L4\-gen, L1, and L3 shows that gains do not arise from adding a judging layer by itself\. Public tests are strongest, but the deployable generated\-test verifier is not statistically distinguishable from L1 in this setting\. The relevant axis is not simply test\-based versus natural\-language selection; it is the joint effect of signal fidelity, coverage, conditional quality, fallback behavior, and harm\. The large L4\-public–L4\-gen gap identifies generated\-test fidelity and activation as the current bottlenecks\.

### 5\.2Best\-of\-NNSupplies Candidate Space

Best\-of\-NNand self\-consistency are not outside collaboration\. They create the candidate pool; selectors and verifiers determine whether that pool is converted into a better final answer\. Without an oracle gap, no fixed\-pool selector can help\. With an oracle gap but weak verification, the available improvement remains uncaptured\. This two\-stage view separates candidate generation from selection and makes clear why agent count or sampling budget alone is not an explanation of realized gain\.

### 5\.3Scope and Limitations

First, LiveCodeBench selected\-output evaluation and stored candidate oracle labels contain a small timeout/harness inconsistency\. We identifyseed43/lcb\_abc323\_das the responsible task\-seed and exclude it only from the clean decomposition slice, while retaining official selected\-output evaluation in the main pass@1 table\.

Second, candidate\-level fidelity is fully measured for L4\-public and L4\-gen, not for the L1/L3 natural\-language selectors\. The L3 availability audit also covers only the available trace subset rather than the full selected\-output slice\. Adding LLM verdict matrices would support a broader fidelity comparison; we leave this extension for future work rather than treating final pass@1 as a substitute for signal fidelity\.

Third, L4\-public uses task\-visible tests and should be read as a public\-test verifier or partial\-oracle diagnostic upper bound, not as a fully deployable mechanism\. The current overlap audit is a 50\-task sample rather than a complete 1,055\-task non\-overlap proof\.

Fourth, MATH contains two mechanism and label\-source regimes\. The trusted L4\-symbolic per\-task selector is the formal positive result\. L4\-grounded uses a separate grounded\-summary label source and is interpreted separately rather than merged into the same horizontal comparison\.

Fifth, GPQA\-Diamond establishes a low\-recoverable boundary for the studied Qwen3\.6\-35B pool, not for every natural\-language knowledge task\. The oracle gap is a joint property of the task, model, prompt, temperature, and sampling configuration; more diverse sampling could change the available recoverable mass\.

Sixth, this paper studies selector and verifier mechanisms, not repair\. Repair critics can create answers outside the fixed pool, but they introduce a distinct detection–generation trade\-off and a different harm profile\.

Seventh, we do not claim compute optimality\. All main selector results usek=5k=5candidate pools, and the LiveCodeBench token\-cost proxy was not recomputed on the final all\-tier common set\. Earlier probes suggest comparable gain per selector token for L4\-gen and L1, with lower efficiency for L3, but those are appendix\-level diagnostics rather than a compute\-matched conclusion\. A deployment decision should compare verifier calls with larger\-kksampling, normalized self\-consistency where applicable, and execution cost\(Snellet al\.,[2024](https://arxiv.org/html/2607.17531#bib.bib14); Singhiet al\.,[2025](https://arxiv.org/html/2607.17531#bib.bib19)\)\.

Finally, the diagnostic requires labels on a development set\. Estimating oracle gap, fidelity, quality, and harm is not label\-free in the form used here; it is intended as a small labeled audit before investing in a collaboration mechanism\. GPQA illustrates the potential sample efficiency of that audit: 594 task\-seed observations are sufficient to identify a 3\.03pp oracle gap and predict net\-negative LLM selection\. Label\-free reliability estimation for generated tests is a promising substitute when development labels are unavailable\.

## 6Conclusion

Training\-free collaboration is not valuable merely because it adds an agent or critic\. In fixed\-pool selection, recoverable mass bounds gain; signal coverage, fidelity, quality, and harm determine how much is realized\. High\-fidelity public tests capture substantial oracle space on LiveCodeBench, symbolic equivalence beats self\-consistency on MATH, and LLM selectors become net\-negative on homogeneous GPQA\-Diamond pools with a 3\.03pp oracle gap\. The deployment rule is therefore: estimate oracle gap first, then audit signal availability, fidelity, and harm before spending additional inference compute\.

#### Reproducibility Statement

The paper reports task\-set denominators, seed counts, model provenance, label sources, selection rules, fixed/harmed counts, and confidence\-interval procedures\. The supplementary material documents the routing extension and additional audit details\. Code and artifacts, including the run\-label registry, selected\-output records, candidate verdicts, and analysis scripts, are available at[https://github\.com/AmGarfield/OracleGap](https://github.com/AmGarfield/OracleGap)\.

## References

- B\. Brown, J\. Juravsky, R\. Ehrlich, R\. Clark, Q\. V\. Le, C\. Ré, and A\. Mirhoseini \(2024\)Large language monkeys: scaling inference compute with repeated sampling\.arXiv preprint arXiv:2407\.21787\.External Links:[Link](https://arxiv.org/abs/2407.21787)Cited by:[§2\.1](https://arxiv.org/html/2607.17531#S2.SS1.p1.1)\.
- M\. Cemri, M\. Z\. Pan, S\. Yang, L\. A\. Agrawal, B\. Chopra, R\. Tiwari, K\. Keutzer, A\. Parameswaran, D\. Klein, K\. Ramchandran, M\. Zaharia, J\. E\. Gonzalez, and I\. Stoica \(2025\)Why do multi\-agent LLM systems fail?\.arXiv preprint arXiv:2503\.13657\.External Links:[Link](https://arxiv.org/abs/2503.13657)Cited by:[§2\.3](https://arxiv.org/html/2607.17531#S2.SS3.p2.1)\.
- B\. Chen, F\. Zhang, A\. Nguyen, D\. Zan, Z\. Lin, J\. Lou, and W\. Chen \(2022\)CodeT: code generation with generated tests\.arXiv preprint arXiv:2207\.10397\.External Links:[Link](https://arxiv.org/abs/2207.10397)Cited by:[§2\.1](https://arxiv.org/html/2607.17531#S2.SS1.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.External Links:[Link](https://arxiv.org/abs/2110.14168)Cited by:[§2\.2](https://arxiv.org/html/2607.17531#S2.SS2.p1.1)\.
- F\. E\. Dorner, Y\. Chen, A\. F\. Cruz, and F\. Yang \(2025\)ROC\-n\-reroll: how verifier imperfection affects test\-time scaling\.arXiv preprint arXiv:2507\.12399\.External Links:[Link](https://arxiv.org/abs/2507.12399)Cited by:[§2\.2](https://arxiv.org/html/2607.17531#S2.SS2.p2.1)\.
- Y\. Du, S\. Li, A\. Torralba, J\. B\. Tenenbaum, and I\. Mordatch \(2023\)Improving factuality and reasoning in language models through multiagent debate\.arXiv preprint arXiv:2305\.14325\.External Links:[Link](https://arxiv.org/abs/2305.14325)Cited by:[§2\.3](https://arxiv.org/html/2607.17531#S2.SS3.p2.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the MATH dataset\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks,Cited by:[§1](https://arxiv.org/html/2607.17531#S1.p5.1)\.
- Z\. Hu, C\. Liu, T\. Huang, J\. Li, Y\. Yang, J\. Wu, Z\. Zhong, M\. Yang, and Y\. Yue \(2026\)CoSPlay: cooperative self\-play at test\-time with self\-generated code and unit test\.arXiv preprint arXiv:2605\.23491\.External Links:[Link](https://arxiv.org/abs/2605.23491)Cited by:[§2\.4](https://arxiv.org/html/2607.17531#S2.SS4.p1.1)\.
- J\. Huang, X\. Chen, S\. Mishra, H\. S\. Zheng, A\. W\. Yu, X\. Song, and D\. Zhou \(2024\)Large language models cannot self\-correct reasoning yet\.arXiv preprint arXiv:2310\.01798\.External Links:[Link](https://arxiv.org/abs/2310.01798)Cited by:[§2\.3](https://arxiv.org/html/2607.17531#S2.SS3.p2.1)\.
- N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica \(2024\)LiveCodeBench: holistic and contamination free evaluation of large language models for code\.arXiv preprint arXiv:2403\.07974\.External Links:[Link](https://arxiv.org/abs/2403.07974)Cited by:[§1](https://arxiv.org/html/2607.17531#S1.p5.1)\.
- Y\. Kim, S\. Lee, C\. Jang, H\. Lee, and J\. Lee \(2026\)Parallel test\-time scaling with multi\-sequence verifiers\.arXiv preprint arXiv:2603\.03417\.External Links:[Link](https://arxiv.org/abs/2603.03417)Cited by:[§2\.2](https://arxiv.org/html/2607.17531#S2.SS2.p1.1)\.
- Y\. Kim, K\. Gu, C\. Park, C\. Park, S\. Schmidgall, A\. A\. Heydari, Y\. Yan, Z\. Zhang, Y\. Zhuang, Y\. Liu, M\. Malhotra, P\. P\. Liang, H\. W\. Park, Y\. Yang, X\. Xu, Y\. Du, S\. Patel, T\. Althoff, D\. McDuff, and X\. Liu \(2025\)Towards a science of scaling agent systems\.arXiv preprint arXiv:2512\.08296\.External Links:[Link](https://arxiv.org/abs/2512.08296)Cited by:[§2\.3](https://arxiv.org/html/2607.17531#S2.SS3.p2.1)\.
- J\. Kwok, S\. Li, P\. Atreya, Y\. Liu, Y\. Jiang, C\. Finn, M\. Pavone, I\. Stoica, and A\. Mirhoseini \(2026\)LLM\-as\-a\-Verifier: a general\-purpose verification framework\.arXiv preprint arXiv:2607\.05391\.External Links:[Link](https://arxiv.org/abs/2607.05391)Cited by:[§2\.2](https://arxiv.org/html/2607.17531#S2.SS2.p1.1)\.
- D\. Lee, C\. Hwang, and K\. Lee \(2026\)Learning to generate unit test via adversarial reinforcement learning\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2508.21107)Cited by:[§2\.4](https://arxiv.org/html/2607.17531#S2.SS4.p1.1)\.
- Y\. Li, D\. Choi, J\. Chung, N\. Kushman, J\. Schrittwieser, R\. Leblond, T\. Eccles, J\. Keeling, F\. Gimeno, A\. Dal Lago, T\. Hubert, P\. Choy, C\. de Masson d’Autume, I\. Babuschkin, X\. Chen, P\. Huang, J\. Welbl, S\. Gowal, A\. Cherepanov, J\. Molloy, D\. J\. Mankowitz, E\. Robson, P\. Kohli, N\. de Freitas, K\. Kavukcuoglu, and O\. Vinyals \(2022\)Competition\-level code generation with AlphaCode\.Science378\(6624\),pp\. 1092–1097\.External Links:[Document](https://dx.doi.org/10.1126/science.abq1158)Cited by:[§2\.1](https://arxiv.org/html/2607.17531#S2.SS1.p1.1)\.
- S\. Lifshitz, S\. A\. McIlraith, and Y\. Du \(2025\)Multi\-agent verification: scaling test\-time compute with multiple verifiers\.arXiv preprint arXiv:2502\.20379\.External Links:[Link](https://arxiv.org/abs/2502.20379)Cited by:[§2\.2](https://arxiv.org/html/2607.17531#S2.SS2.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2024\)Let’s verify step by step\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2305.20050)Cited by:[§2\.2](https://arxiv.org/html/2607.17531#S2.SS2.p1.1)\.
- J\. Liu, C\. S\. Xia, Y\. Wang, and L\. Zhang \(2023\)Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1](https://arxiv.org/html/2607.17531#S1.p5.1)\.
- J\. Lu, R\. Teehan, J\. Jin, and M\. Ren \(2025\)When does verification pay off? a closer look at LLMs as solution verifiers\.arXiv preprint arXiv:2512\.02304\.External Links:[Link](https://arxiv.org/abs/2512.02304)Cited by:[§2\.2](https://arxiv.org/html/2607.17531#S2.SS2.p3.1)\.
- A\. Prasad, E\. Stengel\-Eskin, J\. C\. Chen, Z\. Khan, and M\. Bansal \(2025\)Learning to generate unit tests for automated debugging\.InConference on Language Modeling,External Links:[Link](https://arxiv.org/abs/2502.01619)Cited by:[§2\.4](https://arxiv.org/html/2607.17531#S2.SS4.p1.1)\.
- D\. Qiao, B\. Chen, F\. Cai, J\. Chen, W\. Li, F\. Jiang, Z\. Chen, H\. Zha, T\. Zhang, and B\. Wang \(2026\)Epistemic gain, aleatoric cost: uncertainty decomposition in multi\-agent debate for math reasoning\.arXiv preprint arXiv:2603\.01221\.External Links:[Link](https://arxiv.org/abs/2603.01221)Cited by:[§2\.3](https://arxiv.org/html/2607.17531#S2.SS3.p1.1)\.
- D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman \(2024\)GPQA: a graduate\-level google\-proof Q&A benchmark\.InFirst Conference on Language Modeling,External Links:[Link](https://arxiv.org/abs/2311.12022)Cited by:[§1](https://arxiv.org/html/2607.17531#S1.p5.1)\.
- A\. Setlur, N\. Rajaraman, S\. Levine, and A\. Kumar \(2025\)Scaling test\-time compute without verification or RL is suboptimal\.arXiv preprint arXiv:2502\.12118\.External Links:[Link](https://arxiv.org/abs/2502.12118)Cited by:[§2\.2](https://arxiv.org/html/2607.17531#S2.SS2.p2.1)\.
- F\. Shi, D\. Fried, M\. Ghazvininejad, L\. Zettlemoyer, and S\. I\. Wang \(2022\)Natural language to code translation with execution\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 3533–3546\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.231),[Link](https://aclanthology.org/2022.emnlp-main.231/)Cited by:[§2\.1](https://arxiv.org/html/2607.17531#S2.SS1.p1.1)\.
- N\. Singhi, H\. Bansal, A\. Hosseini, A\. Grover, K\. Chang, M\. Rohrbach, and A\. Rohrbach \(2025\)When to solve, when to verify: compute\-optimal problem solving and generative verification for LLM reasoning\.arXiv preprint arXiv:2504\.01005\.External Links:[Link](https://arxiv.org/abs/2504.01005)Cited by:[§2\.2](https://arxiv.org/html/2607.17531#S2.SS2.p2.1),[§5\.3](https://arxiv.org/html/2607.17531#S5.SS3.p7.2)\.
- C\. Snell, J\. Lee, K\. Xu, and A\. Kumar \(2024\)Scaling LLM test\-time compute optimally can be more effective than scaling model parameters\.arXiv preprint arXiv:2408\.03314\.External Links:[Link](https://arxiv.org/abs/2408.03314)Cited by:[§2\.1](https://arxiv.org/html/2607.17531#S2.SS1.p1.1),[§5\.3](https://arxiv.org/html/2607.17531#S5.SS3.p7.2)\.
- B\. Stroebl, S\. Kapoor, and A\. Narayanan \(2024\)The limits of inference scaling through resampling\.arXiv preprint arXiv:2411\.17501\.External Links:[Link](https://arxiv.org/abs/2411.17501)Cited by:[§2\.2](https://arxiv.org/html/2607.17531#S2.SS2.p2.1)\.
- D\. Tran and D\. Kiela \(2026\)Single\-agent LLMs outperform multi\-agent systems on multi\-hop reasoning under equal thinking token budgets\.arXiv preprint arXiv:2604\.02460\.External Links:[Link](https://arxiv.org/abs/2604.02460)Cited by:[§2\.3](https://arxiv.org/html/2607.17531#S2.SS3.p2.1)\.
- J\. Uesato, N\. Kushman, R\. Kumar, H\. F\. Song, N\. Siegel, L\. Wang, A\. Creswell, G\. Irving, and I\. Higgins \(2022\)Solving math word problems with process\- and outcome\-based feedback\.arXiv preprint arXiv:2211\.14275\.External Links:[Link](https://arxiv.org/abs/2211.14275)Cited by:[§2\.2](https://arxiv.org/html/2607.17531#S2.SS2.p1.1)\.
- V\. Venktesh, M\. Rathee, and A\. Anand \(2025\)Trust but verify\! a survey on verification design for test\-time scaling\.arXiv preprint arXiv:2508\.16665\.External Links:[Link](https://arxiv.org/abs/2508.16665)Cited by:[§2\.2](https://arxiv.org/html/2607.17531#S2.SS2.p2.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023\)Self\-consistency improves chain of thought reasoning in language models\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2203.11171)Cited by:[§2\.1](https://arxiv.org/html/2607.17531#S2.SS1.p1.1)\.
- Y\. Yang, C\. Qu, M\. Wen, L\. Shi, Y\. Wen, W\. Zhang, A\. Wierman, and S\. Gu \(2026\)Understanding agent scaling in LLM\-based multi\-agent systems via diversity\.arXiv preprint arXiv:2602\.03794\.External Links:[Link](https://arxiv.org/abs/2602.03794)Cited by:[§2\.3](https://arxiv.org/html/2607.17531#S2.SS3.p1.1)\.
- Y\. Zhao, S\. Chen, and N\. Su \(2026\)On the Uncertainty of Large Language Model\-Based Multi\-Agent Systems\.arXiv preprint arXiv:2602\.04234\.External Links:[Link](https://arxiv.org/abs/2602.04234)Cited by:[§2\.3](https://arxiv.org/html/2607.17531#S2.SS3.p1.1)\.

## Appendix ASupplementary Results

### A\.1HumanEval\+ Saturation Anchor

HumanEval\+ serves only as a saturated code anchor\. In the Qwen3\.6\-35B V100 follow\-up, first\-sample pass@1 is 138/164, 139/164, and 138/164 across seeds 43/44/45, while oracle any@5 is 144/164, 144/164, and 142/164\. The corresponding recoverable counts are only 6/164, 5/164, and 4/164, so HumanEval\+ is not used as the main code selector benchmark\.

### A\.2Routing\-η\\etaExtension

Worker routing chooses which model should answer an input rather than selecting among candidates from a fixed pool\. We define

η=score​\(learned​router\)−score​\(default​worker\)score​\(oracle​router\)−score​\(default​worker\)\.\\eta=\\frac\{\\mathrm\{score\}\(\\mathrm\{learned\\ router\}\)\-\\mathrm\{score\}\(\\mathrm\{default\\ worker\}\)\}\{\\mathrm\{score\}\(\\mathrm\{oracle\\ router\}\)\-\\mathrm\{score\}\(\\mathrm\{default\\ worker\}\)\}\.\(7\)Anη\\etanear one means that the learned router captures most of the oracle routing space; anη\\etanear zero means it does not exploit the available worker differences\. Table[8](https://arxiv.org/html/2607.17531#A1.T8)summarizes the strongest\-feature runs\. These results are descriptive and are not part of the selector\-ladder evidence because the default\-worker identities and confidence\-interval pipeline are not integrated into the paper\-level provenance system\.

Table 8:Routing\-η\\etaextension\.The LCB logistic router degenerates to selecting the default worker, yieldingη=0\\eta=0\. The remaining results show that input\-level routing can recover some oracle\-routing space, but this depends on worker differences and feature predictability rather than candidate\-level verification\. Candidate selection and worker routing are both test\-time collaboration mechanisms, but they obey different constraints\.

### A\.3Additional Audit Details

One task\-seed observation,seed43/lcb\_abc323\_d, is excluded from the clean decomposition slice because selected\-output evaluation marks it as passing while all stored candidate oracle labels fail with timeout or harness codes\. The main pass@1 table retains the official selected\-output evaluation\. The raw generated\-test trace audit contains 2,705 available rows, versus 2,887 rows in the clean decomposition slice, because the seed\-45 trace artifact is incomplete; final pass@1 uses the official selected\-output details\.

The public\-test verifier should be interpreted as a partial\-oracle diagnostic upper bound, not a fully deployable mechanism\. The current overlap audit covers a 50\-task sample rather than proving non\-overlap for all 1,055 LiveCodeBench tasks\. The MATH grounded artifact uses a separate label source and is therefore not merged with the trusted symbolic\-equivalence result\.

### A\.4LLM Usage Disclosure

An LLM\-based assistant was used for language editing and preparation of theLaTeXmanuscript\. The author reviewed the resulting text, equations, tables, citations, and numerical claims and retains full responsibility for the submission\.