硬预算重复评估中诚实不确定性的精确界限

# Sharp Limits for Honest Uncertaintyin Hard-Budget Repeated Evaluation
Source: [https://arxiv.org/html/2609.29140](https://arxiv.org/html/2609.29140)
###### Abstract

Repeated evaluation can estimate a benchmark score accurately while still requiring replication to certify narrow uncertainty\. We characterize that requirement on a fixed grid ofMMtasks withLLbinary paths per task under the hard budget\(M\+t\)​K\(M\+t\)K, where each path costs at mostKKresponses or episodes\. For fixedL≥3L\\geq 3and0<α≤1/120<\\alpha\\leq 1/12, the optimal expected width on the worst pure cohort, where planned labels agree within each task, isΘα,L\(\[M\(t\+1\)\]−1/2\)\\Theta\_\{\\alpha,L\}\(\[M\(t\+1\)\]^\{\-1/2\}\)when every task is observed andΘα,L\(\[M\(t\+M\)\]−1/2\)\\Theta\_\{\\alpha,L\}\(\[M\(t\+\\sqrt\{M\}\)\]^\{\-1/2\}\)when omission is allowed\. The lower bounds cover adaptive hard\-budget policies; fixed random\-subset designs attain both rates through disagreement certificates\. A joint mean/disagreement interval turns the task\-covering law into practical finite\-budget inference\. In an equal\-budget LiveCodeBench replay with 16 models, 880 tasks, and five outputs per task, Joint and exact pooled uniform inference each use 1100 evaluated outputs\. Joint is narrower in 15/16 panels, with median reductions of 30\.6% in mean interval width and 87\.0% in MSE\. Finite\-regime analyses further identify how task coverage and within\-task agreement govern the useful operating region\. Together, the sharp laws and fixed\-budget evidence make replication an explicit design variable for information\-efficient language\-model and agent evaluation\.

## 1Introduction

Repeated evaluation allocates a finite budget across tasks and repeated runs of each task\. This allocation matters for language\-model inference scaling\([Brown et al\., 2024](https://arxiv.org/html/2609.29140#bib.bib2);[Kazdan et al\., 2025](https://arxiv.org/html/2609.29140#bib.bib3)\)and agent benchmarks, where repeated runs can differ even under nominally deterministic decoding\([Bjarnason et al\., 2026](https://arxiv.org/html/2609.29140#bib.bib4)\)\. Task stratification can make a point estimate highly accurate: when every planned label within a task agrees, one observation per task recovers the grid mean exactly\. Certifying that accuracy, however, requires evidence about the unobserved repetitions\. This separation between estimation and certification creates a concrete budget\-design problem\.

For example, suppose an evaluator fixes 880 coding tasks, a model checkpoint, a decoding configuration, and five run seeds per task before evaluation\. A budget permits evaluating only 1100 of the 4400 prespecified task–run slots, and the model advances to the next evaluation stage only if its score on this grid clears a qualification thresholdτ\\tau\. A lower endpoint aboveτ\\tausupports advancing the model, an upper endpoint belowτ\\tausupports holding it back, and an interval crossingτ\\tausignals that more repetitions are needed\. A fixed\-cohort honest interval remains valid for this decision even when tasks and runs are heterogeneous or dependent; generalization to future tasks answers a different question\.

We ask:*how much replication is necessary and sufficient for narrow honest intervals under a hard evaluation budget, and how does the answer change if tasks may be omitted?*We study the mean of a prespecified finite grid of potential runs, conditional on its realized outcomes, and require coverage for every such grid\. Each path contains at mostKKresponses or episodes, and the budget holds for every outcome and policy randomization\. The policy class is broad: it may recycle savings from early decisive events, interleave tasks, and choose observations adaptively\.

Our central contribution is a pair of matching certification laws that expose how replication budget and task coverage jointly determine honest interval width\. For fixedL≥3L\\geq 3, requiring at least one completed path per task gives the optimal worst\-pure\-cohort rateΘα,L\(\[M\(t\+1\)\]−1/2\)\\Theta\_\{\\alpha,L\}\(\[M\(t\+1\)\]^\{\-1/2\}\)\(Theorem[3](https://arxiv.org/html/2609.29140#Thmtheorem3)\); allowing omission givesΘα,L\(\[M\(t\+M\)\]−1/2\)\\Theta\_\{\\alpha,L\}\(\[M\(t\+\\sqrt\{M\}\)\]^\{\-1/2\}\)\(Theorem[4](https://arxiv.org/html/2609.29140#Thmtheorem4)\)\. At zero extra reservation, the latter changes the attainable order fromM−1/2M^\{\-1/2\}toM−3/4M^\{\-3/4\}\. Fixed random\-subset designs attain both rates through disagreement\-based certificates, so the theory yields executable evaluation policies rather than lower bounds alone\.

The finite\-budget contribution couples the sampled mean and disagreement count in a Joint interval\. AtK=1K=1, an equal\-budget LiveCodeBench replay gives Joint and exact pooled uniform inference the same 1100\-output budget \(M=880,t=220M=880,t=220\)\. Joint produces narrower mean intervals in 15/16 model panels, with median reductions of 30\.6% in width and 87\.0% in MSE\. A sufficient dominance region and controlled departures from within\-task agreement then connect the sharp laws to finite\-cohort behavior\.

The paper contributes three mutually supporting results:

- •matching lower and upper rates for task\-covering and omission\-enabled hard\-budget policies, including adaptive policies in both lower bounds;
- •fixed randomized designs and disagreement\-aware intervals that realize the theoretical rates and sharpen finite\-budget inference; and
- •equal\-budget evidence that identifies a useful LiveCodeBench operating regime and boundary conditions observed across benchmarks\.

## 2Related work

#### Repeated and adaptive evaluation\.

[Brown et al\. \(2024\)](https://arxiv.org/html/2609.29140#bib.bib2)study inference scaling with repeated sampling, and[Kazdan et al\. \(2025\)](https://arxiv.org/html/2609.29140#bib.bib3)predict pass@KKscaling using a model for task\-level heterogeneity\.[Bjarnason et al\. \(2026\)](https://arxiv.org/html/2609.29140#bib.bib4)analyze repeated coding\-agent runs\.[Huang \(2026\)](https://arxiv.org/html/2609.29140#bib.bib25)replay partial agent benchmarks to study pairwise decisions, task\-group coverage and deferral\. We complement these empirical objectives with honest fixed\-cohort intervals for the repeated\-evaluation mean under an exact pathwise budget\.[Pilditch \(2026\)](https://arxiv.org/html/2609.29140#bib.bib6)develop Bayesian precision\-based stopping and explicitly distinguish posterior precision from frequentist coverage\. Our design\-based guarantee supplies the corresponding conditional statement for a prespecified benchmark grid and requires no model for task or run generation\.

#### Active inference and stratification\.

Active statistical inference\([Zrnic and Candès, 2024](https://arxiv.org/html/2609.29140#bib.bib7)\), cost\-aware evaluation\([Angelopoulos et al\., 2025](https://arxiv.org/html/2609.29140#bib.bib8)\), stratified prediction\-powered inference\([Fisch et al\., 2024](https://arxiv.org/html/2609.29140#bib.bib9)\), and finite\-population adaptive evaluation\([Wu et al\., 2026](https://arxiv.org/html/2609.29140#bib.bib10)\)already use evaluation information to improve statistical efficiency\.[Yauney et al\. \(2026\)](https://arxiv.org/html/2609.29140#bib.bib11)show why strong random micro\-benchmark baselines matter\. Our setting adds partially observed repetition paths and an exact total response cap, without a learned rater or externally labeled pilot set\. Stratified empirical Bernstein inference\([Burgess and Chapman, 2021](https://arxiv.org/html/2609.29140#bib.bib20)\)and sequential union\-of\-intersections inference\([Spertus et al\., 2024](https://arxiv.org/html/2609.29140#bib.bib21)\)already provide variance\-aware alternatives\. One\-per\-stratum variance problems and replication remedies are classical\([Barabesi et al\., 2012](https://arxiv.org/html/2609.29140#bib.bib23)\), as are variance\-sensitive bounds\([Maurer and Pontil, 2009](https://arxiv.org/html/2609.29140#bib.bib24)\)and honest expected\-length adaptation\([Cai and Low, 2004](https://arxiv.org/html/2609.29140#bib.bib22)\)\. Building on these tools, we derive the matching jointM,tM,trate first with task coverage and then over all hard\-budget policies, and attain both laws with fixed random\-subset designs \(Appendices[K](https://arxiv.org/html/2609.29140#A11)and[P](https://arxiv.org/html/2609.29140#A16)\)\.

#### Random continuation and sequential uncertainty\.

Randomly abandoning an expensive simulation and reweighting completed outcomes predates language\-model evaluation; Lazy ABC is a direct example\([Prangle, 2014](https://arxiv.org/html/2609.29140#bib.bib12)\)\. DAPRO\([Feldman and Romano, 2026](https://arxiv.org/html/2609.29140#bib.bib13)\)is especially close: its Appendix F\.6 includes unbiased population event\-rate estimation under dynamic continuation, with budget guarantees stated in expectation\. We instead study fixed\-cohort certification under a pathwise hard cap; fixed\-count reservation provides the budget guarantee while replication determines honest interval width\. Our recursive weights follow classical multiphase sampling\([Lumley, 2024](https://arxiv.org/html/2609.29140#bib.bib14)\)\. The uncertainty tools are likewise established: finite\-population confidence sequences\([Waudby\-Smith and Ramdas, 2020](https://arxiv.org/html/2609.29140#bib.bib15)\), normal\-mixture boundaries\([Howard et al\., 2021](https://arxiv.org/html/2609.29140#bib.bib16)\), and, in exploratory variants, profile e\-value inversion\([Wasserman et al\., 2020](https://arxiv.org/html/2609.29140#bib.bib17)\)\. The contribution is the resulting sharp replication law and its constructive realization under an exact evaluation budget\.

Table[1](https://arxiv.org/html/2609.29140#S2.T1)compares levels of guarantee rather than topic alone\.

Table 1:Positioning relative to the nearest guarantees\.

## 3A fixed\-cohort measurement problem

There areMMtasks andLLplanned repetition paths per task, forN=M​LN=MLpaths\. A path contains at mostKKresponses or completed agent episodes\. WriteTi​r∈\{1,…,K,K\+1\}T\_\{ir\}\\in\\\{1,\\ldots,K,K\+1\\\}for its first decisive\-event index, withK\+1K\+1meaning no decisive event\. Its terminal label isYi​r=𝟏\{Ti​r\>K\}Y\_\{ir\}=\\mathbf\{1\}\\\{T\_\{ir\}\>K\\\}and the target is

θ𝒞=1M​L​∑i=1M∑r=1LYi​r\.\\theta\_\{\\mathcal\{C\}\}=\\frac\{1\}\{ML\}\\sum\_\{i=1\}^\{M\}\\sum\_\{r=1\}^\{L\}Y\_\{ir\}\.\(1\)For a first\-success event,1−θ𝒞1\-\\theta\_\{\\mathcal\{C\}\}is the block empirical success\-at\-KKscore\. For a first\-failure event,θ𝒞\\theta\_\{\\mathcal\{C\}\}is the block empirical all\-KKsuccess score\. Blocks are prespecified and disjoint; this target retains the planned repetition paths rather than replacing them by the all\-subsets binomial U\-statistic\.

An evaluator may advance an unresolved path from observed depthaatobb, payingmin⁡\(Ti​r−a,b−a\)\\min\(T\_\{ir\}\-a,b\-a\)units\. It learns whether the path remains unresolved and its charged cost\. Already paid prefixes are reused\. The policy receives task identities and observed histories; future outcomes remain hidden\. A hard budget requiresC≤BC\\leq Bfor every allowed outcome table and every randomization\. One unit is one model response or one complete agent episode; token, dollar, tool\-call, and elapsed\-time costs are separate resource measures\.

#### What coverage means\.

All design guarantees have the formℙR​\{θ𝒞∈I⁡\(D\)∣𝒞\}≥1−α\\mathbb\{P\}\_\{R\}\\\{\\theta\_\{\\mathcal\{C\}\}\\in I\(D\)\\mid\\mathcal\{C\}\\\}\\geq 1\-\\alpha, for every fixed potential cohort𝒞\\mathcal\{C\}\. Outcomes may be heterogeneous or dependent; randomnessRRis the evaluator’s sampling design\. A population survival meanθ∗=M−1​∑iℙ⁡\(Ti\>K\)\\theta\_\{\*\}=M^\{\-1\}\\sum\_\{i\}\\mathbb\{P\}\(T\_\{i\}\>K\)is a different estimand\. Under independent planned paths, an additional Hoeffding term can bridge the two \(Appendix[D](https://arxiv.org/html/2609.29140#A4)\)\. Our primary target instead supports a budgeted estimate of a prespecified benchmark grid, with a coverage guarantee that does not assume a stochastic model for its outcomes\. Requiring one completed path per task ensures direct representation of every benchmark item; random\-subset auditing then spends the remaining reservation on certification\. This is a design choice that prioritizes full task representation\. The companion omission law quantifies the alternative, while population generalization to future tasks or runs remains a separate inferential objective\.

## 4Sharp certification laws

###### Theorem 1\(Late\-event hard\-budget limit\)\.

Letm=min⁡\{N,⌊B/K⌋\}m=\\min\\\{N,\\lfloor B/K\\rfloor\\\}\. In the access model above, the worst\-case MSE of any estimator is at least

\(N−m\)​\(N\+2\)6​N2​\(m\+2\)\.\\frac\{\(N\-m\)\(N\+2\)\}\{6N^\{2\}\(m\+2\)\}\.\(2\)IfB<KB<K, there are two indistinguishable cohorts with targets zero and one, so the worst\-case MSE is at least1/41/4\.

The construction places every decisive event atKKor later\. An incomplete path gives no terminal information, and every completed label costsKK\. A uniform prior on the number of positive paths gives Equation \([2](https://arxiv.org/html/2609.29140#S4.E2)\)\. Thus adaptive continuation cannot uniformly improve theK/BK/Border away from census\. Useful gains must exploit a favorable distribution of decisive\-event times, not just a different weighting formula\.

The next result separates an estimator’s actual error from an interval’s ability to establish that error\. It is a confidence\-length consequence of the classical one\-observation\-per\-stratum information problem\.

###### Theorem 2\(One completed path per task\)\.

SupposeL≥2L\\geq 2and writeq=⌊L/2⌋/Lq=\\lfloor L/2\\rfloor/L\. An evaluator completes exactly one uniformly sampled path per task and observes no other terminal labels\. Any additional observed prefixes have depth belowKK\. LetI⁡\(D\)I\(D\)be an interval with coverage at least1−α1\-\\alphafor every fixed cohort\. There exists a cohort in which every task’sLLterminal labels are identical, and one\-per\-task averaging is exact, yet𝔼R​\|I⁡\(D\)\|≥bM,q​\(α\)\\mathbb\{E\}\_\{R\}\|I\(D\)\|\\geq b\_\{M,q\}\(\\alpha\), where forZ∼Bin⁡\(M,q\)Z\\sim\\operatorname\{Bin\}\(M,q\),

bM,q​\(α\)=inf0≤w≤1,𝔼​w​\(Z\)≥1−2​α𝔼⁡\[\|Z/M−q\|​w​\(Z\)\]\.b\_\{M,q\}\(\\alpha\)=\\inf\_\{0\\leq w\\leq 1,\\,\\mathbb\{E\}w\(Z\)\\geq 1\-2\\alpha\}\\mathbb\{E\}\\bigl\[\|Z/M\-q\|w\(Z\)\\bigr\]\.\(3\)For0<α<1/20<\\alpha<1/2,MbM,q\(α\)→2q⁡\(1−q\)\{1−exp\[−z1−α2/2\]\}/2​π\\sqrt\{M\}\\,b\_\{M,q\}\(\\alpha\)\\to 2\\sqrt\{q\(1\-q\)\}\\\{1\-\\exp\[\-z\_\{1\-\\alpha\}^\{2\}/2\]\\\}/\\sqrt\{2\\pi\}, wherezuz\_\{u\}is the standard\-normaluu\-quantile\. Atα=\.05\\alpha=\.05, the constant is0\.29580\.2958for evenLLand0\.28980\.2898forL=5L=5\.

One prior makes each task internally constant with a Bernoulli\(qq\) label; a second makes exactly⌊L/2⌋\\lfloor L/2\\rfloorpaths positive\. One observed label per task has the same law under both priors, but the targets are respectively the observed mean andqq\. Honest coverage must accommodate both\. The theorem isolates the certification cost even when task\-balanced point estimation is exact; the matching interpolation laws below show how replication removes it\. Complete proofs appear in Appendix[A](https://arxiv.org/html/2609.29140#A1)\.

#### How much replication removes the obstruction?

Complete one uniform path in every task, then a second distinct uniform path in a uniform subsetSSof exactlytttasks\. ChooseS,tS,tindependently of all labels before observation\. LetAiA\_\{i\}average the one or two labels purchased in taskiiand setθ^=M−1​∑iAi\\widehat\{\\theta\}=M^\{\-1\}\\sum\_\{i\}A\_\{i\}\. No terminal savings are recycled\. This design reserves\(M\+t\)​K\(M\+t\)Kand is unbiased\. Let𝒫\\mathcal\{P\}be all internally constant task cohorts\. LetΠM,t\\Pi\_\{M,t\}contain all pathwise\-\(M\+t\)​K\(M\+t\)K\-budget policies that reveal at least one terminal label in every task on every cohort\. They may adapt task/path choices, interleave audits, stop, and recycle savings\. Forπ∈ΠM,t\\pi\\in\\Pi\_\{M,t\},ℐα​\(π\)\\mathcal\{I\}\_\{\\alpha\}\(\\pi\)contains intervals honest over*all*fixed cohorts\.

###### Theorem 3\(Sharp partial\-replication interpolation\)\.

For fixedL≥3L\\geq 3and0<α≤1/120<\\alpha\\leq 1/12, there are constants0<cα,L≤Cα,L<∞0<c\_\{\\alpha,L\}\\leq C\_\{\\alpha,L\}<\\infty, independent ofM,tM,t, such that

cα,LM⁡\(t\+1\)≤infπ∈ΠM,t,I∈ℐα​\(π\)sup𝒞∈𝒫𝔼R​\|I\|≤Cα,LM⁡\(t\+1\),0≤t≤M\.\\frac\{c\_\{\\alpha,L\}\}\{\\sqrt\{M\(t\+1\)\}\}\\leq\\inf\_\{\\pi\\in\\Pi\_\{M,t\},\\,I\\in\\mathcal\{I\}\_\{\\alpha\}\(\\pi\)\}\\sup\_\{\\mathcal\{C\}\\in\\mathcal\{P\}\}\\mathbb\{E\}\_\{R\}\|I\|\\leq\\frac\{C\_\{\\alpha,L\}\}\{\\sqrt\{M\(t\+1\)\}\},\\qquad 0\\leq t\\leq M\.\(4\)The fixed random\-subset audit attains the upper bound on every pure cohort\. The lower bound holds for every task\-covering policy, but is worst\-pure\-cohort, not pointwise over that class\.

Thust=⌊ρ​M⌋t=\\lfloor\\rho M\\rfloor, for any fixed0<ρ≤10<\\rho\\leq 1, attains theM−1M^\{\-1\}order with reservation approaching\(1\+ρ\)​M​K\(1\+\\rho\)MK\. Doubling is unnecessary\. Ift→∞t\\to\\inftybutt=o⁡\(M\)t=o\(M\), the attainable width is order\(Mt\)−1/2\(Mt\)^\{\-1/2\}, while the reservation ratio tends to one\. No task\-covering adaptive policy can then attainO⁡\(M−1\)O\(M^\{\-1\}\)worst\-pure width\. Task coverage is not intrinsic, however: a policy may sacrifice pure\-cohort point exactness to learn about the tasks it leaves unobserved\.

LetΠM,tall\\Pi^\{\\rm all\}\_\{M,t\}contain every pathwise\-\(M\+t\)​K\(M\+t\)K\-budget policy with no external cohort information; tasks need not be observed\. Define honest intervals as above and retain the same internally constant class𝒫\\mathcal\{P\}\.

###### Theorem 4\(Sharp interpolation with task omission\)\.

For fixedL≥3L\\geq 3and0<α≤1/120<\\alpha\\leq 1/12, constants0<cα,L′≤Cα,L′<∞0<c^\{\\prime\}\_\{\\alpha,L\}\\leq C^\{\\prime\}\_\{\\alpha,L\}<\\inftyexist such that, uniformly forM≥1M\\geq 1and0≤t≤M0\\leq t\\leq M,

cα,L′M⁡\(t\+M\)≤infπ∈ΠM,tall,I∈ℐα​\(π\)sup𝒞∈𝒫𝔼R​\|I\|≤Cα,L′M⁡\(t\+M\)\.\\frac\{c^\{\\prime\}\_\{\\alpha,L\}\}\{\\sqrt\{M\(t\+\\sqrt\{M\}\)\}\}\\leq\\inf\_\{\\pi\\in\\Pi^\{\\rm all\}\_\{M,t\},\\,I\\in\\mathcal\{I\}\_\{\\alpha\}\(\\pi\)\}\\sup\_\{\\mathcal\{C\}\\in\\mathcal\{P\}\}\\mathbb\{E\}\_\{R\}\|I\|\\leq\\frac\{C^\{\\prime\}\_\{\\alpha,L\}\}\{\\sqrt\{M\(t\+\\sqrt\{M\}\)\}\}\.\(5\)

For the upper bound, whenM≥16M\\geq 16andt2<Mt^\{2\}<M, selectn=M−⌊M⌋n=M\-\\lfloor\\sqrt\{M\}\\rfloortasks uniformly and use the saved task labels to auditq=t\+⌊M⌋q=t\+\\lfloor\\sqrt\{M\}\\rfloorof them; otherwise use the task\-covering design\. A finite\-population bound transfers the selected\-task certificate to the full grid\. Att=0t=0this givesM−3/4M^\{\-3/4\}width, but its pure\-cohort point MSE is no longer zero\. The lower proof covers adaptive task choice, stopping, and omission via an omitted\-bit branch and a posterior\-coupling branch \(Appendix[P](https://arxiv.org/html/2609.29140#A16)\)\. The displayed branch establishes the sharp order; Section[6](https://arxiv.org/html/2609.29140#S6)evaluates its finite\-budget constants\.

#### Construction and proof mechanism\.

Letddcount the disagreeing audited pairs\. Fort≥1t\\geq 1, defineU⁡\(d,δ,t\)U\(d,\\delta;t\)as the largestu∈\[d,t\]u\\in\[d,t\]withu−d\+d​log⁡\(d/u\)≤log⁡\(1/δ\)u\-d\+d\\log\(d/u\)\\leq\\log\(1/\\delta\), interpreting0​log⁡0=00\\log 0=0\. Set

VU=min⁡\{14​M,\(L−1\)​U​\(d,α/2,t\)2​L​M​t\},c0=L−1L​M,r=c0​x3\+2​VU​x\+\(c0​x/3\)2,V\_\{U\}=\\min\\left\\\{\\frac\{1\}\{4M\},\\frac\{\(L\-1\)U\(d,\\alpha/2;t\)\}\{2LMt\}\\right\\\},\\quad c\_\{0\}=\\frac\{L\-1\}\{LM\},\\quad r=\\frac\{c\_\{0\}x\}\{3\}\+\\sqrt\{2V\_\{U\}x\+\(c\_\{0\}x/3\)^\{2\}\},\(6\)wherex=log⁡\(4/α\)x=\\log\(4/\\alpha\)\. For the fixed random\-subset design, the intervalθ^±r\\widehat\{\\theta\}\\pm ris honest, after clipping to feasible labels; att=0t=0use a Hoeffding interval\. A random\-subset exponential\-moment comparison bounds average within\-task variation fromdd\. Conditional\-on\-subset Bernstein and a union bound then control mean error, without assuming independence of mean and disagreement\. On pure cohortsd=0d=0, giving widthO\(\(Mt\)−1/2\+M−1\)O\(\(Mt\)^\{\-1/2\}\+M^\{\-1\}\)\. For the converse, late outcomes force every revealed label to costKK\. On agreement transcripts, the sparse\-contamination prior’s likelihood ratio to a pure prior lies in\[5/6,1\]\[5/6,1\], even under adaptive auditing\. Independent but noncentered posterior task errors retain enough variance; a noncentral fourth\-moment argument and a single\-positive alternative complete the bound \(Appendix[K](https://arxiv.org/html/2609.29140#A11)\)\. The Audit formula applies to the fixed random\-subset design; outcome\-adaptive audit subsets require their own valid inference rule\.

#### Sharper fixed\-design constants\.

Att=Mt=M, all tasks have two draws\. Their exact variance relationVar⁡\(Ai\)=\(L−2\)​𝔼​Di/\(4​L\)\\operatorname\{Var\}\(A\_\{i\}\)=\(L\-2\)\\mathbb\{E\}D\_\{i\}/\(4L\)sharpens Equation \([6](https://arxiv.org/html/2609.29140#S4.E6)\) to Theorem[6](https://arxiv.org/html/2609.29140#Thmtheorem6)in Appendix[A\.3](https://arxiv.org/html/2609.29140#A1.SS3)\. A direct exponential envelope for the three\-point mean/disagreement law avoids the separate variance/mean error split \(Appendix[H](https://arxiv.org/html/2609.29140#A8)\)\. Its fixed data\-independent mixture is the*Pair*interval\. For0<t<M0<t<M, a joint moment bound averages over the random subset using Maclaurin’s inequality; a fixed mixture gives the*Joint*refinement \(Appendix[N](https://arxiv.org/html/2609.29140#A14)\)\. It retains Hoeffding att=0t=0and Pair att=Mt=M\. These finite\-constant refinements preserve the sampling design, while the Audit construction establishes the minimax upper rate\. Appendices[H](https://arxiv.org/html/2609.29140#A8)and[N](https://arxiv.org/html/2609.29140#A14)give their derivations and finite comparisons\.

#### Finite\-regime boundary\.

For a cohort withhih\_\{i\}positive paths in taskii, writeH=∑ihiH=\\sum\_\{i\}h\_\{i\}andq¯=M−1​∑i2​hi​\(L−hi\)/\{L⁡\(L−1\)\}\\bar\{q\}=M^\{\-1\}\\sum\_\{i\}2h\_\{i\}\(L\-h\_\{i\}\)/\\\{L\(L\-1\)\\\}\. Letxt​\(d\)x\_\{t\}\(d\)be the Joint boundary andWU​\(H\)W\_\{U\}\(H\)the exact expected width of the equal\-tailed hypergeometric interval afterM\+tM\+tpooled labels\.

###### Proposition 5\(Finite\-cohort dominance region\)\.

For0<t<M0<t<M,xtx\_\{t\}is nondecreasing and concave,𝔼R​\|IJoint\|≤2​xt​\(t​q¯\)/M\\mathbb\{E\}\_\{R\}\|I\_\{\\rm Joint\}\|\\leq 2x\_\{t\}\(t\\bar\{q\}\)/M, and the computable thresholdq⋆​\(H\)=sup\{q:2​xt​\(t​q\)/M≤WU​\(H\)\}q\_\{\\star\}\(H\)=\\sup\\\{q:2x\_\{t\}\(tq\)/M\\leq W\_\{U\}\(H\)\\\}is sufficient:q¯≤q⋆​\(H\)\\bar\{q\}\\leq q\_\{\\star\}\(H\)implies no larger expected width than fixed uniform\.

Appendix[O](https://arxiv.org/html/2609.29140#A15)gives the proof, exact sums and selector corollary\. The region describes sufficient cohort structure for a gain\. Evaluating the condition with full\-cohortH,q¯H,\\bar\{q\}explains the observed regime; a future prospective selector could instead use observable disagreement proxies\. Outside the sufficient region, the width ordering remains an empirical question\.

## 5Budget\-feasible designs and interval comparisons

#### Complete\-path baselines\.

Uniform fixed sampling completesmin⁡\(N,⌊B/K⌋\)\\min\(N,\\lfloor B/K\\rfloor\)paths chosen without replacement\. It is unbiased and has an exact hypergeometric interval\. Uniform adaptive sampling uses saved cost to draw more complete paths, stopping when fewer thanKKbudget units remain; we use a finite\-population confidence sequence\([Waudby\-Smith and Ramdas, 2020](https://arxiv.org/html/2609.29140#bib.bib15)\)\. Task\-balanced variants randomly permute paths within each task and tasks within each round\. The fixed\-count variant is unbiased; its interval combines bounded\-variable and finite\-complement concentration\. The cost\-stopped task\-balanced average is retained as a strong practical MSE baseline; statistical intervals are reported only for methods with a corresponding valid construction\.

#### Earlier continuation comparisons\.

Pooled and hierarchical progressive policies thin unresolved paths while inverse\-probability weighting preserves the fixed\-cohort mean; a pathwise reservation guarantees at least one completion\. A simpler full\-prefix policy spends at most a quarter ofBBon a common prefix and uses the remainder for fixed task\-balanced terminal sampling\. These methods test whether early\-event cost savings help\. Their schedules, weights, hard\-budget proofs, and interval restrictions appear in Appendices[B](https://arxiv.org/html/2609.29140#A2)and[C](https://arxiv.org/html/2609.29140#A3)\.

#### Use all purchased label information\.

IfPPdistinct paths have known label one andZZhave known label zero, the target certainly lies in\[P/N,1−Z/N\]\[P/N,1\-Z/N\]\. We intersect every interval with this feasible range\. The refinement costs no extra observations and preserves coverage; it is applied uniformly to all methods\. It also gives the cost\-stopped balanced heuristic a purely logical 100% certificate, distinguished from the statistical intervals\.

## 6Finite\-budget evidence

The empirical question is whether the constructive certificates improve width at usable budgets\. The LiveCodeBench replay compares the theory’s fixed designs at equal charged label counts\. For this fixed\-cohort target, replay reveals only each policy’s selected records and measures error against the fully observed grid mean\. It directly evaluates the sampling and inference design without generating new model outputs\. Synthetic cohorts isolate how finite size and within\-task disagreement shape the transition predicted by the theory, while earlier response and agent studies test transfer across different horizons and event structures\. Before inspecting LiveCodeBench correctness labels, we fixed the source revision, eligibility rule, budgets, designs, and random seeds; Appendix[Q](https://arxiv.org/html/2609.29140#A17)records the protocol and identifies the subsequent explanatory analyses\.

#### Measurements\.

We report conditional bias, MSE, interval width and coverage, charged cost, and hard\-budget violations\. Each method gets an independent stable RNG stream\. Replay\-bootstrap intervals quantify Monte Carlo uncertainty conditional on the fixed bank; generalization to new models or tasks is a separate question\. Zero\-error and census cases are reported separately, without epsilon\-stabilized ratios\. Complete method grids and run\-level results appear in the appendices\.

### 6\.1Joint converts replication into finite\-budget precision

#### Equal\-cost LiveCodeBench comparison\.

Sixteen released model panels contain all 880 tasks and at least five evaluated outputs per task; we use the first five binary labels, givingM=880,L=5,K=1M=880,L=5,K=1\. Every design purchases exactlyM\+tM\+tlabels, with 1000 evaluator randomizations per panel and budget\. Replay exposes only the records selected by a policy and scores them against the fully observed grid mean, directly testing allocation and confidence construction at equal observed\-label cost\.

#### More information from the same 1100 outputs\.

Joint beats exact fixed\-uniform mean width in 9/16 panels att=15t=15, 14/16 att=88t=88, and 15/16 att=220t=220; the corresponding median width ratios are 0\.966, 0\.807, and 0\.694\. Att=220t=220, both methods use 1100 labels, and the median MSE ratio is 0\.130\. Joint therefore reduces median mean interval width by 30\.6% and median MSE by 87\.0% without increasing the evaluation budget\. Its MSE is lower in every panel at every tested replication budget\. Att=440t=440and880880, median width ratios remain 0\.673 and 0\.677, with 14/16 width wins\.

\(a\) Sharp certification lawsEvery task observedfixed random\-subset auditΘα,L\(\[M\(t\+1\)\]−1/2\)\\displaystyle\\Theta\_\{\\alpha,L\}\\\!\\left\(\[M\(t\+1\)\]^\{\-1/2\}\\right\)t=0:M−1/2t=0:\\ M^\{\-1/2\}t=ρ​M:M−1t=\\rho M:\\ M^\{\-1\}Task omission allowedrandomized omissionΘα,L\(\[M\(t\+M\)\]−1/2\)\\displaystyle\\Theta\_\{\\alpha,L\}\\\!\\left\(\[M\(t\+\\sqrt\{M\}\)\]^\{\-1/2\}\\right\)t=0:M−3/4t=0:\\ M^\{\-3/4\}t=ρ​M:M−1t=\\rho M:\\ M^\{\-1\}\(b\) LiveCodeBench at equal label budgetsttlog scaleJoint / uniform mean width0\.51\.01\.5parity081529/30882204408809/1614/1615/16labels: panels with narrower intervalsFigure 1:The theory and its same\-budget finite realization\. Panel \(a\) gives rates up to constants depending onα,L\\alpha,L; both lower bounds include adaptive hard\-budget policies\. Panel \(b\) shows the median Joint/exact fixed\-uniform mean\-width ratio across 16 LiveCodeBench panels \(M=880,L=5M=880,L=5\), withM\+tM\+tlabels per method; horizontal positions uselog2⁡\(t\+1\)\\log\_\{2\}\(t\+1\)and ticks showtt\. Att=220t=220\(1100 labels\), Joint is narrower in 15/16 panels and reduces median mean width by 30\.6% and MSE by 87\.0%\.The finite result directly instantiates the rate theorem: the fixed random\-subset design supplies the observations, while Joint uses their mean and disagreement jointly to obtain a tighter certificate\. The two width exceptions at one or more large budgets still retain MSE gains, showing that finite width depends on within\-task disagreement as well as the asymptotic rate\. Appendix[Q](https://arxiv.org/html/2609.29140#A17)provides the full panel\- and budget\-level results\.

### 6\.2Cohort structure explains the finite gains

#### Information in disagreement\.

A same\-observation explanatory comparison holds the sampled records fixed and replaces Joint by a count\-only Hull interval\. Joint is narrower in all 16 panels at every testedt≥8t\\geq 8; median Joint/Hull width ratios are 0\.732 att=8t=8and 0\.425 att=220t=220\. The comparison isolates the value of coupling the sampled mean with disagreement evidence\. On the nine\-budget grid, all panels reach mean width 0\.04 with median requirements of 1100 labels for Joint and 1760 for uniform, a median saving of 660 labels\. Appendix[R](https://arxiv.org/html/2609.29140#A18)summarizes these descriptive grid comparisons across all targets and panels\.

#### A finite dominance region\.

Proposition[5](https://arxiv.org/html/2609.29140#Thmtheorem5)relates expected width to full\-cohort prevalence and pair disagreement\. Evaluated retrospectively on the LiveCodeBench bank, its sufficient condition holds in 73 of 112 interior panel–budget cells, and every certified cell is an observed width win\. Five additional wins lie outside the sufficient region\. This alignment connects the finite pattern to the theorem’s mechanism and motivates future prospective selectors based on observable disagreement proxies\.

#### Controlled departures from within\-task agreement\.

The synthetic study fixesθ=1/2,L=5\\theta=1/2,L=5, variesM=128,…,32768M=128,\\ldots,32768, audit fraction, and seven mixed\-task fractions up to 20%\. Numerical integration of exact finite laws yields Audit wins in 185 of 236 cells, while the dedicated Pair endpoint wins 57/59 simulated cells\. The transition from a 1\.307 width ratio atM=128M=128to 0\.661 atM=512M=512under approximately 10% auditing shows how the asymptotic advantage becomes useful at finite size\. Figure[4](https://arxiv.org/html/2609.29140#A13.F4)and the complete grid are in Appendix[M](https://arxiv.org/html/2609.29140#A13)\.

### 6\.3Boundary and transfer evidence

The same experiments locate where each theorem is operationally useful\. The order\-attaining omission branch has favorable asymptotic width and is wider than exact uniform in all 16 LiveCodeBench panels at its four active finite budgets; the tested constants therefore favor task\-covering Joint on this cohort\. The 19\-cell exact tiny\-design program similarly places Joint at 1\.116–1\.818 times the local numerical optimum, quantifying room for tighter finite constants\.

The earlier public response and agent banks test different horizons and event structures\. Across 868 conditions and 2,492,000 replay executions, task\-aware progressive sampling has median MSE ratios 1\.02 on responses and 1\.30 on agents against balanced adaptive completion, and width ratios 1\.66 and 1\.90 against uniform adaptive completion\. Later exploratory Pair and partial\-Joint checks identify additional gains in narrower operating regions\. Together, these comparisons locate where the sharp law and finite certificates are useful across evaluation structures\. Appendices[F](https://arxiv.org/html/2609.29140#A6)–[N](https://arxiv.org/html/2609.29140#A14)provide the full tables, charged costs, coverage, zero\-reference accounting, and the all\-500 missing\-outcome sensitivity analysis\.

## 7Conclusion

Replication closes a quantifiable gap between point accuracy and honest certification\. Requiring task coverage yields the sharp law\[M\(t\+1\)\]−1/2\[M\(t\+1\)\]^\{\-1/2\}; permitting omission changes it to\[M\(t\+M\)\]−1/2\[M\(t\+\\sqrt\{M\}\)\]^\{\-1/2\}\. Disagreement\-based fixed designs attain these orders, while Joint improves finite constants\. At the same 1100\-output budget, Joint beats exact pooled uniform mean width in 15/16 LiveCodeBench panels, with median width and MSE reductions of 30\.6% and 87\.0%\. The theoretical laws and this replay demonstrate how replication can improve certification within a fixed evaluation budget\.

The formal scope is worst\-pure\-cohort expected width under honesty over all fixed cohorts, for fixedL≥3L\\geq 3and0<α≤1/120<\\alpha\\leq 1/12\. Finite constants, near\-pure rates, and population reliability remain complementary targets\. The strongest public replay usesK=1K=1, so it isolates task allocation and inference; the tested finite omission rule and the earlier response/agent banks identify regimes where task\-covering Joint is preferable\. Prospective stopping and realized token or financial cost require their own validation\.

Within this scope, the paper turns replication from an informal recommendation into a quantitative design choice\. The two sharp laws specify what confidence width is achievable for a given hard budget, and the constructive interval shows how that information can translate into more precise model evaluation without purchasing additional outputs\.

### AI use statement

The core theoretical ideas, mathematical claims, derivations, proofs, and experimental design were developed by the author\. Generative AI tools were used to support research exploration and literature review, assist with data organization, plotting, utility scripts, and general code editing, and polish the language of the manuscript\. The author has completed a manual review of the manuscript, proofs, code, results, and all AI\-assisted material, and takes full responsibility for the final content\.

### Ethics statement

This work uses public research outcome records and does not recruit human participants or deploy an agent into a live external environment\. The replay code consumes correctness and task identifiers, not private communications\. The response\-bank and agent\-artifact licenses and primary sources are retained in the reproducibility record\. Efficiency estimates must not be interpreted as safety guarantees or as evidence that unobserved failures did not occur\.

### Reproducibility statement

The supplement records the prospective protocol, immutable source revisions, download and output hashes, code archive, fixed seed derivation, raw grading audits, and all replay outputs\. Appendix proofs state the assumptions required by each confidence calculation\. The code enforces prefix\-only access and charges actual observed response/episode counts\. No paid model calls or new model training are required to reproduce the replays\.

## References

- A\. N\. Angelopoulos, J\. Eisenstein, J\. Berant, A\. Agarwal, and A\. FischCost\-optimal active AI model evaluation\.External Links:2506\.07949,[Link](https://arxiv.org/abs/2506.07949)Cited by:[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px2.p1.1)\.
- Aubry \(2024\)P\. AubryOn the implementation of stratified two\-stage simple random sampling without replacement, with possible collapsed strata\.MethodsX13,pp\. 102928\.External Links:[Document](https://dx.doi.org/10.1016/j.mex.2024.102928),[Link](https://doi.org/10.1016/j.mex.2024.102928)Cited by:[§A\.2](https://arxiv.org/html/2609.29140#A1.SS2.p3.2)\.
- Barabesiet al\.\(2012\)L\. Barabesi, S\. Franceschi, and M\. MarcheselliProperties of design\-based estimation under stratified spatial sampling with application to canopy coverage estimation\.The Annals of Applied Statistics6\(1\),pp\. 210–228\.External Links:[Document](https://dx.doi.org/10.1214/11-AOAS509),[Link](https://arxiv.org/abs/1203.4065)Cited by:[§K\.5](https://arxiv.org/html/2609.29140#A11.SS5.p1.1),[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px2.p1.1)\.
- Bardenet and Maillard \(2015\)R\. Bardenet and O\. MaillardConcentration inequalities for sampling without replacement\.Bernoulli21\(3\),pp\. 1361–1385\.External Links:[Document](https://dx.doi.org/10.3150/14-BEJ605),[Link](https://arxiv.org/abs/1309.4029)Cited by:[§P\.1](https://arxiv.org/html/2609.29140#A16.SS1.p2.3),[§H\.1](https://arxiv.org/html/2609.29140#A8.SS1.p1.1)\.
- Bjarnasonet al\.\(2026\)B\. H\. Bjarnason, A\. Silva, and M\. MonperrusOn randomness in agentic evals\.InICLR Workshop on Agents in the Wild,External Links:[Link](https://arxiv.org/abs/2602.07150)Cited by:[Appendix F](https://arxiv.org/html/2609.29140#A6.SS0.SSS0.Px1.p3.1),[§F\.4](https://arxiv.org/html/2609.29140#A6.SS4.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2609.29140#S1.p1.1),[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px1.p1.1)\.
- Brownet al\.\(2024\)B\. Brown, J\. Juravsky, R\. Ehrlich, R\. Clark, Q\. V\. Le, C\. Ré, and A\. MirhoseiniLarge language monkeys: scaling inference compute with repeated sampling\.External Links:2407\.21787,[Link](https://arxiv.org/abs/2407.21787)Cited by:[Appendix F](https://arxiv.org/html/2609.29140#A6.SS0.SSS0.Px1.p2.1),[§1](https://arxiv.org/html/2609.29140#S1.p1.1),[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px1.p1.1)\.
- Burgess and Chapman \(2021\)M\. A\. Burgess and A\. C\. ChapmanApproximating the Shapley value using stratified empirical Bernstein sampling\.InProceedings of the Thirtieth International Joint Conference on Artificial Intelligence,pp\. 73–81\.External Links:[Link](https://www.ijcai.org/proceedings/2021/0011.pdf)Cited by:[§K\.5](https://arxiv.org/html/2609.29140#A11.SS5.p1.1),[§H\.1](https://arxiv.org/html/2609.29140#A8.SS1.p3.1),[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2609.29140#S2.T1.2.2.1.1.1)\.
- Cai and Low \(2004\)T\. T\. Cai and M\. G\. LowAn adaptation theory for nonparametric confidence intervals\.The Annals of Statistics32\(5\),pp\. 1805–1840\.External Links:[Document](https://dx.doi.org/10.1214/009053604000000049),[Link](https://arxiv.org/abs/math/0503662)Cited by:[§K\.5](https://arxiv.org/html/2609.29140#A11.SS5.p1.1),[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2609.29140#S2.T1.2.3.1.1.1)\.
- Feldman and Romano \(2026\)S\. Feldman and Y\. RomanoHow many iterations to jailbreak? dynamic budget allocation for multi\-turn LLM evaluation\.Note:Version 5, July 2026External Links:2605\.06605,[Link](https://arxiv.org/abs/2605.06605v5)Cited by:[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px3.p1.1),[Table 1](https://arxiv.org/html/2609.29140#S2.T1.2.5.1.1.1)\.
- Fischet al\.\(2024\)A\. Fisch, J\. Maynez, R\. A\. Hofer, B\. Dhingra, A\. Globerson, and W\. W\. CohenStratified prediction\-powered inference for hybrid language model evaluation\.External Links:2406\.04291,[Link](https://arxiv.org/abs/2406.04291)Cited by:[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px2.p1.1)\.
- Hoeffding \(1963\)W\. HoeffdingProbability inequalities for sums of bounded random variables\.Journal of the American Statistical Association58\(301\),pp\. 13–30\.External Links:[Document](https://dx.doi.org/10.1080/01621459.1963.10500830),[Link](https://doi.org/10.1080/01621459.1963.10500830)Cited by:[§P\.1](https://arxiv.org/html/2609.29140#A16.SS1.p2.3),[§B\.6](https://arxiv.org/html/2609.29140#A2.SS6.p4.1)\.
- Howardet al\.\(2021\)S\. R\. Howard, A\. Ramdas, J\. McAuliffe, and J\. SekhonTime\-uniform, nonparametric, nonasymptotic confidence sequences\.The Annals of Statistics49\(2\),pp\. 1055–1080\.External Links:[Document](https://dx.doi.org/10.1214/20-AOS1991),[Link](https://arxiv.org/abs/1810.08240)Cited by:[§B\.6](https://arxiv.org/html/2609.29140#A2.SS6.p5.1),[§H\.4](https://arxiv.org/html/2609.29140#A8.SS4.p1.4),[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px3.p1.1)\.
- Huang \(2026\)W\. HuangHow many tasks are enough for agent benchmark decisions? a replay analysis of public LLM agent benchmarks\.Note:KDD Workshop on Evaluation and Trustworthiness of Agentic AIExternal Links:2607\.12338,[Link](https://arxiv.org/abs/2607.12338v1)Cited by:[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px1.p1.1)\.
- Jainet al\.\(2025\)N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. StoicaLiveCodeBench: holistic and contamination free evaluation of large language models for code\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=chfJJYC3iL)Cited by:[Appendix Q](https://arxiv.org/html/2609.29140#A17.SS0.SSS0.Px1.p1.1)\.
- Kazdanet al\.\(2025\)J\. Kazdan, R\. Schaeffer, Y\. Allouah, C\. Sullivan, K\. Yu, N\. Levi, and S\. KoyejoEfficient prediction of pass@k scaling in large language models\.External Links:2510\.05197,[Link](https://arxiv.org/abs/2510.05197)Cited by:[§1](https://arxiv.org/html/2609.29140#S1.p1.1),[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px1.p1.1)\.
- Lumley \(2024\)T\. LumleyMultiphase sampling\.Note:Survey package methods vignetteExternal Links:[Link](https://stat.ethz.ch/CRAN/web/packages/survey/vignettes/multiphase.html)Cited by:[§B\.3](https://arxiv.org/html/2609.29140#A2.SS3.p1.1),[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px3.p1.1)\.
- Maurer and Pontil \(2009\)A\. Maurer and M\. PontilEmpirical Bernstein bounds and sample variance penalization\.InProceedings of the 22nd Annual Conference on Learning Theory,External Links:[Link](https://arxiv.org/abs/0907.3740)Cited by:[§K\.5](https://arxiv.org/html/2609.29140#A11.SS5.p1.1),[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px2.p1.1)\.
- Pilditch \(2026\)T\. D\. PilditchKnowing when to stop: bayesian optimal stopping for LLM evaluations\.External Links:2608\.14425,[Link](https://arxiv.org/abs/2608.14425)Cited by:[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px1.p1.1)\.
- Prangle \(2014\)D\. PrangleLazy ABC\.External Links:1405\.7867,[Link](https://arxiv.org/abs/1405.7867)Cited by:[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px3.p1.1)\.
- Spertuset al\.\(2024\)J\. V\. Spertus, M\. Sridhar, and P\. B\. StarkSequential stratified inference for the mean\.Note:arXiv:2409\.06680; version 5, revised August 2026External Links:[Link](https://arxiv.org/abs/2409.06680v5)Cited by:[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2609.29140#S2.T1.2.4.1.1.1)\.
- Wassermanet al\.\(2020\)L\. Wasserman, A\. Ramdas, and S\. BalakrishnanUniversal inference\.Proceedings of the National Academy of Sciences\.External Links:[Document](https://dx.doi.org/10.1073/pnas.1922664117),[Link](https://arxiv.org/abs/1912.11436)Cited by:[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px3.p1.1)\.
- Waudby\-Smith and Ramdas \(2020\)I\. Waudby\-Smith and A\. RamdasConfidence sequences for sampling without replacement\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2006.04347)Cited by:[§B\.1](https://arxiv.org/html/2609.29140#A2.SS1.p2.2),[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px3.p1.1),[Table 1](https://arxiv.org/html/2609.29140#S2.T1.2.4.1.1.1),[§5](https://arxiv.org/html/2609.29140#S5.SS0.SSS0.Px1.p1.1)\.
- Wuet al\.\(2026\)S\. Wu, Y\. Nair, and E\. J\. CandèsEfficient evaluation of LLM performance with statistical guarantees\.External Links:2601\.20251,[Link](https://arxiv.org/abs/2601.20251)Cited by:[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px2.p1.1)\.
- Yauneyet al\.\(2026\)G\. Yauney, S\. S\. Warraich, and S\. SwayamdiptaHow reliable is language model micro\-benchmarking?\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/hash/2e2960f2fe9e981f33f51c78656e3ca2-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px2.p1.1)\.
- Zrnic and Candès \(2024\)T\. Zrnic and E\. J\. CandèsActive statistical inference\.External Links:2403\.03208,[Link](https://arxiv.org/abs/2403.03208)Cited by:[§2](https://arxiv.org/html/2609.29140#S2.SS0.SSS0.Px2.p1.1)\.

## Supplementary Material \(Appendix\)

The remainder of this document contains the supplementary proofs, experimental details, complete results, and reproducibility information for the paper\.

## Appendix AInformation\-limit proofs

### A\.1Proof of Theorem[1](https://arxiv.org/html/2609.29140#Thmtheorem1)

Restrict the cohort toTj∈\{K,K\+1\}T\_\{j\}\\in\\\{K,K\+1\\\}for every path\. Every prefix shorter thanKKhas the same observation and cost under all these cohorts\. A complete path costs exactlyKK, whether its terminal label is zero or one\. Consequently no budget\-feasible policy obtains more thanm=min⁡\(N,⌊B/K⌋\)m=\\min\(N,\\lfloor B/K\\rfloor\)terminal labels\. If it obtains fewer, supplying additional labels up tommcan only decrease the minimum Bayes risk, so a lower bound formmlabels is also a lower bound for that policy\.

ChooseHHuniformly from\{0,…,N\}\\\{0,\\ldots,N\\\}and, conditional onHH, choose the positive\-label set uniformly among subsets of sizeHH\. No task identity or unobserved path identity breaks this symmetry\. Adaptive selection of an unqueried path therefore gives the same label experiment as sampling without replacement\. Equivalently, first choosep∼Uniform⁡\[0,1\]p\\sim\\operatorname\{Uniform\}\[0,1\]and then choose theNNlabels independently conditional onpp\. This induces the same uniform prior onHH\.

Aftermmlabels withsspositives, the number of positives in the remainingN−mN\-mpaths has a beta\-binomial distribution with parameters\(N−m,s\+1,m−s\+1\)\(N\-m,s\+1,m\-s\+1\)\. Its variance is

Var⁡\(H∣s\)=\(N−m\)​\(s\+1\)​\(m−s\+1\)​\(N\+2\)\(m\+2\)2​\(m\+3\)\.\\operatorname\{Var\}\(H\\mid s\)=\\frac\{\(N\-m\)\(s\+1\)\(m\-s\+1\)\(N\+2\)\}\{\(m\+2\)^\{2\}\(m\+3\)\}\.\(7\)The marginal distribution ofssis uniform on\{0,…,m\}\\\{0,\\ldots,m\\\}, and

1m\+1​∑s=0m\(s\+1\)​\(m−s\+1\)=\(m\+2\)​\(m\+3\)6\.\\frac\{1\}\{m\+1\}\\sum\_\{s=0\}^\{m\}\(s\+1\)\(m\-s\+1\)=\\frac\{\(m\+2\)\(m\+3\)\}\{6\}\.\(8\)Under squared loss the posterior mean minimizes Bayes risk\. Dividing the expected posterior variance byN2N^\{2\}gives Equation \([2](https://arxiv.org/html/2609.29140#S4.E2)\); minimax risk is at least this Bayes risk\. This is an elementary finite\-population lower\-bound construction, not a new general minimax technique\.

IfB<KB<K, compare the all\-zero and all\-one terminal\-label cohorts, again placing decisive events only atKK\. Every feasible transcript is identical\. For any estimatorAAwith that common observation law,max⁡\{𝔼​A2,𝔼​\(A−1\)2\}≥1/4\\max\\\{\\mathbb\{E\}A^\{2\},\\mathbb\{E\}\(A\-1\)^\{2\}\\\}\\geq 1/4\.

### A\.2Proof of Theorem[2](https://arxiv.org/html/2609.29140#Thmtheorem2)

Again setTi​r=KT\_\{ir\}=Kfor label zero andTi​r=K\+1T\_\{ir\}=K\+1for label one\. All observed nonterminal prefixes are now identical and every completed path costsKK\. Consider two priors on fixed cohorts:

- •Prior A independently chooses Bernoulli\(qq\) bitsU1,…,UMU\_\{1\},\\ldots,U\_\{M\}and makes every path in taskiihave labelUiU\_\{i\}\.
- •Prior B independently chooses a uniform subset ofh=⌊L/2⌋h=\\lfloor L/2\\rfloorpositive paths in each task, with the otherL−hL\-hpaths negative\.

Under A the target isU¯=M−1​∑iUi\\bar\{U\}=M^\{\-1\}\\sum\_\{i\}U\_\{i\}, and one\-per\-task averaging recovers it without error\. Under B every realized cohort has targetq=h/Lq=h/L\. One uniformly selected label per task has the same joint distribution of labels and selected identities under both priors\. The same remains true if task order depends on previously observed tasks, because unobserved tasks are independent under each prior\. Internal interval randomization may be included in this common observation law\.

Uniform design coverage over fixed cohorts implies coverage after integrating over either prior\. Hence under the common observation law

ℙ⁡\(U¯∈I\)≥1−α,ℙ⁡\(q∈I\)≥1−α\.\\mathbb\{P\}\(\\bar\{U\}\\in I\)\\geq 1\-\\alpha,\\qquad\\mathbb\{P\}\(q\\in I\)\\geq 1\-\\alpha\.\(9\)Both points belong toIIwith probability at least1−2​α1\-2\\alpha\. On this event its length is at least\|U¯−q\|\|\\bar\{U\}\-q\|\. Conditioning the event probability onZ=∑iUiZ=\\sum\_\{i\}U\_\{i\}gives an admissible functionwwin Equation \([3](https://arxiv.org/html/2609.29140#S4.E3)\)\. Thus the average expected interval length under prior A is at leastbM,qb\_\{M,q\}; some fixed pure\-task cohort attains at least that average\.

The infimum definingbM,qb\_\{M,q\}keeps the smallest1−2​α1\-2\\alphaprobability mass of\|Z/M−q\|\|Z/M\-q\|, allowing fractional mass at a discrete boundary\. The central limit theorem givesM/\[q⁡\(1−q\)\]​\(U¯−q\)⇒Z0∼N⁡\(0,1\)\\sqrt\{M/\[q\(1\-q\)\]\}\(\\bar\{U\}\-q\)\\Rightarrow Z\_\{0\}\\sim N\(0,1\)\. Its uniformly bounded second moment supplies uniform integrability\. The corresponding trimmed expectation therefore converges to

q⁡\(1−q\)𝔼\[\|Z0\|𝟏\{\|Z0\|≤z1−α\}\]\\displaystyle\\sqrt\{q\(1\-q\)\}\\mathbb\{E\}\\bigl\[\|Z\_\{0\}\|\\mathbf\{1\}\\\{\|Z\_\{0\}\|\\leq z\_\{1\-\\alpha\}\\\}\\bigr\]=2​q⁡\(1−q\)​∫0z1−αz​e−z2/22​π​𝑑z\\displaystyle=2\\sqrt\{q\(1\-q\)\}\\int\_\{0\}^\{z\_\{1\-\\alpha\}\}z\\frac\{e^\{\-z^\{2\}/2\}\}\{\\sqrt\{2\\pi\}\}\\,dz\(10\)=2q⁡\(1−q\)\(1−e−z1−α2/2\)2​π\.\\displaystyle=\\frac\{2\\sqrt\{q\(1\-q\)\}\(1\-e^\{\-z\_\{1\-\\alpha\}^\{2\}/2\}\)\}\{\\sqrt\{2\\pi\}\}\.\(11\)AtM=130M=130,q=1/2q=1/2, andα=\.05\\alpha=\.05, exact binomial summation givesbM,q=0\.0258917180b\_\{M,q\}=0\.0258917180\. The result concerns the existence of a difficult pure\-task cohort for any honest procedure, not a lower bound at every cohort\. Additional terminal labels or informative side information can invalidate the indistinguishability construction\. Classical one\-per\-stratum variance limitations and collapsed\-stratum methods are discussed by[Aubry \(2024\)](https://arxiv.org/html/2609.29140#bib.bib19)\.

### A\.3Sharper full\-replication specialization

###### Theorem 6\(Two\-replicate certification\)\.

Independently sample two paths without replacement in each task, withL≥3L\\geq 3\. LetAiA\_\{i\}be their mean,DiD\_\{i\}their disagreement indicator,d=∑iDid=\\sum\_\{i\}D\_\{i\}, andθ^=M−1​∑iAi\\widehat\{\\theta\}=M^\{\-1\}\\sum\_\{i\}A\_\{i\}\. DefineU⁡\(d,δ\)U\(d,\\delta\)as the largestu∈\[d,M\]u\\in\[d,M\]satisfyingu−d\+d​log⁡\(d/u\)≤log⁡\(1/δ\)u\-d\+d\\log\(d/u\)\\leq\\log\(1/\\delta\), with0​log⁡0=00\\log 0=0\. Set

VU=L−24​L​M2​U​\(d,α/2\),c=L−2L​M,x=log⁡\(4/α\),r=c​x3\+2​VU​x\+\(c​x/3\)2\.V\_\{U\}=\\frac\{L\-2\}\{4LM^\{2\}\}U\(d,\\alpha/2\),\\quad c=\\frac\{L\-2\}\{LM\},\\quad x=\\log\(4/\\alpha\),\\quad r=\\frac\{cx\}\{3\}\+\\sqrt\{2V\_\{U\}x\+\(cx/3\)^\{2\}\}\.\(12\)Then\[θ^−r,θ^\+r\]∩\[0,1\]\[\\widehat\{\\theta\}\-r,\\widehat\{\\theta\}\+r\]\\cap\[0,1\]has coverage at least1−α1\-\\alphafor every fixed cohort\. On every internally constant task cohort, the estimate is exact and the width isOα,L​\(M−1\)O\_\{\\alpha,L\}\(M^\{\-1\}\)\. Conversely, for0<α<1/40<\\alpha<1/4, every uniformly honest interval based on this two\-path design has, at the all\-zero cohort,

𝔼0​\|I\|≥1−α−α/\(1−2/L\)M​L\.\\mathbb\{E\}\_\{0\}\|I\|\\geq\\frac\{1\-\\alpha\-\\alpha/\(1\-2/L\)\}\{ML\}\.\(13\)Thus the optimal worst\-pure\-cohort expected\-width order isM−1M^\{\-1\}for fixedL≥3L\\geq 3\. ForL=2L=2, two paths are census\.

#### Proof\.

Writepi=L−1​∑rYi​rp\_\{i\}=L^\{\-1\}\\sum\_\{r\}Y\_\{ir\}\. Under independent simple random sampling of two labels per task,

𝔼​Di=2​LL−1​pi​\(1−pi\),Var⁡\(Ai\)=L−22​\(L−1\)​pi​\(1−pi\)=L−24​L​𝔼​Di\.\\mathbb\{E\}D\_\{i\}=\\frac\{2L\}\{L\-1\}p\_\{i\}\(1\-p\_\{i\}\),\\qquad\\operatorname\{Var\}\(A\_\{i\}\)=\\frac\{L\-2\}\{2\(L\-1\)\}p\_\{i\}\(1\-p\_\{i\}\)=\\frac\{L\-2\}\{4L\}\\mathbb\{E\}D\_\{i\}\.\(14\)HenceV=Var⁡\(θ^\)=\(L−2\)​λ/\(4​L​M2\)V=\\operatorname\{Var\}\(\\widehat\{\\theta\}\)=\(L\-2\)\\lambda/\(4LM^\{2\}\), whereλ=∑i𝔼​Di\\lambda=\\sum\_\{i\}\\mathbb\{E\}D\_\{i\}\. TheDiD\_\{i\}are independent Bernoulli variables but need not be identically distributed\. Fort≥0t\\geq 0,

𝔼e−t∑iDi=∏i\[1\+\(e−t−1\)𝔼Di\]≤exp\{\(e−t−1\)λ\}\.\\mathbb\{E\}e^\{\-t\\sum\_\{i\}D\_\{i\}\}=\\prod\_\{i\}\[1\+\(e^\{\-t\}\-1\)\\mathbb\{E\}D\_\{i\}\]\\leq\\exp\\\{\(e^\{\-t\}\-1\)\\lambda\\\}\.\(15\)Optimizing the Chernoff inequality at a thresholda≤λa\\leq\\lambdagives

ℙ\{d≤a\}≤exp\{−λ\+a−alog\(a/λ\)\}\.\\mathbb\{P\}\\\{d\\leq a\\\}\\leq\\exp\\\{\-\\lambda\+a\-a\\log\(a/\\lambda\)\\\}\.\(16\)The exponent is monotone inaabelowλ\\lambda\. Inverting this lower\-tail bound yieldsℙ\{λ\>U\(d,δ\)\}≤δ\\mathbb\{P\}\\\{\\lambda\>U\(d,\\delta\)\\\}\\leq\\delta, including the discrete thresholds and the cased=0d=0\. In particular,U⁡\(0,δ\)=min⁡\{M,log⁡\(1/δ\)\}U\(0,\\delta\)=\\min\\\{M,\\log\(1/\\delta\)\\\}\.

The random variables\(Ai−pi\)/M\(A\_\{i\}\-p\_\{i\}\)/Mare independent and centered\. Their absolute values are at mostc=\(L−2\)/\(L​M\)c=\(L\-2\)/\(LM\): if the selected two\-label mean isAiA\_\{i\}and the mean of the remaining labels isBiB\_\{i\}, thenAi−pi=\(L−2\)​\(Ai−Bi\)/LA\_\{i\}\-p\_\{i\}=\(L\-2\)\(A\_\{i\}\-B\_\{i\}\)/L\. The two\-sided Bernstein inequality therefore gives

ℙ\{\|θ^−θ𝒞\|\>cx/3\+2​V​x\+\(c​x/3\)2\}≤2e−x\.\\mathbb\{P\}\\left\\\{\|\\widehat\{\\theta\}\-\\theta\_\{\\mathcal\{C\}\}\|\>cx/3\+\\sqrt\{2Vx\+\(cx/3\)^\{2\}\}\\right\\\}\\leq 2e^\{\-x\}\.\(17\)Useδ=α/2\\delta=\\alpha/2,x=log⁡\(4/α\)x=\\log\(4/\\alpha\)and a union bound\. On the joint event,V≤VUV\\leq V\_\{U\}and the radius is increasing inVV\. No independence between the sample mean and the estimated variance is needed\. Clipping to\[0,1\]\[0,1\], or to any deterministic feasible set containing the target, cannot reduce coverage\. On a pure\-task cohortd=0d=0andθ^\\widehat\{\\theta\}is exact, giving the deterministic upper bound

\|I\|≤2M​\[\(L−2\)​x3​L\+\(L−2\)​x​log⁡\(2/α\)2​L\+\(\(L−2\)​x3​L\)2\]\.\|I\|\\leq\\frac\{2\}\{M\}\\left\[\\frac\{\(L\-2\)x\}\{3L\}\+\\sqrt\{\\frac\{\(L\-2\)x\\log\(2/\\alpha\)\}\{2L\}\+\\left\(\\frac\{\(L\-2\)x\}\{3L\}\\right\)^\{2\}\}\\right\]\.\(18\)For one path, a bounded\-variable Hoeffding interval supplies a uniformO\(M−1/2\)O\(M^\{\-1/2\}\)upper bound, matching Theorem[2](https://arxiv.org/html/2609.29140#Thmtheorem2)’s order\.

For the two\-path lower bound, compare the all\-zero cohort to a prior that places exactly one positive path at a uniformly random location in a single fixed task\. Under that alternative prior, the probability of missing the positive in two uniform observations isq0=1−2/Lq\_\{0\}=1\-2/L\. Conditional on missing it, the entire observed identity/label transcript has the all\-zero law: each possible sampled pair misses a uniform positive location with the same probabilityq0q\_\{0\}\. Place decisive events atKKso all shorter prefixes and costs are also uninformative\. The alternative target isΔ=1/\(M​L\)\\Delta=1/\(ML\)\. Uniform honesty over each alternative cohort implies honesty under the mixture, and therefore

1−α≤q0​ℙ0​\(Δ∈I\)\+\(1−q0\),ℙ0​\(0∈I\)≥1−α\.1\-\\alpha\\leq q\_\{0\}\\mathbb\{P\}\_\{0\}\(\\Delta\\in I\)\+\(1\-q\_\{0\}\),\\qquad\\mathbb\{P\}\_\{0\}\(0\\in I\)\\geq 1\-\\alpha\.\(19\)Both endpoints belong toIIwith probability at least1−α−α/q01\-\\alpha\-\\alpha/q\_\{0\}, proving Equation \([13](https://arxiv.org/html/2609.29140#A1.E13)\)\. Its coefficient is positive for everyL≥3L\\geq 3ifα<1/4\\alpha<1/4\.

### A\.4Fixed\-prefix extension of the disagreement certificate

Condition on a fully observed prefix\. Suppose allGGremaining active tasks are sampled, with active\-path countsnin\_\{i\}, predetermined terminal counts1≤mi≤ni1\\leq m\_\{i\}\\leq n\_\{i\}, and original population sizeNN\. The active\-task weights arewi=ni/Nw\_\{i\}=n\_\{i\}/N; previously resolved paths have label zero\. The target is∑iwi​pi\\sum\_\{i\}w\_\{i\}p\_\{i\}and the unbiased estimate is∑iwi​Y¯i\\sum\_\{i\}w\_\{i\}\\bar\{Y\}\_\{i\}\. The counts may depend on observed prefix risk sizes, but not on the terminal sample outcomes\. DefineDiD\_\{i\}using the first two uniform terminal draws whenmi≥2m\_\{i\}\\geq 2\. For noncensus replicated tasks,

Var⁡\(wi​Y¯i\)=wi2​\(ni−mi\)mi​\(ni−1\)​pi​\(1−pi\)=ai​𝔼​Di,ai=wi2​\(ni−mi\)2​ni​mi\.\\operatorname\{Var\}\(w\_\{i\}\\bar\{Y\}\_\{i\}\)=\\frac\{w\_\{i\}^\{2\}\(n\_\{i\}\-m\_\{i\}\)\}\{m\_\{i\}\(n\_\{i\}\-1\)\}p\_\{i\}\(1\-p\_\{i\}\)=a\_\{i\}\\mathbb\{E\}D\_\{i\},\\qquad a\_\{i\}=\\frac\{w\_\{i\}^\{2\}\(n\_\{i\}\-m\_\{i\}\)\}\{2n\_\{i\}m\_\{i\}\}\.\(20\)Leta∗=maxi⁡aia\_\{\*\}=\\max\_\{i\}a\_\{i\}over those tasks andS=∑iai​Di/a∗S=\\sum\_\{i\}a\_\{i\}D\_\{i\}/a\_\{\*\}\. Forbi=ai/a∗∈\[0,1\]b\_\{i\}=a\_\{i\}/a\_\{\*\}\\in\[0,1\], convexity givese−t​bi−1≤bi​\(e−t−1\)e^\{\-tb\_\{i\}\}\-1\\leq b\_\{i\}\(e^\{\-t\}\-1\), so the same negative\-MGF argument boundsΛ=∑iai​𝔼​Di/a∗\\Lambda=\\sum\_\{i\}a\_\{i\}\\mathbb\{E\}D\_\{i\}/a\_\{\*\}by the Poisson–Chernoff inversion atSS\. Replace the capMMinUUby∑iai/a∗\\sum\_\{i\}a\_\{i\}/a\_\{\*\}; this inversion also permits nonintegerSS\. Multiply its upper bound bya∗a\_\{\*\}and, if smaller, use the deterministic variance bound∑iwi2​\(ni−mi\)/\[4​mi​\(ni−1\)\]\\sum\_\{i\}w\_\{i\}^\{2\}\(n\_\{i\}\-m\_\{i\}\)/\[4m\_\{i\}\(n\_\{i\}\-1\)\]\. For tasks withmi=1<nim\_\{i\}=1<n\_\{i\}, add their deterministic boundwi2/4w\_\{i\}^\{2\}/4; census tasks contribute zero\. If no replicated noncensus task remains, use only these deterministic bounds\.

The centered mean increments satisfy\|wi​\(Y¯i−pi\)\|≤\(ni−mi\)/N\|w\_\{i\}\(\\bar\{Y\}\_\{i\}\-p\_\{i\}\)\|\\leq\(n\_\{i\}\-m\_\{i\}\)/N\. Substituting their maximum forccin Equation \([12](https://arxiv.org/html/2609.29140#A1.E12)\) establishes conditional coverage, and averaging over the prefix establishes unconditional design coverage\. The deterministic feasible range is\[∑isi/N,\(∑ini−∑i\(mi−si\)\)/N\]\[\\sum\_\{i\}s\_\{i\}/N,\\,\(\\sum\_\{i\}n\_\{i\}\-\\sum\_\{i\}\(m\_\{i\}\-s\_\{i\}\)\)/N\], wheresis\_\{i\}is the observed positive count\. The implementation intersects with this range\. If only a subset of active tasks is sampled, or counts are chosen by recycling terminal outcome\-dependent cost savings, this proof does not apply\. We do not attach this interval to the cost\-stopped task\-balanced heuristic\.

## Appendix BComplete design and inference specification

### B\.1Uniform completion and finite\-population confidence sequences

For a binary population of sizeNNwithHHpositives,sspositives among a fixed uniform sample of sizemmhave lawHypergeom⁡\(N,H,m\)\\operatorname\{Hypergeom\}\(N,H,m\)\. We invert both tails at levelα/2\\alpha/2, respecting the integer feasible ranges≤H≤N−m\+ss\\leq H\\leq N\-m\+s\. At census the interval is the observed point\. The fixed\-count design completesm=min⁡\(N,⌊B/K⌋\)m=\\min\(N,\\lfloor B/K\\rfloor\)paths; earlier decisive events cannot prevent completion of this predetermined count\.

For the cost\-adaptive design letsms\_\{m\}denote positives among the firstmmpaths of an independently uniform permutation\. Its ordered\-label likelihood under candidateHHis

PH\(Y1:m\)=\(H\)sm​\(N−H\)m−sm\(N\)m,\(a\)b=a\(a−1\)⋯\(a−b\+1\)\.P\_\{H\}\(Y\_\{1:m\}\)=\\frac\{\(H\)\_\{s\_\{m\}\}\(N\-H\)\_\{m\-s\_\{m\}\}\}\{\(N\)\_\{m\}\},\\qquad\(a\)\_\{b\}=a\(a\-1\)\\cdots\(a\-b\+1\)\.\(21\)The uniform prior onH∈\{0,…,N\}H\\in\\\{0,\\ldots,N\\\}gives ordered predictive massQ\(Y1:m\)=Beta\(sm\+1,m−sm\+1\)Q\(Y\_\{1:m\}\)=\\operatorname\{Beta\}\(s\_\{m\}\+1,m\-s\_\{m\}\+1\)\. For the trueHH, the ratioQ/PHQ/P\_\{H\}is a nonnegative test supermartingale, as in the prior\-posterior\-ratio construction of[Waudby\-Smith and Ramdas \(2020\)](https://arxiv.org/html/2609.29140#bib.bib15)\. RetainHHwhenQ/PH<1/αQ/P\_\{H\}<1/\\alpha\. At each draw, the next path is uniform among unqueried paths, so its conditional positive probability remains\(H−sm\)/\(N−m\)\(H\-s\_\{m\}\)/\(N\-m\)even when previous path costs are in the history\. This permits stopping based on those costs\. It does not makesm/ms\_\{m\}/mat the stopping time unbiased\.

The implementation batches as many next complete paths as the remaining worst\-case budget permits, charges actual cost, and repeats while at leastKKunits remain\. Batching uses no unobserved outcomes and is equivalent to revealing the corresponding segment of the fixed random permutation\.

### B\.2Task\-balanced completion

Independently permute theLLpaths within each task\. For each repetition round, independently permute theMMtask identities and query their next paths in that order\. At a fixed completed\-path countm<Mm<M, the observed tasks form a uniform sample ofmmtasks, each with one uniform inner draw\. Atm≥Mm\\geq M, every task has a countmim\_\{i\}determined independently of outcomes, and the estimate isM−1​∑iY¯iM^\{\-1\}\\sum\_\{i\}\\bar\{Y\}\_\{i\}\. Conditional on the allocation counts it is unbiased\. Ifm<Mm<M, average only the observed tasks; uniform outer sampling restores unbiasedness\.

Form<Mm<Muse the bounded Chernoff interval from Appendix[C](https://arxiv.org/html/2609.29140#A3)with no prefix\. Form≥Mm\\geq Mapply that interval to theMMindependent task averages and intersect it, usingα/2\\alpha/2for each bound, with a Hoeffding interval of radius

2​V​log⁡\(4/α\),V=∑i=1Mmin⁡\(mi,L−mi\)4​M2​mi2\.\\sqrt\{2V\\log\(4/\\alpha\)\},\\qquad V=\\sum\_\{i=1\}^\{M\}\\frac\{\\min\(m\_\{i\},L\-m\_\{i\}\)\}\{4M^\{2\}m\_\{i\}^\{2\}\}\.\(22\)The complement factor gives zero width at census\. For cost\-stopped task balancing the observed counts depend on outcomes, and some tasks may be unobserved\. We retain the average of available per\-task means as a practical baseline but attach no design\-unbiasedness or statistical CI claim\. Exhaustive examples with two or three tasks show absolute bias as large as0\.1250\.125; these are counterexamples to a general claim, not estimates of bias in the public banks\.

### B\.3Pooled progressive weights and budget reservation

Before a uniform thinning there arennactive paths with common weightWW\. LetHHof them have terminal label one\. Retainmmpaths uniformly and writeSSfor the number of terminal positives retained, whether or not they are yet observed\. The latent contribution changes fromW​H/NWH/NtoW​n​S/\(N​m\)WnS/\(Nm\)and has conditional expectationW​H/NWH/N\. Removing a path after a decisive event changes no latent terminal\-positive total\. Thus these weighted totals form a martingale over the evaluator’s random choices, conditional on the entire potential cohort\. At termination the latent total equals the observable estimator\. The product of conditional selection fractions is used recursively; no assertion equates it with a marginal inclusion probability\([Lumley, 2024](https://arxiv.org/html/2609.29140#bib.bib14)\)\.

For an advance fromaatobb, cost is at mostm⁡\(b−a\)m\(b\-a\)\. Enforcing

m≤⌊R−\(K−b\)b−a⌋m\\leq\\left\\lfloor\\frac\{R\-\(K\-b\)\}\{b\-a\}\\right\\rfloor\(23\)when the remaining budget isRRleavesK−bK\-bunits for one surviving path\. At the first stageB≥KB\\geq Kpermits at least one path\. Induction preserves that reservation until depthKK; if the active set becomes empty the score contribution is exactly zero and the policy stops\. The estimator is bounded by one because after every thinning/advance the weighted active count is nonincreasing from its initial valueNN\.

### B\.4Frozen working\-model allocation

The depth schedule is the sorted unique set\{min⁡\(2j,K\):0≤j≤⌈log2⁡K⌉\}∪\{K\}\\\{\\min\(2^\{j\},K\):0\\leq j\\leq\\lceil\\log\_\{2\}K\\rceil\\\}\\cup\\\{K\\\}\. WhenK\>1K\>1, the initial proposal retainsmax⁡\(1,⌊B/2⌋\)\\max\(1,\\lfloor B/2\\rfloor\)paths, subject to the risk set and reservation cap\. Later proposals fit a working conditional\-survival model\[\(1\+b\)/\(1\+a\)\]−γ\[\(1\+b\)/\(1\+a\)\]^\{\-\\gamma\}to at most the three most recent observed stages\. Each survivor and decisive\-event count receives a1/21/2pseudocount\. The negative binomial\-count log\-likelihood is minimized overγ∈\[0\.005,5\]\\gamma\\in\[0\.005,5\]with a bounded scalar optimizer\. No calibration guarantee for this model is assumed or used in inference\.

At current depthaa, setμ⁡\(u\)=\[\(1\+u\)/\(1\+a\)\]−γ\\mu\(u\)=\[\(1\+u\)/\(1\+a\)\]^\{\-\\gamma\}for future depths\. For each remaining interval\(aℓ,bℓ\]\(a\_\{\\ell\},b\_\{\\ell\}\]define

cℓ=∑u=aℓbℓ−1μ⁡\(u\),vℓ=μ​\(bℓ\)−1−μ​\(aℓ\)−1\.c\_\{\\ell\}=\\sum\_\{u=a\_\{\\ell\}\}^\{b\_\{\\ell\}\-1\}\\mu\(u\),\\qquad v\_\{\\ell\}=\\mu\(b\_\{\\ell\}\)^\{\-1\}\-\\mu\(a\_\{\\ell\}\)^\{\-1\}\.\(24\)The usual working variance/cost allocation minimizes∑ℓvℓ/pℓ\\sum\_\{\\ell\}v\_\{\\ell\}/p\_\{\\ell\}subject to∑ℓcℓ​pℓ≤R/n\\sum\_\{\\ell\}c\_\{\\ell\}p\_\{\\ell\}\\leq R/nand1≥p1≥p2≥⋯\>01\\geq p\_\{1\}\\geq p\_\{2\}\\geq\\cdots\>0\. Pool adjacent intervals whenever theirv/cv/cratios increase, assigning each pooled blockp=min⁡\(1,λ​∑v/∑c\)p=\\min\(1,\\lambda\\sqrt\{\\sum v/\\sum c\}\)\. A fixed 50\-step bisection choosesλ\\lambda\. Only the first proposed inclusion is applied; the model is refit after new observations\. The proposed count ismax⁡\(1,⌊n​p1⌋\)\\max\(1,\\lfloor np\_\{1\}\\rfloor\), clipped to the hard reservation cap andnn\. This standard allocation construction is a scheduling heuristic here, not a new optimality theorem\. Its misspecification does not change the unbiasedness argument\.

### B\.5Forward confidence inversion: restricted policy class

LetGaG\_\{a\}andGbG\_\{b\}be the numbers of paths in the original cohort with event times exceeding depthsaaandbb\. For an anonymous policy using only aggregate past event counts and costs, the active paths are a uniform subset of the originalGaG\_\{a\}survivors conditional on that aggregate history\. All paths that surviveaahave identical earlier event/cost contributions\. Uniform thinning to sizemmpreserves this symmetry, so the next observed survivor count obeys

s\|aggregate past∼Hypergeom⁡\(Ga,Gb,m\)\.s\\mid\\text\{aggregate past\}\\sim\\operatorname\{Hypergeom\}\(G\_\{a\},G\_\{b\},m\)\.\(25\)This statement can fail if allocation uses features distinguishing survivors\.

Let\[La,Ua\]\[L\_\{a\},U\_\{a\}\]be the current confidence range forGaG\_\{a\}\. For each feasible population size invertHypergeom⁡\(Ga,Gb,m\)\\operatorname\{Hypergeom\}\(G\_\{a\},G\_\{b\},m\)in its success\-total argumentGbG\_\{b\}, using tailsα/\(2​J\)\\alpha/\(2J\), whereJJis the predetermined maximum number of depth intervals\. Hypergeometric endpoints are nondecreasing in population size, so the union over plausibleGaG\_\{a\}is enclosed by usingmax⁡\(La,m\)\\max\(L\_\{a\},m\)for the lower endpoint andUaU\_\{a\}for the upper\. If the feasible set is empty, return the full population interval conservatively\. A union bound over stages gives final coverage at least1−α1\-\\alpha\. Extinguished stages are padded to the predeterminedJJ, so the error allocation never depends on the observed stopping stage\. The code checks endpoint monotonicity and the conditional transition law by exhaustive enumeration\.

### B\.6Hierarchical progressive design and mixture interval

LetAAbe the global task\-selection weight andWiW\_\{i\}a retained task’s within\-task weight\. Its latent terminal\-positive count isHiH\_\{i\}\. The current weighted target isA​∑iWi​Hi/NA\\sum\_\{i\}W\_\{i\}H\_\{i\}/N\. InitiallyA=Wi=1A=W\_\{i\}=1andWi​ni≤LW\_\{i\}n\_\{i\}\\leq Lremains true after all within\-task thinnings and event removals\.

Given a proposed total ofqqretained paths, ifq<Gq<Guniformly retainqqof theGGactive tasks and multiplyAAbyG/qG/q\. Otherwise keep all active tasks\. Allocate at least one path per retained task by deterministic water filling, then uniformly retainmim\_\{i\}of itsnin\_\{i\}paths and multiplyWiW\_\{i\}byni/min\_\{i\}/m\_\{i\}\. Both operations preserve the latent target in conditional expectation\.

IfR≥G⁡\(K−a\)R\\geq G\(K\-a\), reserve one terminal path per active task and impose

q≤⌊R−G⁡\(K−b\)b−a⌋,q≥G\.q\\leq\\left\\lfloor\\frac\{R\-G\(K\-b\)\}\{b\-a\}\\right\\rfloor,\\qquad q\\geq G\.\(26\)This condition persists at subsequent stages because the active task count cannot increase\. Otherwise use the one\-global\-path reservation\. Working\-model proposals and geometric depths are the same as in the pooled design\.

For a uniform sample ofmmout ofnnvalues in\[0,1\]\[0,1\], the centered sample sum has sub\-Gaussian variance proxymin⁡\(m,n−m\)/4\\min\(m,n\-m\)/4\. Apply Hoeffding’s sampling\-without\-replacement comparison either to the sample or to its complement\([Hoeffding, 1963](https://arxiv.org/html/2609.29140#bib.bib18)\)\. At a whole\-task gate, valuesXi=Wi​Hi/LX\_\{i\}=W\_\{i\}H\_\{i\}/Llie in\[0,1\]\[0,1\], and the increment coefficient isc=A​G/\(M​q\)c=AG/\(Mq\)\. Addc2​min⁡\(q,G−q\)/4c^\{2\}\\min\(q,G\-q\)/4to the cumulative proxyVV\. At a within\-task gate the coefficient isci=A​Wi​ni/\(N​mi\)c\_\{i\}=AW\_\{i\}n\_\{i\}/\(Nm\_\{i\}\), withA,WiA,W\_\{i\}measured before that gate, and the increment contributesci2​min⁡\(mi,ni−mi\)/4c\_\{i\}^\{2\}\\min\(m\_\{i\},n\_\{i\}\-m\_\{i\}\)/4\. Identity gates contribute zero\.

These proxies are known before their respective random gates\. Standard Gaussian mixing of the exponential test supermartingales\([Howard et al\., 2021](https://arxiv.org/html/2609.29140#bib.bib16)\)gives

Et=ρρ\+Vt​exp⁡\{\(Zt−θ𝒞\)22​\(ρ\+Vt\)\},ρ=K4​B\.E\_\{t\}=\\sqrt\{\\frac\{\\rho\}\{\\rho\+V\_\{t\}\}\}\\exp\\left\\\{\\frac\{\(Z\_\{t\}\-\\theta\_\{\\mathcal\{C\}\}\)^\{2\}\}\{2\(\\rho\+V\_\{t\}\)\}\\right\\\},\\qquad\\rho=\\frac\{K\}\{4B\}\.\(27\)The initial mixture scale is fixed before observing the cohort\. At terminationZtZ\_\{t\}is the reported estimate, and invertingEt<1/αE\_\{t\}<1/\\alphayields radius

\(ρ\+Vt\)​\{2​log⁡\(1/α\)\+log⁡\(1\+Vt/ρ\)\}\.\\sqrt\{\(\\rho\+V\_\{t\}\)\\\{2\\log\(1/\\alpha\)\+\\log\(1\+V\_\{t\}/\\rho\)\\\}\}\.\(28\)Clip the interval to\[0,1\]\[0,1\]; if no random gate occurs, the cohort score is known exactly\. This conservative bound uses no estimated outcome variance\.

## Appendix CFull\-prefix sampling proof and uncertainty

Letddbe the full\-cohort prefix depth,CdC\_\{d\}its actual charged cost,nin\_\{i\}the task risk\-set sizes, andG=\|\{i:ni\>0\}\|G=\|\\\{i:n\_\{i\}\>0\\\}\|\. These are fixed conditional on the cohort because the prefix queries every unresolved path\. The remaining budget safely finishesq=min⁡\{∑ini,⌊\(B−Cd\)/\(K−d\)⌋\}q=\\min\\\{\\sum\_\{i\}n\_\{i\},\\lfloor\(B\-C\_\{d\}\)/\(K\-d\)\\rfloor\\\}paths\. Ifq<Gq<G, uniformly selectqqactive tasks and one unresolved path per selected task\. Otherwise, distribute fixed countsmi≥1m\_\{i\}\\geq 1by deterministic water filling the observednin\_\{i\}, then sample uniformly within tasks\. Terminal savings are not recycled\. With sampled terminal averagesY¯i\\bar\{Y\}\_\{i\}, the estimator is

θ^prefix=\{GM​q​∑i∈SniL​Y¯i,q<G,1M∑i:ni\>0niLY¯i,q≥G\.\\widehat\{\\theta\}\_\{\\rm prefix\}=\\begin\{cases\}\\dfrac\{G\}\{Mq\}\\sum\_\{i\\in S\}\\dfrac\{n\_\{i\}\}\{L\}\\bar\{Y\}\_\{i\},&q<G,\\\\\[3\.0pt\] \\dfrac\{1\}\{M\}\\sum\_\{i:n\_\{i\}\>0\}\\dfrac\{n\_\{i\}\}\{L\}\\bar\{Y\}\_\{i\},&q\\geq G\.\\end\{cases\}\(29\)If no path remains or the prefix reachesKK, report its exact census score\. This is an ordinary two\-stage sampling estimator, not a new weighting identity\.

###### Proposition 7\(Full\-prefix validity\)\.

The full\-prefix estimator is design\-unbiased and spends at mostBB\. Ifq<Gq<G, writeZ¯=q−1​∑i∈S\(ni/L\)​Y¯i\\bar\{Z\}=q^\{\-1\}\\sum\_\{i\\in S\}\(n\_\{i\}/L\)\\bar\{Y\}\_\{i\}\. A1−α1\-\\alphainterval is\(G/M\)\(G/M\)times the set ofμ∈\[0,1\]\\mu\\in\[0,1\]satisfying

q​kl⁡\(Z¯,μ\)≤log⁡\(2/α\)\.q\\operatorname\{kl\}\(\\bar\{Z\},\\mu\)\\leq\\log\(2/\\alpha\)\.\(30\)If allGGactive tasks are sampled, the same bound usesGGtask averages; it may be intersected, with an error\-probability split, with the finite\-complement bound below\.

### C\.1Budget and unbiasedness

The prefix allowance ismin⁡\(⌊B/4⌋,B−K\+1\)\\min\(\\lfloor B/4\\rfloor,B\-K\+1\)\. Starting with allNNpaths at depth zero, advance the whole active set by the largest integer span whose worst\-case cost fits the remaining allowance, capped atKK\. Charge the actual cost and repeat\. Before any random subsampling, this fixesd,Cd,ni,Gd,C\_\{d\},n\_\{i\},Gconditional on the potential cohort\. If no paths remain or depthKKis reached, return the known score exactly\.

Otherwise at least one terminal continuation is affordable: ifd=0d=0,B≥KB\\geq K; ifd≥1d\\geq 1,Cd≤B−K\+1C\_\{d\}\\leq B\-K\+1leaves at leastK−dK\-dunits\. The fixed terminal countqqtherefore satisfiesq≥1q\\geq 1andCd\+q⁡\(K−d\)≤BC\_\{d\}\+q\(K\-d\)\\leq B\. LetHiH\_\{i\}count the original task’s paths with terminal label one\. All lie in its prefix risk set, and a uniform sample from that set has meanHi/niH\_\{i\}/n\_\{i\}\. Thus𝔼⁡\[\(ni/L\)​Y¯i\]=Hi/L\\mathbb\{E\}\[\(n\_\{i\}/L\)\\bar\{Y\}\_\{i\}\]=H\_\{i\}/L\. Uniform outer task sampling whenq<Gq<G, or summation over all active tasks whenq≥Gq\\geq G, proves unbiasedness of Equation \([29](https://arxiv.org/html/2609.29140#A3.E29)\)\.

### C\.2Chernoff domination for partial task coverage

For each active task independently presample one inner path, including tasks that will not be retained\. This is a proof coupling, not an extra observation or uncharged evaluation\. WriteZi=\(ni/L\)​Yi∈\[0,1\]Z\_\{i\}=\(n\_\{i\}/L\)Y\_\{i\}\\in\[0,1\],Xi=Hi/LX\_\{i\}=H\_\{i\}/L, andμ=G−1​∑iXi\\mu=G^\{\-1\}\\sum\_\{i\}X\_\{i\}\. For anyλ∈ℝ\\lambda\\in\\mathbb\{R\}letai=𝔼​eλ​Zi\>0a\_\{i\}=\\mathbb\{E\}e^\{\\lambda Z\_\{i\}\}\>0\. Independently choose the uniform outer subsetSSof sizeqq\. Then

𝔼​eλ​∑i∈SZi\\displaystyle\\mathbb\{E\}e^\{\\lambda\\sum\_\{i\\in S\}Z\_\{i\}\}=\(Gq\)−1​∑\|S\|=q∏i∈Sai\\displaystyle=\\binom\{G\}\{q\}^\{\-1\}\\sum\_\{\|S\|=q\}\\prod\_\{i\\in S\}a\_\{i\}\(31\)≤\(1G​∑iai\)q≤\(1−μ\+μ​eλ\)q\.\\displaystyle\\leq\\left\(\\frac\{1\}\{G\}\\sum\_\{i\}a\_\{i\}\\right\)^\{q\}\\leq\(1\-\\mu\+\\mu e^\{\\lambda\}\)^\{q\}\.\(32\)The first inequality is Maclaurin’s elementary\-symmetric\-mean inequality\. For completeness, with all other coordinates fixed andai\+aja\_\{i\}\+a\_\{j\}fixed, the elementary symmetric polynomial is a constant plus a nonnegative multiple ofai​aja\_\{i\}a\_\{j\}; averaging the pair cannot decrease it\. Repeated averaging at fixed total achieves the equal\-coordinate maximum\. The second inequality follows fromeλ​z≤1−z\+z​eλe^\{\\lambda z\}\\leq 1\-z\+ze^\{\\lambda\}on\[0,1\]\[0,1\]and𝔼​Zi=Xi\\mathbb\{E\}Z\_\{i\}=X\_\{i\}\.

Optimizing the Chernoff bound forZ¯≥x\>μ\\bar\{Z\}\\geq x\>\\muorZ¯≤x<μ\\bar\{Z\}\\leq x<\\mugivesexp⁡\[−q​kl⁡\(x,μ\)\]\\exp\[\-q\\operatorname\{kl\}\(x,\\mu\)\]\. A two\-tail union bound and inversion yield Equation \([30](https://arxiv.org/html/2609.29140#A3.E30)\)\. No identical\-distribution assumption on tasks or probabilistic assumption on the fixed cohort was used\. AtZ¯=0\\bar\{Z\}=0or one the interval has the corresponding exact analytic KL endpoint; other endpoints are solved numerically\. Our bound is deliberately not asserted to be the sharpest possible bounded\-variable interval\.

### C\.3All active tasks and finite complements

Whenq≥Gq\\geq G, the countsmim\_\{i\}are functions of the fixed observed prefix\. The independent task\-level variablesZi=\(ni/L\)​Y¯iZ\_\{i\}=\(n\_\{i\}/L\)\\bar\{Y\}\_\{i\}still lie in\[0,1\]\[0,1\]with meansHi/LH\_\{i\}/L\. Arithmetic\-geometric mean and the same convexity argument give Equation \([32](https://arxiv.org/html/2609.29140#A3.E32)\) withq=Gq=Gfor their full sum\.

LetSiS\_\{i\}be the positive count in the within\-task sample, andpi=Hi/nip\_\{i\}=H\_\{i\}/n\_\{i\}\. Its contribution to estimation error is

niN​mi​\(Si−mi​pi\)\.\\frac\{n\_\{i\}\}\{Nm\_\{i\}\}\(S\_\{i\}\-m\_\{i\}p\_\{i\}\)\.\(33\)Applying the sample/complement Hoeffding bound independently within tasks gives variance proxy

V=∑i:ni\>0\(niN​mi\)2min⁡\(mi,ni−mi\)4\.V=\\sum\_\{i:n\_\{i\}\>0\}\\left\(\\frac\{n\_\{i\}\}\{Nm\_\{i\}\}\\right\)^\{2\}\\frac\{\\min\(m\_\{i\},n\_\{i\}\-m\_\{i\}\)\}\{4\}\.\(34\)The two\-sided radius at failure probabilityα/2\\alpha/2is2​V​log⁡\(4/α\)\\sqrt\{2V\\log\(4/\\alpha\)\}\. Intersect it with the KL interval also at failure probabilityα/2\\alpha/2\. The union bound preserves total coverage1−α1\-\\alphadespite this data\-dependent intersection\. At censusV=0V=0and all terminal labels of unresolved paths are known\.

### C\.4Oracle variance decomposition for one draw per sampled task

The special caseq≤Gq\\leq Goffers a useful decomposition\. SetXi=Hi/LX\_\{i\}=H\_\{i\}/Landvi=\(ni​Hi−Hi2\)/L2=Var⁡\(Zi\)v\_\{i\}=\(n\_\{i\}H\_\{i\}\-H\_\{i\}^\{2\}\)/L^\{2\}=\\operatorname\{Var\}\(Z\_\{i\}\)\. ForG\>1G\>1letSX2S\_\{X\}^\{2\}be the finite population variance ofXiX\_\{i\}with denominatorG−1G\-1\. The law of total variance over the selected task set gives

Var⁡\(θ^\)=\(GM\)2​1q​\[\(1−qG\)​SX2\+1G​∑ivi\]\.\\operatorname\{Var\}\(\\widehat\{\\theta\}\)=\\left\(\\frac\{G\}\{M\}\\right\)^\{2\}\\frac\{1\}\{q\}\\left\[\\left\(1\-\\frac\{q\}\{G\}\\right\)S\_\{X\}^\{2\}\+\\frac\{1\}\{G\}\\sum\_\{i\}v\_\{i\}\\right\]\.\(35\)ForG=1G=1the between\-task term is zero\. Task coverage removes the first component, whereas informative prefixes can reduce the second\. This is a classical two\-stage variance decomposition, not an additional new estimator\. It uses unknownHiH\_\{i\}and is an oracle explanation, not free information available to the policy\.

## Appendix DA conditional bridge to a population target

Suppose the planned path outcomes are independent acrossi,ri,rand, within each fixed taskii, have common survival probabilitypi​\(K\)p\_\{i\}\(K\)\. No assumption of independence among theKKattempts within a path is needed for this definition\. Then𝔼​θ𝒞=θ∗=M−1​∑ipi​\(K\)\\mathbb\{E\}\\theta\_\{\\mathcal\{C\}\}=\\theta\_\{\*\}=M^\{\-1\}\\sum\_\{i\}p\_\{i\}\(K\)and Hoeffding’s inequality gives

ℙ\{\|θ𝒞−θ∗\|\>log⁡\(2/β\)2​M​L\}≤β\.\\mathbb\{P\}\\left\\\{\|\\theta\_\{\\mathcal\{C\}\}\-\\theta\_\{\*\}\|\>\\sqrt\{\\frac\{\\log\(2/\\beta\)\}\{2ML\}\}\\right\\\}\\leq\\beta\.\(36\)If\[l⁡\(D\),u⁡\(D\)\]\[l\(D\),u\(D\)\]has conditional design coverage1−α1\-\\alphafor every cohort, expanding both endpoints by that radius gives joint coverage at least1−α−β1\-\\alpha\-\\betaforθ∗\\theta\_\{\*\}over cohort generation and design randomization\. The proof is a union bound; no independence between the two error events is required\. To obtain a total nominal 95% statement, the probabilities must be split in advance, not appended to a 95% conditional interval for free\.

For independent identically distributed Bernoulli attempts within a task,pi​\(K\)p\_\{i\}\(K\)becomes\(1−pi\)K\(1\-p\_\{i\}\)^\{K\}for first success andpiKp\_\{i\}^\{K\}for first failure\. Without that additional assumption, those power identities need not hold\. Shared execution conditions can also violate independence across paths\. Accordingly, this bridge is not used to relabel the public design\-coverage results as latent\-reliability guarantees\.

## Appendix EMissing agent outcomes and reproducibility checks

The agent source contains 120 grading runs\. The data adapter verifies source Git\-blob hashes, run indices, submitted/completed/empty\-patch/error ID partitions, and common task universes\. In the one incomplete configuration, nine tasks lack a submitted patch in run 8\. Keeping the 491 tasks with all ten observed submissions changes that configuration’s estimand\. It does not establish that excluded tasks are exchangeable with retained tasks\.

Letw=Mc/500w=M\_\{c\}/500be the complete\-task fraction\. Because each omitted task’s score lies in\[0,1\]\[0,1\], a confidence interval\[l,u\]\[l,u\]for the complete\-task mean implies\[w​l,w​u\+1−w\]\[wl,wu\+1\-w\]for the all\-task mean, with the same design coverage for every assignment of missing outcomes\. ForMc=491M\_\{c\}=491, the additional identification width is0\.0180\.018\. This simple extension is conservative because it ignores the observed runs of omitted tasks\. We retain the full known/unknown matrix so tighter task/path\-specific sensitivity analyses remain reproducible\.

Exact small\-cohort checks cover sampling unbiasedness, finite budget, interval coverage, hypergeometric endpoint monotonicity, conditional transition laws, and e\-value expectations\. Additional checks enumerate the two lower\-bound priors and compute finite\-MMtrimmed interval\-length bounds\. Such checks are useful for implementation validation but do not establish sharpness from small conservative coverage examples\. All randomized replay outputs include the fixed target, method, budget, seed index, estimate, interval when claimed, and charged cost\. Policies and the prospective protocol are hash\-locked; data schema corrections and the missingness deviation are recorded separately\.

## Appendix FData inventory and additional empirical accounting

#### Design, budgets, and study status\.

The original experiment protocol, policy code, seeds, budgets, and main comparisons were archived before confirmatory outcomes were read\. Earlier exploration used only the first half of four Llama\-3\-8B response banks\. Replication studies and interval refinements are separately declared extensions\. Protocol hashes document local content consistency, not independently witnessed preregistration or historical blindness\.

For responses, we use the last 5000 outcomes per task in all 22 public panels of[Brown et al\. \(2024\)](https://arxiv.org/html/2609.29140#bib.bib2), across mathematics, code, and formal proof\. We form nonoverlapping blocks atK∈\{10,100,1000\}K\\in\\\{10,100,1000\\\}\. AtK=1000K=1000the budgets are104,3⋅104,10510^\{4\},3\\cdot 10^\{4\},10^\{5\}, with 1000 evaluator randomizations per cell/method; at shorter horizons we addB=2000B=2000and use 200 randomizations\.

For agents, we use the ten numbered SWE\-bench grading runs for each of twelve model, scaffold, and temperature configurations released by[Bjarnason et al\. \(2026\)](https://arxiv.org/html/2609.29140#bib.bib4)\. Eleven cover all 500 tasks\. One DeepSWE\-preview / R2E\-Gym run has nine incomplete submissions; its configuration uses the 491 tasks observed in every run\. Missing submissions are not relabeled as observed failures\. Submitted empty patches and grading errors count as operational non\-resolutions under the resolved\-ID label rule\. We examineK∈\{1,2,5,10\}K\\in\\\{1,2,5,10\\\}and budgets500,1500,3000,5000500,1500,3000,5000, with 1000 randomizations atK=5K=5and 200 otherwise\. Appendix[E](https://arxiv.org/html/2609.29140#A5)gives the assumption\-free extension from the complete\-case target to the all\-500 score\.

### F\.1Accuracy gains depend on the baseline and budget

The completed grid contains 868 cohort/event/horizon/budget conditions, seven methods, and 2,492,000 randomized method executions, with no hard\-budget violations\. This count measures replay precision, not independent evidence from millions of new agent runs\. Table[2](https://arxiv.org/html/2609.29140#A6.T2)reports all primary conditions; Figure[3](https://arxiv.org/html/2609.29140#A6.F3)examines the proof\-evaluation condition specified in the prospective protocol because it motivated the study\.

AtB=10,000B=10\{,\}000and30,00030\{,\}000, task\-aware progressive sampling reduces MSE relative to task\-balanced adaptive completion by6\.05×6\.05\\timesand8\.04×8\.04\\times\(95% replay\-bootstrap intervals\[5\.26,6\.94\]\[5\.26,6\.94\]and\[7\.00,9\.26\]\[7\.00,9\.26\]\)\. AtB=100,000B=100\{,\}000, that advantage disappears: its MSE\-gain ratio is0\.940\.94\[0\.83,1\.05\]\[0\.83,1\.05\]\. The pooled design is then worse than task\-balanced completion by3\.02×3\.02\\timesin MSE; this reverses the favorable impression obtained by comparing it only with uniform adaptive completion\.

These gains are not universal, and the implemented width gaps need not be unavoidable \(Appendix[F\.2](https://arxiv.org/html/2609.29140#A6.SS2)\)\. Across 132 primary response conditions, the median task\-aware MSE ratio to task\-balanced adaptive completion is 1\.02; for 96 agent conditions it is 1\.30\. These ratios use the 64 and 57 conditions, respectively, in which both MSEs are nonzero\. Other conditions are retained explicitly in Table[2](https://arxiv.org/html/2609.29140#A6.T2), rather than turning zero reference error into a large but arbitrary finite ratio\. Pooled progressive MSE is worse in the median by factors 1\.46 and 1\.78\. The fixed full\-prefix design also loses on median point accuracy\. Thus the large proof\-benchmark gains do not support a recommendation to replace task\-balanced completion generally\.

The favorable proof case is strongly stratified: 96\.2% of its tasks are internally constant atK=1000K=1000\. From the complete cohort, the design variance of one uniform path per task is5\.21⋅10−55\.21\\cdot 10^\{\-5\}, versus1\.52⋅10−31\.52\\cdot 10^\{\-3\}for the same number of uniformly pooled paths\. This29\.2×29\.2\\timesratio quantifies why task identities matter; it is an oracle\-only explanation, not a free feature available to the allocation policy or a prediction of the adaptive methods’ exact MSE\.

Table 2:All primary conditions: responses atK=1000K=1000\(132 conditions\) and agents atK=5K=5\(96\)\. MSE ratio is method / task\-balanced adaptive; width ratio is method / uniform adaptive\. Brackets contain the 10th and 90th percentiles across conditions, not confidence limits\. Smaller is better\. MSE ratios use 64 response and 57 agent conditions\. “Zero” counts give both methods zero / reference only zero; no candidate\-only zero cases occur\. Width ratios use 65 response and 57 agent conditions\. All excluded width\-zero counts are in Appendix[F](https://arxiv.org/html/2609.29140#A6)\. All widths include the uniform post\-confirmation feasible\-set refinement\. Balanced adaptive has no statistical interval; its separate logical certificate is not included in this 95% comparison\.
### F\.2Honest uncertainty can favor a different design

The strong small\-budget MiniF2F MSE gains still come with wider intervals, but much of the original penalty was avoidable\. After the feasible\-set intersection, task\-aware widths fall from 0\.995 and 0\.635 to 0\.744 and 0\.419, versus 0\.611 and 0\.389 for uniform adaptive completion\. Revised width differences are0\.13360\.1336\[0\.1315,0\.1358\]\[0\.1315,0\.1358\]and0\.03030\.0303\[0\.0291,0\.0317\]\[0\.0291,0\.0317\]under independent replay bootstrap\. The original intervals and every refined cell are retained; estimates, costs, seeds, and empirical coverage are unchanged in all 2\.492 million same\-seed replays\.

AtB=100,000B=100\{,\}000, task\-aware width becomes slightly smaller than uniform adaptive \(0\.2006 versus 0\.2084\)\. Full prefix is narrower still \(0\.0818\), despite not beating balanced adaptive point accuracy; Appendix[F](https://arxiv.org/html/2609.29140#A6)retains the differences and replay\-bootstrap intervals\.

Across primary conditions, however, every nonuniform interval design has a median width ratio above one against uniform adaptive completion \(Table[2](https://arxiv.org/html/2609.29140#A6.T2)\)\. Task\-aware progressive ratios are 1\.66 for responses and 1\.90 for agents\. Figure[2](https://arxiv.org/html/2609.29140#A6.F2)displays the condition\-level spread\. The intervals use different valid concentration constructions; these comparisons rank the*implemented policy–interval pairs*, not the smallest achievable confidence width for each observation design\. In particular, Theorem[2](https://arxiv.org/html/2609.29140#Thmtheorem2)does not prove that the entire observed gap is unavoidable\. Improving conservative adaptive inference remains open\.

### F\.3Coverage, cost, and interpretation

Hierarchical intervals cover in every replay, evidence of conservativeness rather than optimality\. Minimum empirical coverage is 0\.995 for pooled progressive on response banks, 0\.920 on agents, and 0\.915 for uniform fixed in some 200\-replay cells\. Such cell minima do not overturn analytical guarantees; all cell\-wise Monte Carlo intervals and unfavorable values are retained\.

Hard\-budget compliance differs from utilization: mean cost/budget ranges from 0\.39 to 0\.83 for the highlighted adaptive and prefix designs because savings are not always recycled\. Lower use alone does not establish better error per response\. The primaryK=5,L=2K=5,L=2agent grid also cannot activate quarter\-budget prefixing below census; shorter active horizons remain in the artifact\.

The practical implication is conditional\. Include task\-balanced completion when judging point accuracy\. When a valid interval is required, compare a simple fixed or full\-prefix design as well as a sequential confidence procedure, and disclose any resulting change in the accuracy ranking\. Neither a point\-only cost\-stopped estimate nor a narrow population\-model posterior should be silently substituted for the fixed\-cohort coverage requirement used here\.

### F\.4Complete figures and reproducibility accounting

Figure 2:Every primary condition with nonzero reference error or width, after the uniform logical refinement\. MSE reference: task\-balanced adaptive; width reference: uniform adaptive\. Dots are correlated conditions, not independent datasets; black bars are medians\. The dashed line marks parity\. Zero\-reference conditions are separately tabulated, not plotted on logarithmic axes\.Figure 3:Held\-out MiniF2F\-MATH / Llama\-3\-8B\-Instruct, first success,K=1000K=1000\. Lower is better on both axes\. At small budgets, task\-aware progressive sampling has the best point accuracy but a wider interval than uniform adaptive completion\. At the largest budget, task\-balanced adaptive completion has the lowest RMSE; full\-prefix sampling has the narrowest interval\. The adaptive balanced baseline’s purely logical certificate is not plotted\. Curves show 1000\-replay means after the uniform logical refinement\.#### Uniform post\-confirmation inference correction\.

Private review identified that purchased terminal labels imply the deterministic range\[P/N,1−Z/N\]\[P/N,1\-Z/N\]\. The refined analysis intersects this range with every original confidence interval, while preserving all sampling policies and RNGs\. Every one of the 2,492,000 replay estimates and costs matched the original row\. An initial 6076\-cell check also recomputed and matched the original interval endpoints, before the full replay reused saved endpoints to reduce computation\. Both original and revised summaries are included\. Coverage is identical, because the true target always belongs to the feasible range\. The balanced\-adaptive heuristic has only the logical range, with 100% coverage; its median width ratio to uniform\-adaptive 95% intervals is 3\.36 on primary responses and 4\.57 on primary agents\. It is not omitted because of poor point performance or granted a nominal\-95% statistical claim\.

#### Response panels\.

The CodeContests panels have 140 tasks and use Gemma\-2B, Gemma\-7B, Llama\-3\-8B, Llama\-3\-8B\-Instruct, and Llama\-3\-70B\-Instruct\. GSM8K has 127 tasks with Llama\-3\-8B\-Instruct and Llama\-3\-70B\-Instruct\. MATH has 128 tasks with the same five models as CodeContests plus Pythia\-70M, 160M, 410M, 1B, 1\.4B, 2\.8B, 6\.9B, and 12B\. MiniF2F\-MATH has 130 tasks with Llama\-3\-8B\-Instruct and Llama\-3\-70B\-Instruct\. Each panel provides 10,000 correctness outcomes per task; only columns 5000–9999 are used for confirmation\. The source release isScalingIntelligence/monkey\_business, source revisiona9f8f73bcd6948a57ed922cba4e48062ef95f553, converted revision5acc07474317fe5280a2618d2b6652cdb740d101\. The sample release is MIT licensed; original problem licenses remain applicable\. The adapter retrieves only required correctness columns rather than generated responses\.

#### Agent configurations\.

The twelve panels cross Qwen3\-32B, DeepSWE\-preview, and Devstral\-2 with the nano\-agent and R2E\-Gym scaffolds, each at default and zero temperature\. Ten run\-indexed grading files per configuration give the observed terminal resolution labels\. The source isASSERT\-KTH/agentic\-evals\-artifacts, revision5db0c4b69382d160a313d7ceaded915398c63e13, licensed CC BY 4\.0\. These records originate from the repeated coding\-agent study of[Bjarnason et al\. \(2026\)](https://arxiv.org/html/2609.29140#bib.bib4); the present work does not rerun those agents or claim the source’s generation compute as its own\. The complete\-case convention for the single incomplete configuration is stated in the main text\.

#### Zero\-error and zero\-width accounting\.

An empirical zero MSE means every replay estimate equals the known cohort target, up to an absolute score\-roundoff tolerance of10−1210^\{\-12\}; it need not imply that the procedure has certified the score or performed a census\. No positive constant is inserted into a ratio denominator\. In the primary response comparison, uniform adaptive intervals have zero width in 67 of 132 conditions\. Of these 67, the numbers with both widths zero / only the reference width zero are respectively: uniform fixed 0/67, balanced fixed 0/67, pooled progressive 67/0, task\-aware progressive 65/2, full prefix 64/3\. For agents the corresponding denominator\-zero count is 39 of 96; the both/reference\-only counts are uniform fixed 24/15, balanced fixed 24/15, pooled 36/3, task\-aware 36/3, full prefix 24/15\. There are no hidden infinite ratios favoring the proposed designs in these exclusions\.

#### Monte Carlo uncertainty\.

Each primary cell uses 1000 independent evaluator randomizations; secondary cells use 200\. Coverage intervals are two\-sided 95% Clopper–Pearson intervals over those Bernoulli coverage indicators\. Selected MSE\-gain ratios and width differences use 2000 independent bootstrap resamples, separately resampling each method’s stream \(seed 20260916 for original MSE comparisons; 20260925 for the uniformly refined interval widths\)\. This is a numerical precision assessment conditional on one bank, not a test treating repeated tasks, related models, or budget settings as independent scientific replications\. All displayed point estimates, run\-level rows, and bootstrap endpoints are available in the anonymous supplement\.

#### Code validation\.

Exact checks include 600 pooled\-design/inference cases with 7250 randomization leaves; 160 hierarchical cases with 2064 leaves; 480 full\-prefix cases with 3800 leaves; 40 task\-balanced fixed/count\-stopped cases; and 378 one\-per\-task prior\-law checks\. The largest absolute unbiasedness discrepancy in these enumeration families is below1\.6⋅10−151\.6\\cdot 10^\{\-15\}\. A separate 3310\-case hypergeometric coverage enumeration has minimum coverage 0\.950226\. A synthetic access/budget smoke grid tests all seven methods on 756 cases, including early, late, and mixed decisive\-event times\. These finite checks supplement, rather than replace, the general mathematical arguments\. They do not validate the sharpness of confidence bounds from their conservative coverage\.

#### Exploration and reproducibility\.

The four explored panels were Llama\-3\-8B\-Instruct on CodeContests, GSM8K, MATH, and MiniF2F\-MATH, using their first 5000 columns only\. Exploration considered prediction\-powered, multilevel, and profile\-inference variants before the seven\-method confirmation was frozen\. The previously observed11\.24×11\.24\\timespooled MSE gain over uniform adaptive sampling at MiniF2F’s largest budget was not robust to a task\-balanced baseline and is not presented as a confirmatory result\. Confirmatory policies were not retuned after acquisition\. The protocol lock is timestamped and hashed locally; it is not an external registry entry\. The missing\-data handling and filename\-schema adapter fixes are documented as deviations\. Replay and acquisition scripts resolve paths from the artifact root; exact checks run from that root\. Replays use a stable SHA\-256 cell\-to\-seed mapping, not Python’s process\-dependent hash\. The computation uses CPU\-only NumPy/SciPy; no model weights, accelerators, paid API calls, or model training are required\.

## Appendix GReplication validation and post\-review comparisons

The following tables report every two\-path condition:K=1000K=1000,L=5L=5, 500 randomizations\. Hull is the exact\-law count\-only Chernoff bound; KL\-2​M2Muses all sampled labels; KL/H splits error equally between KL and finite\-complement Hoeffding\. D\-Bern is the original disagreement certificate; Pair is the exact pair mixture\. Uniform uses adaptive completion at the same allowed budget, with its own observations\. All bounds use purchased\-label intersection\. The stronger baselines and Pair are post\-validation analyses, not new held\-out experiments \(Appendix[H](https://arxiv.org/html/2609.29140#A8)\)\.

Table 3:Two\-path design, first success\. Mean 95% interval widths\. The first five columns share identical labels; Uniform uses the same allowed budget\.Table 4:Two\-path design, first failure\. Mean 95% interval widths\. The first five columns share identical labels; Uniform uses the same allowed budget\.Minimum empirical coverage across the 36 conditions is 1\.000 for D\-Bern, 0\.998 for Pair, and 0\.998 for KL\-2​M2M\. These numerical checks do not replace the design\-coverage proofs\. The paired 2000\-resample interval comparisons use seeds 20260924 \(original extension\), 20260926 \(stronger\-KL correction\), and 20260927 \(pair mixture\)\. The archive contains all endpoints and unfavorable comparisons\. Same\-budget Uniform reaches zero width in 23/36 conditions; Pair is narrower in only seven, all first\-success conditions\.

## Appendix HStronger fixed\-count baselines and a direct pair envelope

These are post\-second\-review refinements, not changes to the frozen sampling designs\. All counts in this section are fixed before the sampled terminal labels are read\. Neither the KL count argument nor the pair mixture below is asserted for cost\-stopped terminal counts\.

### H\.1Equal\-count KL and complement inversion

LetSiS\_\{i\}be the number of positives in anrr\-draw uniform sample without replacement from taskii, with fixed positive proportionpip\_\{i\}\. Hoeffding’s convex\-order comparison\([Bardenet and Maillard, 2015](https://arxiv.org/html/2609.29140#bib.bib1)\), or the elementary\-symmetric\-mean argument in Appendix[C](https://arxiv.org/html/2609.29140#A3), gives, for every realtt,

𝔼​et​Si\\displaystyle\\mathbb\{E\}e^\{tS\_\{i\}\}≤\(1−pi\+pi​et\)r,\\displaystyle\\leq\(1\-p\_\{i\}\+p\_\{i\}e^\{t\}\)^\{r\},\(37\)𝔼​et​∑iSi\\displaystyle\\mathbb\{E\}e^\{t\\sum\_\{i\}S\_\{i\}\}≤∏i\(1−pi\+pi​et\)r≤\(1−θ𝒞\+θ𝒞​et\)M​r\.\\displaystyle\\leq\\prod\_\{i\}\(1\-p\_\{i\}\+p\_\{i\}e^\{t\}\)^\{r\}\\leq\(1\-\\theta\_\{\\mathcal\{C\}\}\+\\theta\_\{\\mathcal\{C\}\}e^\{t\}\)^\{Mr\}\.\(38\)The second line uses independent task randomizations and concavity of the logarithm; the fixed task means need not agree\. Chernoff inversion therefore gives a valid1−α1\-\\alphainterval

IKL=\{μ∈\[0,1\]:M​r​kl⁡\(θ^,μ\)≤log⁡\(2/α\)\}\.I\_\{\\rm KL\}=\\\{\\mu\\in\[0,1\]:Mr\\,\\operatorname\{kl\}\(\\widehat\{\\theta\},\\mu\)\\leq\\log\(2/\\alpha\)\\\}\.\(39\)This uses allM​rMrlabels, unlike the original extension’s valid but weaker bound that regards only theMMtask averages as bounded observations\.

Forr<Lr<L, the unsampled complement is itself a within\-task uniform sample ofL−rL\-rlabels\. At the true target its aggregate mean isC=\(L​θ𝒞−r​θ^\)/\(L−r\)C=\(L\\theta\_\{\\mathcal\{C\}\}\-r\\widehat\{\\theta\}\)/\(L\-r\)\. Applying the same inequality to that random complement gives another valid confidence set:

IcKL=\{μ∈F:M⁡\(L−r\)​kl⁡\(L​μ−r​θ^L−r,μ\)≤log⁡\(2/α\)\},I\_\{\\rm cKL\}=\\left\\\{\\mu\\in F:M\(L\-r\)\\,\\operatorname\{kl\}\\\!\\left\(\\frac\{L\\mu\-r\\widehat\{\\theta\}\}\{L\-r\},\\mu\\right\)\\leq\\log\(2/\\alpha\)\\right\\\},\(40\)whereF=\[r​θ^/L,1−r⁡\(1−θ^\)/L\]F=\[r\\widehat\{\\theta\}/L,\\,1\-r\(1\-\\widehat\{\\theta\}\)/L\]is the purchased\-label feasible range\. Joint convexity of KL makes this a sublevel interval containingθ^\\widehat\{\\theta\}\. Its endpoints are found by scalar bisection/Brent inversion, not by reading the unsampled labels\. We report sample KL and complement KL separately at levelα\\alpha, and also report two valid intersections with an explicitα/2\\alpha/2allocation: sample KL with complement KL, and sample KL with finite\-complement Hoeffding\. Selecting the narrower nominal\-95% interval without an error adjustment is not used\.

The artifact also retains the originalMM\-KL, Hoeffding, intersection, and SEBB comparisons atr=1,2,3r=1,2,3\. SEBB\-WR and SEBB\-WOR are two separately fixed factor choices from[Burgess and Chapman \(2021\)](https://arxiv.org/html/2609.29140#bib.bib20), Theorem 4\.5, not its optimized allocation algorithm\. On two\-path samples the pair mixture wins 34/36 against2​M2M\-KL and 35/36 against each error\-split intersection \(median ratios 0\.865, 0\.786, 0\.786\)\. These favorable comparisons are supplemented by the stronger, less favorable exact\-law comparison below\.

### H\.2A stronger count\-only finite\-hull Chernoff comparator

The finite task size permits another strengthening without using disagreements\. Letgt​\(h\)=log⁡𝔼h​et​Sig\_\{t\}\(h\)=\\log\\mathbb\{E\}\_\{h\}e^\{tS\_\{i\}\}for the exactHypergeom⁡\(L,h,r\)\\operatorname\{Hypergeom\}\(L,h,r\)task law\. Its least concave majorantg¯t\\bar\{g\}\_\{t\}on\[0,L\]\[0,L\]is the maximum, at each argument, of linear interpolants between bracketing integer grid points\. Equivalently, maximize∑hπh​gt​\(h\)\\sum\_\{h\}\\pi\_\{h\}g\_\{t\}\(h\)over distributions satisfying∑hπh​h=x\\sum\_\{h\}\\pi\_\{h\}h=x; an optimum uses at most two support points\. Hence

log⁡𝔼​et​∑iSi=∑igt​\(hi\)≤M​g¯t​\(∑ihiM\)=M​g¯t​\(L​θ𝒞\)\.\\log\\mathbb\{E\}e^\{t\\sum\_\{i\}S\_\{i\}\}=\\sum\_\{i\}g\_\{t\}\(h\_\{i\}\)\\leq M\\bar\{g\}\_\{t\}\\\!\\left\(\\frac\{\\sum\_\{i\}h\_\{i\}\}\{M\}\\right\)=M\\bar\{g\}\_\{t\}\(L\\theta\_\{\\mathcal\{C\}\}\)\.\(41\)This relaxation allows fractional nuisance composition counts and is therefore conservative for the actual integer cohort\. It assumes neither equal task proportions nor an unproved concavity property of the rawgtg\_\{t\}values\.

We invert the lower\-tail Chernoff bound over a fixed 256\-point tilt gridt=−2zt=\-2^\{z\}, withzzequally spaced from−10\-10to88; binary complementation gives the upper\-tail bound\. The finite grid can only weaken an optimal Chernoff bound\. There is no extra multiplicity penalty across these tilts: for a fixed candidate target, their lower\-tail rejection regions are nested sets of the scalar totalS=∑iSiS=\\sum\_\{i\}S\_\{i\}\. The largest region corresponds to one deterministic threshold selected using that candidate and the design, not usingSS\. Each of the two tails receives error probabilityα/2\\alpha/2\. All bracketing pairs are used for the majorant; log\-sum\-exp and scalar root inversion avoid a composition enumeration over all tasks\.

This comparator uses the same sampled\-positive total as KL, but more of the known finite sampling law\. AtM=128,L=5,r=2M=128,L=5,r=2, its all\-zero upper endpoint is 0\.01128345, narrower than both2​M2M\-KL \(0\.0143\) and the pair mixture \(0\.01335\)\. We retain this adverse example: the rate\-adaptation theorem is over the worst pure cohort, not a promise of improvement at every boundary target\. Exact checks cover 7252 cohort histograms \(minimum coverage 0\.95833\) and 1,856,512 product\-MGF inequalities\. Every two\-path public row is included; the paired bootstrap uses seed 20260928\. This additional comparator was developed after the pair\-mixture results and is explicitly post\-validation\.

### H\.3Exact finite\-pair exponential envelope

FixL≥3L\\geq 3and two uniform draws per task\. For a task withhhpositive paths, setp=h/Lp=h/L,A=\(Y1\+Y2\)/2A=\(Y\_\{1\}\+Y\_\{2\}\)/2, andD=𝟏\{Y1≠Y2\}D=\\mathbf\{1\}\\\{Y\_\{1\}\\neq Y\_\{2\}\\\}\. The three possible\(A,D\)\(A,D\)values are\(0,0\),\(1/2,1\),\(1,0\)\(0,0\),\(1/2,1\),\(1,0\), with probabilities

q0,h=\(L−h\)​\(L−h−1\)L⁡\(L−1\),q1,h=2​h​\(L−h\)L⁡\(L−1\),q2,h=h⁡\(h−1\)L⁡\(L−1\)\.q\_\{0,h\}=\\frac\{\(L\-h\)\(L\-h\-1\)\}\{L\(L\-1\)\},\\quad q\_\{1,h\}=\\frac\{2h\(L\-h\)\}\{L\(L\-1\)\},\\quad q\_\{2,h\}=\\frac\{h\(h\-1\)\}\{L\(L\-1\)\}\.\(42\)For0<h<L0<h<L, define

f0,h​\(t\)\\displaystyle f\_\{0,h\}\(t\)=q0,h​e−t​p\+q2,h​et⁡\(1−p\),f1,h​\(t\)=q1,h​et⁡\(1/2−p\),\\displaystyle=q\_\{0,h\}e^\{\-tp\}\+q\_\{2,h\}e^\{t\(1\-p\)\},\\qquad f\_\{1,h\}\(t\)=q\_\{1,h\}e^\{t\(1/2\-p\)\},\(43\)ψL​\(t\)\\displaystyle\\psi\_\{L\}\(t\)=max0<h<L⁡log⁡f1,h​\(t\)1−f0,h​\(t\)\.\\displaystyle=\\max\_\{0<h<L\}\\log\\frac\{f\_\{1,h\}\(t\)\}\{1\-f\_\{0,h\}\(t\)\}\.\(44\)The domain is0<t<tcap0<t<t\_\{\\rm cap\}, wheretcapt\_\{\\rm cap\}is the smallest positive solution tof0,h​\(t\)=1f\_\{0,h\}\(t\)=1overhhwithq2,h\>0q\_\{2,h\}\>0\. Every such root exists and is unique:f0,h​\(0\)=1−q1,h<1f\_\{0,h\}\(0\)=1\-q\_\{1,h\}<1, the function is convex, and its positive exponential term diverges\. Theh=1h=1case decreases and imposes no finite upper limit\. Thus this domain is positive and determined only by the knownLL\.

###### Proposition 8\(Finite\-pair envelope\)\.

For every fixed task proportion and everyttin this domain,

𝔼​exp⁡\{±t⁡\(A−p\)−ψL​\(t\)​D\}≤1\.\\mathbb\{E\}\\exp\\\{\\pm t\(A\-p\)\-\\psi\_\{L\}\(t\)D\\\}\\leq 1\.\(45\)Consequently, independent two\-draw randomizations across tasks yield𝔼​exp⁡\{±t​M​\(θ^−θ𝒞\)−ψL​\(t\)​d\}≤1\\mathbb\{E\}\\exp\\\{\\pm tM\(\\widehat\{\\theta\}\-\\theta\_\{\\mathcal\{C\}\}\)\-\\psi\_\{L\}\(t\)d\\\}\\leq 1\.

###### Proof\.

For the positive sign the left side isf0,h​\(t\)\+e−ψL​\(t\)​f1,h​\(t\)≤1f\_\{0,h\}\(t\)\+e^\{\-\\psi\_\{L\}\(t\)\}f\_\{1,h\}\(t\)\\leq 1by construction\. Ath=0,Lh=0,L, bothA−pA\-pandDDare zero and the expectation is exactly one\. ReplacinghhbyL−hL\-hproves the negative\-sign statement with the same maximized envelope\. Independence then permits multiplication\. ∎

### H\.4A fully specified data\-independent mixture

SetJ=⌈log2⁡M⌉\+1J=\\lceil\\log\_\{2\}M\\rceil\+1,t0=tcap​M/\(M\+1\)t\_\{0\}=t\_\{\\rm cap\}M/\(M\+1\), and

tj=t0/2j,wj=1\(j\+1\)​\(j\+2\)​\(0≤j<J\),w∗=1J\+1\.t\_\{j\}=t\_\{0\}/2^\{j\},\\quad w\_\{j\}=\\frac\{1\}\{\(j\+1\)\(j\+2\)\}\\ \(0\\leq j<J\),\\qquad w\_\{\*\}=\\frac\{1\}\{J\+1\}\.\(46\)The weights sum to one, including the constant test value with weightw∗w\_\{\*\}\. The choice depends only onM,LM,L, not on the observed disagreements, cohort labels, or desired empirical win count\. For each sign the mixture

E±​\(μ\)=w∗\+∑j=0J−1wj​exp⁡\{±tj​M​\(θ^−μ\)−ψL​\(tj\)​d\}E\_\{\\pm\}\(\\mu\)=w\_\{\*\}\+\\sum\_\{j=0\}^\{J\-1\}w\_\{j\}\\exp\\\{\\pm t\_\{j\}M\(\\widehat\{\\theta\}\-\\mu\)\-\\psi\_\{L\}\(t\_\{j\}\)d\\\}\(47\)has expectation at most one atμ=θ𝒞\\mu=\\theta\_\{\\mathcal\{C\}\}, by Proposition[8](https://arxiv.org/html/2609.29140#Thmtheorem8)\. Markov’s inequality and a two\-tail union bound give coverage at least1−α1\-\\alphafor\{μ:E\+\(μ\)≤2/α,E−\(μ\)≤2/α\}\\\{\\mu:E\_\{\+\}\(\\mu\)\\leq 2/\\alpha,E\_\{\-\}\(\\mu\)\\leq 2/\\alpha\\\}\. Equivalently, letx≥0x\\geq 0uniquely solve

w∗\+∑j=0J−1wj​exp⁡\{tj​x−ψL​\(tj\)​d\}=2/α\.w\_\{\*\}\+\\sum\_\{j=0\}^\{J\-1\}w\_\{j\}\\exp\\\{t\_\{j\}x\-\\psi\_\{L\}\(t\_\{j\}\)d\\\}=2/\\alpha\.\(48\)The interval is\[θ^−x/M,θ^\+x/M\]∩F\[\\widehat\{\\theta\}\-x/M,\\widehat\{\\theta\}\+x/M\]\\cap F\. Jensen’s inequality givesf0,h\+f1,h≥1f\_\{0,h\}\+f\_\{1,h\}\\geq 1, henceψL≥0\\psi\_\{L\}\\geq 0\. The left side is continuous and strictly increasing, at most one at zero, and diverges\. The code uses log\-sum\-exp and scalar root inversion\. No data\-dependent optimization over uncorrected confidence bounds is used\. Product and mixture inference are classical\([Howard et al\., 2021](https://arxiv.org/html/2609.29140#bib.bib16)\); the specialization here is the exact binary finite\-pair penalty\.

On every pure\-task cohortd=0d=0andθ^=θ𝒞\\widehat\{\\theta\}=\\theta\_\{\\mathcal\{C\}\}\. Thej=0j=0term alone yields

\|I\|≤2​log⁡\(4/α\)M​t0=Oα,L​\(M−1\),\|I\|\\leq\\frac\{2\\log\(4/\\alpha\)\}\{Mt\_\{0\}\}=O\_\{\\alpha,L\}\(M^\{\-1\}\),\(49\)retaining Theorem[6](https://arxiv.org/html/2609.29140#Thmtheorem6)’s sharp order\. AtL=5L=5the numerical domain endpoint istcap=2\.5541281188t\_\{\\rm cap\}=2\.5541281188; atM=130M=130, solving the full mixture gives2​x/M=0\.026286432x/M=0\.02628643\. No optimal\-constant or uniform dominance claim is made\. The rule is fixed\-count and equal\-task\-size; it is not a confidence sequence or an implementation of the unequal\-size prefix extension\.

#### Checks and analysis chronology\.

The exact envelope passed 119,952 signed per\-task MGF checks overL=3L=3–50 and seven task counts\. Exact enumeration of 2914 small cohort histograms with 27,274 randomization leaves gives minimum coverage 0\.99588\. The stronger KL procedures separately pass 7252 exact cohort\-histogram checks\. These diagnostics supplement the proofs and do not establish sharpness\. The envelope and mixture were specified before their public\-row computation, but after those outcomes and the stronger\-baseline comparison were known\. Every two\-path fixed row was then reanalyzed, with no new sampling or discarded conditions\. Paired 2000\-resample bootstrap intervals use seed 20260927; the stronger\-baseline correction uses seed 20260926\. Both are Monte Carlo uncertainty conditional on the fixed cohorts, not new scientific replications\.

## Appendix IComplete horizon, event, and budget stratification

Each entry is the median method/reference ratio across panels with positive numerator and denominator\. MSE reference: task\-balanced adaptive; width reference: uniform adaptive\. A dash denotes no such cells, not parity\. The parenthesized count is the number of ratio\-eligible panels; remaining zero cases and the additional task\-balanced\-fixed width reference are retained in the machine\-readable artifact\. These are descriptive, correlated panel summaries, not population estimates\.

Table 5:Responses,K=10K=10\. Ratios below one favor the row method\.Table 6:Responses,K=100K=100\. Ratios below one favor the row method\.Table 7:Responses,K=1000K=1000\. Ratios below one favor the row method\.Table 8:Agents,K=1K=1\. Ratios below one favor the row method\.Table 9:Agents,K=2K=2\. Ratios below one favor the row method\.Table 10:Agents,K=5K=5\. Ratios below one favor the row method\.Table 11:Agents,K=10K=10\. Ratios below one favor the row method\.
## Appendix JExploratory noncensus agent replication

This post\-review analysis reuses all twelve already\-seen agent configurations atK=2K=2,L=5L=5, two paths per task, andB=2​M​KB=2MK\. It is not held\-out validation\. The declared complete grid uses 500 randomizations \(seed 20260929\) and records code/input hashes before computation: 36,000 executions, three policies, 24 conditions\. No budget violations occurred\. DeepSWE/R2E/default uses 491 complete\-case tasks with the missingness limitation in Appendix[F](https://arxiv.org/html/2609.29140#A6); all others have 500\. Pair and Hull use identical task\-balanced observations; U\-adapt uses cost\-adaptive uniform sampling at the same allowed episode budget\. A subsequent full\-grid correction adds U\-fixed:2​M2Muniformly sampled paths with an exact hypergeometric interval, 12,000 replays \(seed 20261001\)\. This baseline was added after the other results were known\. MSE ratio below is fixed task\-balanced replication / balanced adaptive completion\. Each interval is individually 95%; no uncorrected minimum across intervals is used\.

Table 12:Agent two\-path design, first success\. Mean confidence width, MSE ratio, and realized budget fraction\.Table 13:Agent two\-path design, first failure\. Mean confidence width, MSE ratio, and realized budget fraction\.Pair beats Hull width in 24/24 conditions and U\-adapt in 14/24; all 48 difference Monte Carlo intervals exclude zero \(2000 bootstrap resamples, seed 20260930\)\. However, Pair beats U\-fixed in only 1/24 \(median width ratio 1\.248\), while task\-balanced replication has lower MSE in 24/24 \(median ratio 0\.370\)\. Fixed uniform avoids the sequential confidence penalty, reversing most apparent practical gains\. All 48,000 rows were reaggregated, all Pair/Hull endpoints recomputed, and 144 selected policy runs freshly replayed\. All denominators are positive\. Bootstrap intervals quantify only conditional replay error; independently randomized policies share no common\-random\-number advantage\. Minimum coverage is 0\.998 for Pair and 0\.940 for U\-fixed \(500\-replay binomial interval\[0\.915,0\.959\]\[0\.915,0\.959\]\), not evidence against its exact guarantee\. Balanced adaptive has lower empirical MSE throughout, but not every difference is resolved by its Monte Carlo interval\.

## Appendix KSharp interpolation over task\-covering adaptive policies

This section proves Theorem[3](https://arxiv.org/html/2609.29140#Thmtheorem3)\. The design and theorem were developed after the preceding empirical analyses\. Its exhaustive checks and exploratory agent test are separate from the original locked study\. The upper bound uses a fixed random\-subset design\. The lower bound applies uniformly to all task\-covering adaptive policies with the same hard budget, and hence also to this particular design\. The confidence formula below itself still requires label\-independent audit selection; no task purity is assumed known\. The policy\-class converse was developed after the fixed\-design analysis\.

### K\.1Design and a constructive honest interval

Writepi=L−1​∑rYi​rp\_\{i\}=L^\{\-1\}\\sum\_\{r\}Y\_\{ir\}\. In each task couple the design to a uniform ordered pair of distinct paths, with labels\(Xi,Xi′\)\(X\_\{i\},X^\{\\prime\}\_\{i\}\), independently across tasks\. Choose a uniform subsetSSof exactlytttasks independently of all these draws\. PurchaseXiX\_\{i\}in every task andXi′X^\{\\prime\}\_\{i\}only fori∈Si\\in S\. The other second labels are a proof coupling, not observations\. Every purchased path is completed through the prefix oracle, with reservation\(M\+t\)​K\(M\+t\)K; no saved terminal cost is recycled\. Fori∉Si\\notin SputAi=XiA\_\{i\}=X\_\{i\}; otherwise putAi=\(Xi\+Xi′\)/2A\_\{i\}=\(X\_\{i\}\+X^\{\\prime\}\_\{i\}\)/2\. Conditional on everySS, theAiA\_\{i\}are independent and have meanspip\_\{i\}, soθ^=M−1​∑iAi\\widehat\{\\theta\}=M^\{\-1\}\\sum\_\{i\}A\_\{i\}is unbiased\.

LetDi=𝟏\{Xi≠Xi′\}D\_\{i\}=\\mathbf\{1\}\\\{X\_\{i\}\\neq X^\{\\prime\}\_\{i\}\\\},d=∑i∈SDid=\\sum\_\{i\\in S\}D\_\{i\}, andqi=𝔼​Di=2​L​pi​\(1−pi\)/\(L−1\)q\_\{i\}=\\mathbb\{E\}D\_\{i\}=2Lp\_\{i\}\(1\-p\_\{i\}\)/\(L\-1\), withq¯=M−1​∑iqi\\bar\{q\}=M^\{\-1\}\\sum\_\{i\}q\_\{i\}\. Fort≥1t\\geq 1, the nonnegative elementary\-symmetric\-mean inequality gives, for everyλ≥0\\lambda\\geq 0,

𝔼​e−λ​d\\displaystyle\\mathbb\{E\}e^\{\-\\lambda d\}=\(Mt\)−1​∑\|S\|=t∏i∈S\(1−qi\+qi​e−λ\)\\displaystyle=\\binom\{M\}\{t\}^\{\-1\}\\sum\_\{\|S\|=t\}\\prod\_\{i\\in S\}\(1\-q\_\{i\}\+q\_\{i\}e^\{\-\\lambda\}\)\(50\)≤\(1−q¯\+q¯​e−λ\)t≤exp⁡\{t​q¯​\(e−λ−1\)\}\.\\displaystyle\\leq\(1\-\\bar\{q\}\+\\bar\{q\}e^\{\-\\lambda\}\)^\{t\}\\leq\\exp\\\{t\\bar\{q\}\(e^\{\-\\lambda\}\-1\)\\\}\.This is an unconditional comparison over the randomized subset, not a binomial model forddor a statement conditional on an arbitrarySS\. DefineU⁡\(d,δ,t\)U\(d,\\delta;t\)to be the largestu∈\[d,t\]u\\in\[d,t\]withu−d\+d​log⁡\(d/u\)≤log⁡\(1/δ\)u\-d\+d\\log\(d/u\)\\leq\\log\(1/\\delta\), interpreting0​log⁡0=00\\log 0=0\. The endpoints areU⁡\(0,δ,t\)=min⁡\{t,log⁡\(1/δ\)\}U\(0,\\delta;t\)=\\min\\\{t,\\log\(1/\\delta\)\\\}andU⁡\(t,δ,t\)=tU\(t,\\delta;t\)=t\. Forμ=t​q¯\\mu=t\\bar\{q\}and an integerk<μk<\\mu, optimizing exponential Markov in Equation \([50](https://arxiv.org/html/2609.29140#A11.E50)\) givesℙ⁡\(d≤k\)≤exp⁡\{−μ\+k−k​log⁡\(k/μ\)\}\\mathbb\{P\}\(d\\leq k\)\\leq\\exp\\\{\-\\mu\+k\-k\\log\(k/\\mu\)\\\}\. Fork=0k=0this is the limiting bounde−μe^\{\-\\mu\}\. The eventμ\>U⁡\(d,δ,t\)\\mu\>U\(d,\\delta;t\)is an initial segment of the count values; applying this bound at its last value establishes failure probability at mostδ\\delta\(the empty event causes no difficulty\)\.

Sampling two distinct labels has covariance−pi\(1−pi\)/\(L−1\)\-p\_\{i\}\(1\-p\_\{i\}\)/\(L\-1\)\. Thus, conditional on anySS,

Var⁡\(θ^∣S\)≤V0:=M−2​∑ipi​\(1−pi\)=L−12​L​M​q¯,\|Ai−pi\|/M≤c0:=L−1L​M\.\\operatorname\{Var\}\(\\widehat\{\\theta\}\\mid S\)\\leq V\_\{0\}:=M^\{\-2\}\\sum\_\{i\}p\_\{i\}\(1\-p\_\{i\}\)=\\frac\{L\-1\}\{2LM\}\\bar\{q\},\\qquad\|A\_\{i\}\-p\_\{i\}\|/M\\leq c\_\{0\}:=\\frac\{L\-1\}\{LM\}\.\(51\)The last bound uses the finiteLL\-grid: a nonconstant task’s proportion lies between1/L1/Land1−1/L1\-1/L; the two\-draw bound is smaller still\. Forx=log⁡\(4/α\)x=\\log\(4/\\alpha\), independent\-sum Bernstein conditional on eachSSbounds two\-sided mean error byα/2\\alpha/2at radiusc0​x/3\+2​V0​x\+\(c0​x/3\)2c\_\{0\}x/3\+\\sqrt\{2V\_\{0\}x\+\(c\_\{0\}x/3\)^\{2\}\}\. BothV0V\_\{0\}andc0c\_\{0\}are the same for everySS, so the inequality also holds unconditionally\. On the audit event of probability at least1−α/21\-\\alpha/2,V0V\_\{0\}is at most

VU=min⁡\{14​M,\(L−1\)​U​\(d,α/2,t\)2​L​M​t\}\.V\_\{U\}=\\min\\left\\\{\\frac\{1\}\{4M\},\\frac\{\(L\-1\)U\(d,\\alpha/2;t\)\}\{2LMt\}\\right\\\}\.\(52\)Monotonicity of the radius and a union bound therefore prove honesty ofIaudit=\[θ^−r,θ^\+r\]∩\[0,1\]I\_\{\\rm audit\}=\[\\widehat\{\\theta\}\-r,\\widehat\{\\theta\}\+r\]\\cap\[0,1\], wherer=c0​x/3\+2​VU​x\+\(c0​x/3\)2r=c\_\{0\}x/3\+\\sqrt\{2V\_\{U\}x\+\(c\_\{0\}x/3\)^\{2\}\}\. No independence betweenddandθ^\\widehat\{\\theta\}is invoked\. Att=0t=0use Hoeffding radiuslog⁡\(2/α\)/\(2​M\)\\sqrt\{\\log\(2/\\alpha\)/\(2M\)\}instead\. Purchased\-label bounds can be intersected without error allocation\.

On a pure cohort,d=0d=0andθ^=θ𝒞\\widehat\{\\theta\}=\\theta\_\{\\mathcal\{C\}\}surely\. Fort≥1t\\geq 1, puta=\(L−1\)​log⁡\(2/α\)​x/La=\(L\-1\)\\log\(2/\\alpha\)x/Landb=\(L−1\)​x/\(3​L\)b=\(L\-1\)x/\(3L\)\. The untruncated width is at most2​a/M​t\+4​b/M2\\sqrt\{a\}/\\sqrt\{Mt\}\+4b/M\. Since1≤t≤M1\\leq t\\leq M, this is at most\(2​2​a\+4​2​b\)/M⁡\(t\+1\)\(2\\sqrt\{2a\}\+4\\sqrt\{2\}b\)/\\sqrt\{M\(t\+1\)\}\. The Hoeffding endpoint handlest=0t=0\. These constants are uniform inM,tM,t, for fixedL,αL,\\alpha\.

The exact point\-error identity, useful for checking the implementation, is

𝔼​\(θ^−θ𝒞\)2=∑ipi​\(1−pi\)M2​\(1−tM​L2​\(L−1\)\)\.\\mathbb\{E\}\(\\widehat\{\\theta\}\-\\theta\_\{\\mathcal\{C\}\}\)^\{2\}=\\frac\{\\sum\_\{i\}p\_\{i\}\(1\-p\_\{i\}\)\}\{M^\{2\}\}\\left\(1\-\\frac\{t\}\{M\}\\frac\{L\}\{2\(L\-1\)\}\\right\)\.\(53\)It follows by averaging the conditional variance, because each task is audited with probabilityt/Mt/Mand the conditional mean is constant\. The unknownpip\_\{i\}are available only to the offline analyst, never the policy\.

### K\.2A policy\-uniform late\-event subexperiment

Fix any policy inΠM,t\\Pi\_\{M,t\}and an interval honest on every fixed cohort\. For any binary label table set exactlyTi​r=KT\_\{ir\}=KforYi​r=0Y\_\{ir\}=0andTi​r=K\+1T\_\{ir\}=K\+1forYi​r=1Y\_\{ir\}=1\. Both event times are allowed\. An advance fromaatob<Kb<Kalways survives and costsb−ab\-a, regardless of the label\. An advance toKKcostsK−aK\-aand reveals the terminal label\. Thus each distinct revealed label has cumulative chargeKK; unfinished prefixes add cost but no information\. A pathwise budget\(M\+t\)​K\(M\+t\)Ktherefore implies

ri≥1,n:=∑iri≤M\+t,∑i\(ri−1\)≤t,r\_\{i\}\\geq 1,\\qquad n:=\\sum\_\{i\}r\_\{i\}\\leq M\+t,\\qquad\\sum\_\{i\}\(r\_\{i\}\-1\)\\leq t,\(54\)whererir\_\{i\}counts revealed distinct labels in taskii\. This conclusion uses task coverage, not a predetermined order of visits\. A policy may recycle savings and reveal more labels on other cohorts; the lower\-bound priors below have no such savings\. No padding of the prefix budget is used\.

Putϵ=1/\[4​\(t\+1\)\]\\epsilon=1/\[4\(t\+1\)\]\. Under prior A, tasks are pure with independent fair bits\. Under prior B, each task is independently pure with probability1−ϵ1\-\\epsilon; otherwise it has either one orL−1L\-1positive paths with equal probability and uniformly random placement\. Both are distributions over fixed cohorts, embedded with the event times above\. LetEEbe the event that all revealed labels in each task agree\. LetTTcontain the full transcript and independent policy/interval randomness\. Common unfinished\-prefix observations may be retained inTT\.

For a pure\-compatible transcriptτ\\tau, writebib\_\{i\}for taskii’s first revealed bit andri=ri​\(τ\)r\_\{i\}=r\_\{i\}\(\\tau\)\. Specified distinct positions all having specified bitbib\_\{i\}have probability1/21/2under A andfri/2f\_\{r\_\{i\}\}/2under B, where

f1=1,fr=1−ϵr/L\(2≤r≤L\)\.f\_\{1\}=1,\\qquad f\_\{r\}=1\-\\epsilon r/L\\quad\(2\\leq r\\leq L\)\.\(55\)Forr≥2r\\geq 2, the contaminated component can agree only when its single opposite bit avoids allrrpositions\. The policy’s sequential action factors are the same for the same preceding observations and cancel in a transcript likelihood ratio, even if task identities, path identities, and stopping depend on labels\. Hence, as subprobability measures,

d​ℙB​\(T∈d​τ,E\)=w⁡\(τ\)​d​ℙA​\(T∈d​τ\),w⁡\(τ\)=∏ifri​\(τ\)\.d\\mathbb\{P\}\_\{B\}\(T\\in d\\tau,E\)=w\(\\tau\)d\\mathbb\{P\}\_\{A\}\(T\\in d\\tau\),\\qquad w\(\\tau\)=\\prod\_\{i\}f\_\{r\_\{i\}\(\\tau\)\}\.\(56\)Usingr≤2​\(r−1\)r\\leq 2\(r\-1\)forr≥2r\\geq 2and∏j\(1−aj\)≥1−∑jaj\\prod\_\{j\}\(1\-a\_\{j\}\)\\geq 1\-\\sum\_\{j\}a\_\{j\}gives

1≥w⁡\(τ\)≥1−2​ϵL​\(n−M\)≥1−2​ϵ​tL≥56=:g0\.1\\geq w\(\\tau\)\\geq 1\-\\frac\{2\\epsilon\}\{L\}\(n\-M\)\\geq 1\-\\frac\{2\\epsilon t\}\{L\}\\geq\\frac\{5\}\{6\}=:g\_\{0\}\.\(57\)ThusℙB​\(E\)≥g0\\mathbb\{P\}\_\{B\}\(E\)\\geq g\_\{0\}\. In generalwwdepends onτ\\tau:*the conditional law B givenEEneed not equal A*\.

Task coverage makesm⁡\(T\):=M−1​∑ibim\(T\):=M^\{\-1\}\\sum\_\{i\}b\_\{i\}the true target under A\. The first event below is transcript\-measurable, so likelihood domination and A\-honesty bound it; B\-honesty bounds the second:

ℙB​\{E,m⁡\(T\)∉I\}≤α,ℙB​\{E,θB∉I\}≤α,𝔼A​\|I\|≥𝔼B​\[\|I\|​𝟏E\]\.\\mathbb\{P\}\_\{B\}\\\{E,m\(T\)\\notin I\\\}\\leq\\alpha,\\qquad\\mathbb\{P\}\_\{B\}\\\{E,\\theta\_\{B\}\\notin I\\\}\\leq\\alpha,\\qquad\\mathbb\{E\}\_\{A\}\|I\|\\geq\\mathbb\{E\}\_\{B\}\[\|I\|\\mathbf\{1\}\_\{E\}\]\.\(58\)These are integrated coverage statements, not posterior\-coverage assumptions\. OnEE, failure of simultaneous inclusion has unnormalized probability at most2​α2\\alpha\.

### K\.3Adaptive posterior factorization and separation

Conditional on a complete compatible transcript and its seeds, the policy’s action factors are constants with respect to the latent cohort\. The remaining constraints factor over tasks\. The product B prior consequently gives independent posterior task errorsδi=pi−bi\\delta\_\{i\}=p\_\{i\}\-b\_\{i\}, though their conditional means generally do not vanish\. Forbi=0b\_\{i\}=0, their laws are

Forbi=1b\_\{i\}=1reflect these values\. Atri=1r\_\{i\}=1the variance is

ϵ⁡\(L−1\)L2​\(1−4​ϵ​\(L−1\)L2\)≥ϵ2​L3\.\\frac\{\\epsilon\(L\-1\)\}\{L^\{2\}\}\\left\(1\-\\frac\{4\\epsilon\(L\-1\)\}\{L^\{2\}\}\\right\)\\geq\\frac\{\\epsilon\}\{2L^\{3\}\}\.\(59\)For2≤ri<L2\\leq r\_\{i\}<L,qri≥ϵ/Lq\_\{r\_\{i\}\}\\geq\\epsilon/Landqri≤1/3q\_\{r\_\{i\}\}\\leq 1/3, so the varianceqri​\(1−qri\)/L2q\_\{r\_\{i\}\}\(1\-q\_\{r\_\{i\}\}\)/L^\{2\}has the same lower bound\. At most⌊t/\(L−1\)⌋\\lfloor t/\(L\-1\)\\rfloortasks are fully observed; at leastM/2M/2retain this positive variance\. With all moments below conditional onT,ET,E, write

H=M⁡\(θB−m⁡\(T\)\)=∑iδi,μ=𝔼B​H,v=VarB⁡H≥aL​Mt\+1,aL=116​L3\.H=M\(\\theta\_\{B\}\-m\(T\)\)=\\sum\_\{i\}\\delta\_\{i\},\\quad\\mu=\\mathbb\{E\}\_\{B\}H,\\quad v=\\operatorname\{Var\}\_\{B\}H\\geq a\_\{L\}\\frac\{M\}\{t\+1\},\\qquad a\_\{L\}=\\frac\{1\}\{16L^\{3\}\}\.\(60\)Each centered incrementZi=δi−𝔼​δiZ\_\{i\}=\\delta\_\{i\}\-\\mathbb\{E\}\\delta\_\{i\}has\|Zi\|≤1\|Z\_\{i\}\|\\leq 1\. Thus∑i𝔼​Zi4≤v\\sum\_\{i\}\\mathbb\{E\}Z\_\{i\}^\{4\}\\leq vand\|∑i𝔼​Zi3\|≤v\|\\sum\_\{i\}\\mathbb\{E\}Z\_\{i\}^\{3\}\|\\leq v\. Independence gives, withs=𝔼​H2=μ2\+vs=\\mathbb\{E\}H^\{2\}=\\mu^\{2\}\+v,

𝔼​H4\\displaystyle\\mathbb\{E\}H^\{4\}≤μ4\+6​μ2​v\+4​\|μ\|​v\+3​v2\+v\\displaystyle\\leq\\mu^\{4\}\+6\\mu^\{2\}v\+4\|\\mu\|v\+3v^\{2\}\+v\(61\)≤3​s2\+4​\|μ\|​v\+v\.\\displaystyle\\leq 3s^\{2\}\+4\|\\mu\|v\+v\.The maximum of4​\|μ\|​v/\(μ2\+v\)24\|\\mu\|v/\(\\mu^\{2\}\+v\)^\{2\}overμ\\muis9/\(4​3​v\)9/\(4\\sqrt\{3v\}\), attained at\|μ\|=v/3\|\\mu\|=\\sqrt\{v/3\}, andv/s2≤1/vv/s^\{2\}\\leq 1/v\. Forv≥4v\\geq 4their sum is at most9/\(8​3\)\+1/4<19/\(8\\sqrt\{3\}\)\+1/4<1, so𝔼​H4≤4​s2\\mathbb\{E\}H^\{4\}\\leq 4s^\{2\}\. Paley–Zygmund applied toH2H^\{2\}therefore gives

ℙB\{\|H\|≥s/10∣T,E\}≥81400\.\\mathbb\{P\}\_\{B\}\\\{\|H\|\\geq\\sqrt\{s/10\}\\mid T,E\\\}\\geq\\frac\{81\}\{400\}\.\(62\)IfaL​M/\(t\+1\)≥4a\_\{L\}M/\(t\+1\)\\geq 4, every compatible transcript satisfies this branch\. Sinces≥vs\\geq v, integration overEEand Equation \([58](https://arxiv.org/html/2609.29140#A11.E58)\) give

𝔼A​\|I\|≥c1M⁡\(t\+1\),c1=\(56​81400−2​α\)​aL/10\>0\.\\mathbb\{E\}\_\{A\}\|I\|\\geq\\frac\{c\_\{1\}\}\{\\sqrt\{M\(t\+1\)\}\},\\qquad c\_\{1\}=\\left\(\\frac\{5\}\{6\}\\frac\{81\}\{400\}\-2\\alpha\\right\)\\sqrt\{a\_\{L\}/10\}\>0\.\(63\)Atα=1/12\\alpha=1/12the parenthesis is1/4801/480\. No independence of separation and simultaneous\-coverage events is required\. The supremum over pure cohorts is at least this A\-average\.

### K\.4The small\-scale branch without padding

IfaL​M/\(t\+1\)<4a\_\{L\}M/\(t\+1\)<4, compare the all\-zero cohort lawℙ0\\mathbb\{P\}\_\{0\}with a priorℙ1\\mathbb\{P\}\_\{1\}placing one positive path uniformly among theM​LMLpositions, using the same late\-event embedding\. For a zero\-compatible transcript withn⁡\(τ\)≤M\+tn\(\\tau\)\\leq M\+trevealed labels,

d​ℙ1​\(τ,no hit\)=q⁡\(τ\)​d​ℙ0​\(τ\),q⁡\(τ\)=1−n⁡\(τ\)M​L≥1−M\+tM​L=:qmin≥13\.d\\mathbb\{P\}\_\{1\}\(\\tau,\\text\{no hit\}\)=q\(\\tau\)d\\mathbb\{P\}\_\{0\}\(\\tau\),\\qquad q\(\\tau\)=1\-\\frac\{n\(\\tau\)\}\{ML\}\\geq 1\-\\frac\{M\+t\}\{ML\}=:q\_\{\\min\}\\geq\\frac\{1\}\{3\}\.\(64\)This is a likelihood identity: exactly the unqueried positive locations produce that same zero transcript\. It is not a conditional no\-hit probability given an already observed zero transcript\. Variable stopping makesq⁡\(τ\)q\(\\tau\)nonconstant, but honesty impliesα≥qminℙ0\{1/\(ML\)∉I\}\\alpha\\geq q\_\{\\min\}\\mathbb\{P\}\_\{0\}\\\{1/\(ML\)\\notin I\\\}\. Together withℙ0​\(0∉I\)≤α\\mathbb\{P\}\_\{0\}\(0\\notin I\)\\leq\\alpha, this yields

𝔼0​\|I\|≥1−α−α/qminM​L≥1−4​αM​L\>c2M⁡\(t\+1\),c2=\(1−4​α\)​aL2​L\>0\.\\mathbb\{E\}\_\{0\}\|I\|\\geq\\frac\{1\-\\alpha\-\\alpha/q\_\{\\min\}\}\{ML\}\\geq\\frac\{1\-4\\alpha\}\{ML\}\>\\frac\{c\_\{2\}\}\{\\sqrt\{M\(t\+1\)\}\},\\qquad c\_\{2\}=\\frac\{\(1\-4\\alpha\)\\sqrt\{a\_\{L\}\}\}\{2L\}\>0\.\(65\)The last step usest\+1\>aL​M/4t\+1\>a\_\{L\}M/4\. Takingmin⁡\(c1,c2\)\\min\(c\_\{1\},c\_\{2\}\)gives a lower constant independent of the policy and interval\. Infimizing over all honest task\-covering policy–interval pairs proves Theorem[3](https://arxiv.org/html/2609.29140#Thmtheorem3)\. ForL=2,t=ML=2,t=M, full census is feasible, so that restriction cannot be removed\. Task coverage is also a stated restriction; no result here optimizes over policies allowed to leave tasks unobserved\.

### K\.5Relation to established inference results

Honesty over a larger class while minimizing expected length on a smaller class is a classical adaptation problem\([Cai and Low, 2004](https://arxiv.org/html/2609.29140#bib.bib22)\)\. One\-per\-stratum variance estimation and replication\-based remedies are established in spatial sampling\([Barabesi et al\., 2012](https://arxiv.org/html/2609.29140#bib.bib23)\)\. That analysis uses geometric and smoothness conditions, not arbitrary binary\-cohort honesty\. Empirical Bernstein certificates, including independent nonidentical variables, are classical\([Maurer and Pontil, 2009](https://arxiv.org/html/2609.29140#bib.bib24)\)\. Applying ordinary sample variance across task means still retains between\-task variation on heterogeneous pure cohorts\.[Burgess and Chapman \(2021\)](https://arxiv.org/html/2609.29140#bib.bib20)already give finite\-stratum empirical Bernstein inference; their sampling algorithm initializes at least two observations in every stratum\. The contribution here is the matching uniformM,tM,trate over task\-covering hard\-budget policies, attained by a random\-subset audit, not replication, concentration, or the adaptation formulation themselves\. It does not optimize finite constants or cover policies allowed to omit tasks, and it establishes no general finite\-budget dominance over pooled inference\.

## Appendix LComplete partial\-replication agent exploration

#### Chronology and design\.

After deriving the fixed\-design version of Theorem[3](https://arxiv.org/html/2609.29140#Thmtheorem3), we declared a complete exploratory grid on all twelve previously inspected coding\-agent configurations\. This is not held\-out evidence or a new model\-generation experiment\. The input manifest hashes the protocol, policies, theorem note and all twelve banks before the new computation\. AtK=2,L=5K=2,L=5, both events and 500 evaluator randomizations, we uset∈\{0,⌈M⌉,⌈M/10⌉,⌈M/4⌉,⌈M/2⌉,M\}t\\in\\\{0,\\lceil\\sqrt\{M\}\\rceil,\\lceil M/10\\rceil,\\lceil M/4\\rceil,\\lceil M/2\\rceil,M\\\}andB=2​\(M\+t\)B=2\(M\+t\)\. All 288,000 policy runs are retained\. The491491\-task exception and missingness interpretation are unchanged\. The six choices are not selected from the results\.

#### Comparators and error allocation\.

Partial replication is compared to pooled fixed uniform sampling, uniform adaptive completion, and balanced adaptive completion\. Fixed uniform purchases exactlyM\+tM\+tpaths and uses an exact hypergeometric interval\. It has the same worst\-case reservation and expected realized charge as partial replication, because each path has marginal inclusion\(M\+t\)/\(M​L\)\(M\+t\)/\(ML\)under either design\. Uniform adaptive uses a valid confidence sequence and recycles saved cost\. Balanced adaptive is a point\-estimation heuristic, with only a logical interval\.

On the*same*partial\-replication sample, we compare the Audit interval of Equation \([6](https://arxiv.org/html/2609.29140#S4.E6)\) to a mixed\-count exact\-law Hull interval defined below\. We additionally retain the older bounded\-KL/finite\-complement interval, with fixed countsmi∈\{1,2\}m\_\{i\}\\in\\\{1,2\\\}and Hoeffding variance proxy∑imin⁡\(mi,L−mi\)/\(4​M2​mi2\)\\sum\_\{i\}\\min\(m\_\{i\},L\-m\_\{i\}\)/\(4M^\{2\}m\_\{i\}^\{2\}\), and two valid Bonferroni intersections: Audit/Hull and Audit/older\-bound, each component at failure probabilityα/2\\alpha/2\. An unadjusted minimum of nominal\-95% widths is not a valid selection rule and is not used\. Att=Mt=Mwe separately report the dedicated Pair certificate, rather than representing the generic Audit formula as the strongest two\-path procedure\. All intervals use purchased\-label clipping\.

### L\.1Mixed\-count exact\-law Hull baseline

Here “exact\-law” refers to the finite local sampling distributions\. The composition relaxation and Chernoff inversion yield a conservative interval, not exact tail inversion for the full observation law or an optimal confidence rule for this design\.

Conditional on the preselected subset, use the integer statisticE2=2​∑iAiE\_\{2\}=2\\sum\_\{i\}A\_\{i\}\. For a task withhhpositive paths, its contribution is2​Xi2X\_\{i\}if unaudited, orXi\+Xi′X\_\{i\}\+X^\{\\prime\}\_\{i\}if audited\. For a fixedz<0z<0, define

g1​\(h,z\)\\displaystyle g\_\{1\}\(h,z\)=log⁡\{1−h/L\+\(h/L\)​e2​z\},\\displaystyle=\\log\\\{1\-h/L\+\(h/L\)e^\{2z\}\\\},\(66\)g2​\(h,z\)\\displaystyle g\_\{2\}\(h,z\)=log⁡\{\(L−h\)​\(L−h−1\)\+2​h​\(L−h\)​ez\+h⁡\(h−1\)​e2​zL⁡\(L−1\)\}\.\\displaystyle=\\log\\left\\\{\\frac\{\(L\-h\)\(L\-h\-1\)\+2h\(L\-h\)e^\{z\}\+h\(h\-1\)e^\{2z\}\}\{L\(L\-1\)\}\\right\\\}\.\(67\)Letg¯j​\(⋅,z\)\\bar\{g\}\_\{j\}\(\\cdot,z\)be the least concave majorant of the grid valuesgj​\(0,z\),…,gj​\(L,z\)g\_\{j\}\(0,z\),\\ldots,g\_\{j\}\(L,z\), linearly interpolated between its vertices\. Withn1=M−t,n2=tn\_\{1\}=M\-t,n\_\{2\}=tand total cohort positivesH=M​L​θH=ML\\theta, independence and concavity bound the conditional log MGF by

Gz​\(H\)=max0≤h1,h2≤Ln1​h1\+n2​h2=H⁡\{n1​g¯1​\(h1,z\)\+n2​g¯2​\(h2,z\)\}\.G\_\{z\}\(H\)=\\max\_\{\\begin\{subarray\}\{c\}0\\leq h\_\{1\},h\_\{2\}\\leq L\\\\ n\_\{1\}h\_\{1\}\+n\_\{2\}h\_\{2\}=H\\end\{subarray\}\}\\\{n\_\{1\}\\bar\{g\}\_\{1\}\(h\_\{1\},z\)\+n\_\{2\}\\bar\{g\}\_\{2\}\(h\_\{2\},z\)\\\}\.\(68\)When a group is empty its term and variable are omitted\. This relaxes unknown integer task compositions, so it is conservative\. Because each majorant is piecewise\-linear concave, multiply segment lengths by group size, merge the two groups’ segment slopes in descending order, and consume total resourceHH\. This solves Equation \([68](https://arxiv.org/html/2609.29140#A12.E68)\); decreasing slopes ensure that any segment’s predecessors in its group are consumed first\.

For each candidateθ\\theta, exponential Markov gives the lower\-tail boundℙ⁡\(E2≤e∣S\)≤exp⁡\{Gz​\(M​L​θ\)−z​e\}\\mathbb\{P\}\(E\_\{2\}\\leq e\\mid S\)\\leq\\exp\\\{G\_\{z\}\(ML\\theta\)\-ze\\\}\. We minimize over the fixed gridzj=−2−10\+18​j/255z\_\{j\}=\-2^\{\-10\+18j/255\},j=0,…,255j=0,\\ldots,255\. For any fixed candidate target, the corresponding rejection events are nested lower tails in the scalar observationE2E\_\{2\}, so minimizing these bounds does not require a union penalty over tilts\. Invert theα/2\\alpha/2tail bound for the upper endpoint and use complementary labels for the lower endpoint\. The group bound is independent of which tasks compriseSS, proving coverage conditionally on every selected subset, hence unconditionally\. The numerical implementation uses 46 bisection steps\. Att=Mt=Mit reduces to the earlier equal\-two\-count Hull; att=0t=0it gives its one\-draw counterpart\.

### L\.2Results and implementation checks

Every budget retains all 24 panel/event conditions\. The full tables below report the original Audit comparisons; Table[19](https://arxiv.org/html/2609.29140#A14.T19)instead shows the later Joint refinement from Appendix[N](https://arxiv.org/html/2609.29140#A14)\. Original Audit never beats fixed\-uniform width in the 144 cells; neither does the valid Audit/Hull intersection\. The latter beats nominal\-95% Hull in0,0,5,10,12,00,0,5,10,12,0cells across the six budgets, illustrating the cost of its error split\. Audit improves on same\-label Hull most often at intermediate budgets, not at either endpoint\. Its generic variance upper bound deliberately retains the one\-draw variance envelope, whereas the dedicated Pair and full\-replication bounds exploit the smaller two\-draw variance\. The asymptotic theorem does not optimize these constants\.

Partial replication’s empirical MSE is lower than fixed uniform in all 144 cells, with every difference’s Monte Carlo interval excluding zero\. It is lower than balanced adaptive in only0,2,1,2,1,00,2,1,2,1,0cells; none of those six apparent wins excludes zero under its replay\-bootstrap interval\. Median realized budget utilization is approximately 0\.750 at every budget\. Minimum Audit and Hull empirical coverage is 1\.000, versus 0\.996 for uniform adaptive and 0\.930 for fixed uniform over this grid\. A selected minimum across 144 finite Monte Carlo cells is not a coverage guarantee or evidence against the exact hypergeometric proof; all cellwise values and Monte Carlo uncertainty are retained\. The rate theorem is established by its proof, not by high coverage in these heterogeneous, previously seen panels\.

The analyzer reconstructs 444,000 partial interval pairs, reaggregates all 288,000 rows, and freshly executes 1,152 selected policy randomizations\. Summary discrepancies and budget violations are zero\. Separate synthetic checks enumerate 6,502 Audit coverage cases and 3,251 Hull cases at smallM,L,tM,L,t; check the exact MSE identity; and test 2,292,736 conditional MGF inequalities for the Hull relaxation\. Maximum floating\-point log\-MGF excess in those checks is4\.6×10−134\.6\\times 10^\{\-13\}, and full\-replication endpoint agreement with the previous Hull implementation is within2\.8×10−142\.8\\times 10^\{\-14\}\. Finite enumerations and floating\-point tolerances are numerical checks, not substitutes for the analytical guarantees\.

The complete machine\-readable outputs include bias, MSE, width, coverage, realized charge and every interval variant in each panel/event/budget cell\. Two thousand bootstrap samples use seed 20261006, pairing metrics within a policy and independently resampling independently randomized policies\. These intervals quantify replay Monte Carlo uncertainty conditional on the fixed banks, not new\-task, new\-model or deployment uncertainty\. Tables[14](https://arxiv.org/html/2609.29140#A12.T14)–[15](https://arxiv.org/html/2609.29140#A12.T15)show every Audit width ratio against the two principal fixed\-time comparators\.

Table 14:Audit / same\-label mixed\-count Hull mean\-width ratios, all configurations and both events\. S: first success; F: first failure\. All denominators are positive; below one favors Audit\.Table 15:Audit / fixed uniform hypergeometric mean\-width ratios, all configurations and both events\. S: first success; F: first failure\. All denominators are positive; below one favors Audit\.

## Appendix MA declared finite\-size and near\-purity study

Figure 4:A finite regime for auditing, and its limits:L=5L=5, 95% intervals,t=⌈ρ​M⌉t=\\lceil\\rho M\\rceil, equal hard cost\(M\+t\)​K\(M\+t\)K\. Ratios compare expected Audit width with exact fixed\-uniform width; below one favors Audit\. Left: all declared pure\-cohort sizes\. Right:M=512M=512, with all budgets and both mixing families\. Curves use numerical integration of exact finite laws, not new model outcomes\.#### Scope and chronology\.

After the complete partial\-agent analysis, a private review asked where the pure\-cohort rate advantage becomes useful at finiteMMand whether any advantage survives modest within\-task variation\. We declared the following synthetic grid before computing it\. It is a controlled fixed\-cohort analysis, not a reserved public\-data test, new model generations, or a study of how often these regimes arise\. A later, separately recorded deterministic integration checks all width comparisons on the unchanged grid\. No constants, error splits, interval mixtures or grid points were tuned to its outcomes\.

#### Every cohort and budget\.

SetL=5L=5,α=\.05\\alpha=\.05,M∈\{128,512,2048,8192,32768\}M\\in\\\{128,512,2048,8192,32768\\\},t=⌈ρ​M⌉t=\\lceil\\rho M\\rceilforρ∈\{\.1,\.25,\.5,1\}\\rho\\in\\\{\.1,\.25,\.5,1\\\}, and requestedη∈\{0,\.002,\.01,\.02,\.05,\.1,\.2\}\\eta\\in\\\{0,\.002,\.01,\.02,\.05,\.1,\.2\\\}\. The actual even mixed\-task count ism=2​⌊η​M/2⌋m=2\\lfloor\\eta M/2\\rfloor\. For eachh∈\{1,2\}h\\in\\\{1,2\\\}, assign\(M−m\)/2\(M\-m\)/2tasks each to positive counts zero and five, andm/2m/2tasks each to positive countshhand5−h5\-h\. Thusθ=1/2\\theta=1/2exactly\. At the same mixed fraction,h=2h=2gives more within\-task variation thanh=1h=1\. The late\-event embedding charges exactlyM\+tM\+tlabels, or\(M\+t\)​K\(M\+t\)Kunits, to both partial replication and fixed uniform\. There is no cost recycling\.

Pure duplicates share one canonical cell\. Requestedη=\.002,\.01\\eta=\.002,\.01both round to zero atM=128M=128, andη=\.002\\eta=\.002rounds to zero atM=512M=512\. The full 280\-request map therefore gives 236 partial cells and 20 distinct fixed\-uniform references\. We retain the map, not 280 ostensibly independent experiments\. Each canonical cell has 10,000 evaluator randomizations, with SHA\-256\-derived streams from seed 20261007: 2\.36 million partial samples and 200,000 fixed\-uniform samples\. The pre\-run manifest hashes the protocol, runner and ten inference/sampling modules\.

#### Exact grouped sampling, not an iid approximation\.

Let the four class sizes becjc\_\{j\}and their positive\-path counts behj∈\{0,5,h,5−h\}h\_\{j\}\\in\\\{0,5,h,5\-h\\\}\. The audited class counts have the multivariate hypergeometric law from choosingttof theMMtasks\. Conditional on them, unaudited positive counts are independentBin⁡\(cj−aj,hj/5\)\\operatorname\{Bin\}\(c\_\{j\}\-a\_\{j\},h\_\{j\}/5\); audited pair counts are independent multinomials of sizeaja\_\{j\}with probabilities

\(\(5−hj\)​\(4−hj\)20,2​hj​\(5−hj\)20,hj​\(hj−1\)20\)\.\\left\(\\frac\{\(5\-h\_\{j\}\)\(4\-h\_\{j\}\)\}\{20\},\\frac\{2h\_\{j\}\(5\-h\_\{j\}\)\}\{20\},\\frac\{h\_\{j\}\(h\_\{j\}\-1\)\}\{20\}\\right\)\.\(69\)These are the exact distributions of aggregates over fixed, independent within\-task random draws\. IfUUandSScount unaudited and audited purchased positives, thenE2=2​U\+SE\_\{2\}=2U\+S,P=U\+SP=U\+S, andddcounts pairs with one positive\. The joint sampler retains dependence among all three sufficient statistics\. For fixed uniform, the purchased\-positive count has lawHypergeom⁡\(5​M,5​M/2,M\+t\)\\operatorname\{Hypergeom\}\(5M,5M/2,M\+t\), the same for everyη,h\\eta,hat a givenM,tM,t\. Sharing one reference stream at that budget is therefore appropriate and induces dependence across reported comparisons\.

#### Intervals and uncertainty\.

Every cell reports the frozen generic Audit formula, the old fixed\-count baseline, and Pair att=Mt=M, all with purchased\-label clipping\. Exact\-law mixed\-count Hull is not recomputed in this synthetic grid; its optional omission was recorded before outcomes and does not imply Audit is best on the same labels\. No unadjusted minimum of nominal intervals is treated as valid\. Two thousand empirical bootstrap resamples use seed 20261008, pairing measurements within a policy and separately resampling independent policies\. Cellwise coverage intervals use binomial inversion\. These quantify simulation error, not new\-task uncertainty or a simultaneous guarantee over 236 cells\.

#### Deterministic checks and expected width\.

Withq=2​h​\(5−h\)/20q=2h\(5\-h\)/20andQQthe number of audited mixed tasks,

Q\\displaystyle Q∼Hypergeom⁡\(M,m,t\),\\displaystyle\\sim\\operatorname\{Hypergeom\}\(M,m,t\),d\|Q\\displaystyle d\\mid Q∼Bin⁡\(Q,q\),\\displaystyle\\sim\\operatorname\{Bin\}\(Q,q\),\(70\)MSEpartial\\displaystyle\\operatorname\{MSE\}\_\{\\rm partial\}=m​h​\(5−h\)25​M2​\(1−5​t8​M\),\\displaystyle=\\frac\{mh\(5\-h\)\}\{25M^\{2\}\}\\left\(1\-\\frac\{5t\}\{8M\}\\right\),MSEUF\\displaystyle\\operatorname\{MSE\}\_\{\\rm UF\}=5​M−\(M\+t\)4​\(M\+t\)​\(5​M−1\)\.\\displaystyle=\\frac\{5M\-\(M\+t\)\}\{4\(M\+t\)\(5M\-1\)\}\.\(71\)The variance check usesVar⁡\(d\)=q⁡\(1−q\)​t​m/M\+q2​t​\(m/M\)​\(1−m/M\)​\(M−t\)/\(M−1\)\\operatorname\{Var\}\(d\)=q\(1\-q\)tm/M\+q^\{2\}t\(m/M\)\(1\-m/M\)\(M\-t\)/\(M\-1\)\. The marginal law fordddoes not assert independence from the estimate\.

In this particular symmetric grid, clipping is inactive for*every*possible sample, not only the simulated ones\. Each pure\-one task contributes at least3/\(5​M\)3/\(5M\)toθ^−P/\(5​M\)\\widehat\{\\theta\}\-P/\(5M\); other contributions are nonnegative\. The pure\-zero group gives the analogous upper gap\. Both gaps are at least3​\(M−m\)/\(10​M\)≥\.243\(M\-m\)/\(10M\)\\geq\.24\. By monotonicity indd, the maximum possible Audit radius isr⁡\(min⁡\{t,m\}\)r\(\\min\\\{t,m\\\}\); its largest value over the grid is below \.141\. Consequently width equals2​r​\(d\)2r\(d\)exactly\. The post\-run analysis evaluates its expectation by summing the binomial conditional law over the hypergeometricQQlaw\. The omitted contribution is bounded by omitted probability, at most3\.7×10−143\.7\\times 10^\{\-14\}in this calculation\. Fixed\-uniform expected widths sum their exact count law; omitted probability is below2×10−132\\times 10^\{\-13\}\. These bounds control truncation, not floating\-point roundoff: the results are numerical integrations of exact laws, not certified interval\-arithmetic enclosures\. The complete simulation is separately reaggregated; all 2\.36 million Audit intervals are reconstructed and summary discrepancies are zero\.

#### Results, including the limits\.

Tables[16](https://arxiv.org/html/2609.29140#A13.T16)–[17](https://arxiv.org/html/2609.29140#A13.T17)give every requested Audit comparison\. Exact\-law expectations confirm 185 wins and 51 losses, with no crossing changed from the empirical comparison\. The 10% design’s pure advantage first appears between the tested sizes 128 and 512; the other three audit fractions already win at the smallest tested size\. No finer or universal crossover threshold is claimed\. AtM=128,t=32M=128,t=32, only two mixed tasks remove the pure gain: ratios 1\.0159 and 1\.0417 forh=1,2h=1,2\. AtM=512M=512, all designs still win withm=24m=24, but not all withm=50m=50\. The full\-audith=1,m=50h=1,m=50ratio is only 0\.9986, a negligible practical difference\. AtM=8192M=8192and 32768, the 10% and 25% designs win in both families throughout the tested grid, including about 20% mixing\. These are genuinely nonzero\-variation examples, not a near\-pure minimax theorem\.

Increasing the audit fraction need not improve the ratio\. AtM=32768M=32768,h=2h=2and about 20% mixing, it is0\.8460,0\.8977,1\.0064,1\.24420\.8460,0\.8977,1\.0064,1\.2442across the four budgets\. Fixed uniform’s budget also grows, and generic Audit does not exploit the full\-pair variance reduction\. Pair is narrower than Audit in all 59 full\-audit cells and beats fixed uniform in 57, losing the two largest\-mixingh=2h=2cases atM=128,512M=128,512\(Table[18](https://arxiv.org/html/2609.29140#A13.T18)\)\. The old fixed\-count baseline loses to fixed uniform in all 236 cells\. All intervals and uncertainty summaries, including these losses, remain in the machine\-readable artifact\.

The exact partial MSE is below fixed uniform’s throughout, with ratio zero on pure cohorts and at most 0\.283 over this grid\. Audit empirical coverage ranges from \.9987 to one, Pair from \.9976 to one, and fixed uniform from \.9470 to \.9665; each below\-\.95 fixed\-uniform cell’s marginal Monte Carlo interval includes \.95\. These finite simulations neither prove honesty nor contradict the exact hypergeometric guarantee\. The study fixes equalL=5L=5, balanced prevalence, symmetric mixtures and late\-event costs; it does not establish generalization to other prevalences, real cost distributions, or future agent outputs\. It does not overturn the complete public\-bank width comparisons\.

Table 16:Complete synthetic expected\-width ratios: Audit / fixed uniform,h=1h=1\. Values below one favor Audit\. Columns give requestedη\\eta; the actual mixed count ism=2​⌊η​M/2⌋m=2\\lfloor\\eta M/2\\rfloor\. Repeated pure cells are displayed for completeness, not counted as new evidence\. Both expectations use exact finite laws, numerically summed as described in the text\.Table 17:Complete synthetic expected\-width ratios: Audit / fixed uniform,h=2h=2\. Values below one favor Audit\. Columns give requestedη\\eta; the actual mixed count ism=2​⌊η​M/2⌋m=2\\lfloor\\eta M/2\\rfloor\. Repeated pure cells are displayed for completeness, not counted as new evidence\. Both expectations use exact finite laws, numerically summed as described in the text\.Table 18:Dedicated full\-pair interval / fixed\-uniform mean\-width ratios att=Mt=M\. These are simulation means from 10,000 randomizations per canonical cell, not the deterministic Audit calculation in Tables[16](https://arxiv.org/html/2609.29140#A13.T16)–[17](https://arxiv.org/html/2609.29140#A13.T17)\. All conditions and both losses are retained\. Monte Carlo intervals are in the artifact\.

## Appendix NA joint mean/disagreement refinement

Equation \([6](https://arxiv.org/html/2609.29140#S4.E6)\) establishes the upper rate in Theorem[3](https://arxiv.org/html/2609.29140#Thmtheorem3), but spends separate error probabilities on variance and mean control\. The following post\-review refinement uses a joint exponential moment\. It does not change the minimax claim, sampling design, point estimate or budget, and is not asserted for outcome\-adaptive audit subsets\. The construction uses classical exponential and symmetric\-polynomial inequalities, rather than claiming a new general concentration principle\.

### N\.1An unconditional fixed\-subset moment bound

Fix0<t≤M0<t\\leq Mandρ=t/M\\rho=t/M\. The random subset, distinct within\-task draws,AiA\_\{i\}and disagreement countddare as in Theorem[3](https://arxiv.org/html/2609.29140#Thmtheorem3)\. For a task withh∈\{1,…,L−1\}h\\in\\\{1,\\ldots,L\-1\\\}positive paths, writep=h/Lp=h/Land

q0,h\\displaystyle q\_\{0,h\}=\(L−h\)​\(L−h−1\)L⁡\(L−1\),\\displaystyle=\\frac\{\(L\-h\)\(L\-h\-1\)\}\{L\(L\-1\)\},q1,h\\displaystyle q\_\{1,h\}=2​h​\(L−h\)L⁡\(L−1\),\\displaystyle=\\frac\{2h\(L\-h\)\}\{L\(L\-1\)\},q2,h\\displaystyle q\_\{2,h\}=h⁡\(h−1\)L⁡\(L−1\),\\displaystyle=\\frac\{h\(h\-1\)\}\{L\(L\-1\)\},\(72\)bh​\(u\)\\displaystyle b\_\{h\}\(u\)=\(1−p\)​e−u​p\+p​eu⁡\(1−p\),\\displaystyle=\(1\-p\)e^\{\-up\}\+pe^\{u\(1\-p\)\},a0,h​\(u\)\\displaystyle a\_\{0,h\}\(u\)=q0,h​e−u​p\+q2,h​eu⁡\(1−p\),\\displaystyle=q\_\{0,h\}e^\{\-up\}\+q\_\{2,h\}e^\{u\(1\-p\)\},a1,h​\(u\)\\displaystyle a\_\{1,h\}\(u\)=q1,h​eu⁡\(1/2−p\)\.\\displaystyle=q\_\{1,h\}e^\{u\(1/2\-p\)\}\.Herebhb\_\{h\}is the centered one\-draw MGF;a0,ha\_\{0,h\}anda1,ha\_\{1,h\}partition the centered pair\-mean MGF by agreement and disagreement\. Define

Ch​\(u\)=bh​\(u\)​\{1−ρ−1​log⁡bh​\(u\)\}−a0,h​\(u\)\.C\_\{h\}\(u\)=b\_\{h\}\(u\)\\\{1\-\\rho^\{\-1\}\\log b\_\{h\}\(u\)\\\}\-a\_\{0,h\}\(u\)\.\(73\)Wheneveru\>0u\>0and every mixed\-taskCh​\(u\)\>0C\_\{h\}\(u\)\>0, put

ψρ,L​\(u\)=max⁡\{0,max1≤h<L⁡log⁡a1,h​\(u\)Ch​\(u\)\}\.\\psi\_\{\\rho,L\}\(u\)=\\max\\left\\\{0,\\max\_\{1\\leq h<L\}\\log\\frac\{a\_\{1,h\}\(u\)\}\{C\_\{h\}\(u\)\}\\right\\\}\.\(74\)The admissible set contains an open neighborhood to the right of zero, sinceCh​\(0\)=q1,h\>0C\_\{h\}\(0\)=q\_\{1,h\}\>0\. We need neither a global monotonicity claim nor a unique positive root for its boundary\.

###### Proposition 9\(Joint fixed\-subset certificate\)\.

For any fixed cohort and any deterministic admissibleuu,

𝔼R​exp⁡\{±u​M​\(θ^−θ𝒞\)−ψρ,L​\(u\)​d\}≤1\.\\mathbb\{E\}\_\{R\}\\exp\\\{\\pm uM\(\\widehat\{\\theta\}\-\\theta\_\{\\mathcal\{C\}\}\)\-\\psi\_\{\\rho,L\}\(u\)d\\\}\\leq 1\.\(75\)Consequently, letuju\_\{j\}be finitely many admissible, data\-independent tilts,wj\>0w\_\{j\}\>0, andw∗≥0w\_\{\*\}\\geq 0, withw∗\+∑jwj=1w\_\{\*\}\+\\sum\_\{j\}w\_\{j\}=1and at least one tilt\. For0<α<10<\\alpha<1, the uniquex⁡\(d\)\>0x\(d\)\>0solving

w∗\+∑jwj​exp⁡\{uj​x​\(d\)−ψρ,L​\(uj\)​d\}=2/αw\_\{\*\}\+\\sum\_\{j\}w\_\{j\}\\exp\\\{u\_\{j\}x\(d\)\-\\psi\_\{\\rho,L\}\(u\_\{j\}\)d\\\}=2/\\alpha\(76\)gives an honest intervalθ^±x⁡\(d\)/M\\widehat\{\\theta\}\\pm x\(d\)/M\. Purchased\-label feasible\-set clipping preserves this coverage\.

###### Proof\.

Consider the positive sign\. Setah=a0,h\+e−ψρ,L​\(u\)​a1,ha\_\{h\}=a\_\{0,h\}\+e^\{\-\\psi\_\{\\rho,L\}\(u\)\}a\_\{1,h\}\. Equations \([73](https://arxiv.org/html/2609.29140#A14.E73)\)–\([74](https://arxiv.org/html/2609.29140#A14.E74)\) implyah/bh≤1−ρ−1​log⁡bha\_\{h\}/b\_\{h\}\\leq 1\-\\rho^\{\-1\}\\log b\_\{h\}\. For pure tasks putah=bh=1a\_\{h\}=b\_\{h\}=1\. Conditional on the uniform subsetSS, task sampling is independent\. Averaging the product over alltt\-subsets therefore gives the exact MGF

\(∏i=1Mbhi\)​et​\(ah1/bh1,…,ahM/bhM\)\(Mt\),\\left\(\\prod\_\{i=1\}^\{M\}b\_\{h\_\{i\}\}\\right\)\\frac\{e\_\{t\}\(a\_\{h\_\{1\}\}/b\_\{h\_\{1\}\},\\ldots,a\_\{h\_\{M\}\}/b\_\{h\_\{M\}\}\)\}\{\\binom\{M\}\{t\}\},\(77\)whereete\_\{t\}is the elementary symmetric polynomial\. Maclaurin’s inequality bounds the normalized polynomial by thettth power of the arithmetic mean of its nonnegative arguments\. WithB0=∑ilog⁡bhi≥0B\_\{0\}=\\sum\_\{i\}\\log b\_\{h\_\{i\}\}\\geq 0, this is at most\(1−B0/t\)t\(1\-B\_\{0\}/t\)^\{t\}\. Admissibility implieslog⁡bh<ρ\\log b\_\{h\}<\\rhofor every mixed task, henceB0<tB\_\{0\}<t\. It follows that the MGF is at mosteB0​\(1−B0/t\)t≤1e^\{B\_\{0\}\}\(1\-B\_\{0\}/t\)^\{t\}\\leq 1\. Complementing every binary label mapshhtoL−hL\-h, leavesddunchanged, and proves the negative sign with the same penalty\.

Each fixed mixture has expectation at most one\. For an error exceedingx⁡\(d\)x\(d\)in either direction its mixture exceeds2/α2/\\alpha; two exponential Markov bounds give total noncoverage at mostα\\alpha\. This argument does not assume thatθ^\\widehat\{\\theta\}andddare independent\. Since every penalty is nonnegative, the left side of Equation \([76](https://arxiv.org/html/2609.29140#A14.E76)\) is at most one at zero and is strictly increasing to infinity\. The inversion is therefore well defined\. The true target always belongs to the purchased\-label feasible interval, which proves the clipping assertion\. ∎

For fixedρ,L\\rho,L, expansion of each mixed\-task logarithmic ratio gives

log⁡\{a1,h​\(u\)/Ch​\(u\)\}=\{L−14​L​ρ−18\}​u2\+O⁡\(u3\)\.\\log\\\{a\_\{1,h\}\(u\)/C\_\{h\}\(u\)\\\}=\\left\\\{\\frac\{L\-1\}\{4L\\rho\}\-\\frac\{1\}\{8\}\\right\\\}u^\{2\}\+O\(u^\{3\}\)\.\(78\)The quadratic coefficient does not depend onhh\. It reflects the exact design variance factor1−ρ​L/\[2​\(L−1\)\]1\-\\rho L/\[2\(L\-1\)\], unlike the generic Audit variance envelope\. This local expansion is not a finite\-sample dominance or optimal\-constants theorem\.

### N\.2Declared implementation and complete exploratory comparison

The tested rule, called*Joint*, uses the original Hoeffding interval att=0t=0and the existing Pair interval att=Mt=M\. For0<t<M0<t<M, test the fixed gridu=ρ​2k/32u=\\sqrt\{\\rho\}\\,2^\{k/32\},k=−256,…,192k=\-256,\\ldots,192, retaining only values with everyCh​\(u\)\>10−10C\_\{h\}\(u\)\>10^\{\-10\}in the implementation\. Select its largest retained valueu0u\_\{0\}\. ForJ=⌈log2⁡M⌉\+1J=\\lceil\\log\_\{2\}M\\rceil\+1, testuj=u0/2ju\_\{j\}=u\_\{0\}/2^\{j\},j=0,…,J−1j=0,\\ldots,J\-1, again checking admissibility\. Assignwj=1/\[\(j\+1\)​\(j\+2\)\]w\_\{j\}=1/\[\(j\+1\)\(j\+2\)\]andw∗=1/\(J\+1\)w\_\{\*\}=1/\(J\+1\); move any discarded tilt’s weight tow∗w\_\{\*\}\. If the grid has no admissible tilt, use the original Audit certificate as a design\-dependent fallback\. Neither the fallback nor any tilt, weight or endpoint choice uses observed labels\. No fallback occurs in the reported grid\. This rule is not an unadjusted minimum of nominal intervals\.

The implementation evaluateslog⁡bh\\log b\_\{h\}using the identity

log⁡bh​\(u\)=log⁡\{1\+\(1−p\)​R​\(−u​p\)\+p​R​\(u⁡\(1−p\)\)\},R⁡\(z\)=ez−1−z\.\\log b\_\{h\}\(u\)=\\log\\\{1\+\(1\-p\)R\(\-up\)\+pR\(u\(1\-p\)\)\\\},\\qquad R\(z\)=e^\{z\}\-1\-z\.\(79\)For\|z\|<\.01\|z\|<\.01, a degree\-12 series evaluatesRRwithout losing its quadratic term; otherwise it usesexpm1\. Directly roundingbhb\_\{h\}to one before taking its logarithm is unsafe for very smallρ\\rho\. A10−1210^\{\-12\}outward penalty guard and ordinary numerical inversion are used\. These are double\-precision calculations, not formal interval\-arithmetic enclosures\. The supplement retains the numerical correction and its pre\-empirical\-computation chronology separately\.

After the earlier complete study and its review, the formula, full comparison grid and implementation were fixed before examining the new widths\. All 72,000 existing partial\-design realizations are reused: twelve already\-seen agent configurations, both events, all six budgets, and 500 repeats\. This adds no model observations and changes no purchase, point estimate or cost\. All old comparisons remain in Appendix[L](https://arxiv.org/html/2609.29140#A12)\.

Joint is narrower than generic Audit in all 120 positive\-audit conditions, with 24 exactt=0t=0ties\. Its median Joint/Audit ratios by budget are1\.000,0\.863,0\.837,0\.796,0\.788,0\.6791\.000,0\.863,0\.837,0\.796,0\.788,0\.679\. Against same\-label Hull it wins0,14,24,24,24,240,14,24,24,24,24of the 24 conditions at the respective budgets; Table[19](https://arxiv.org/html/2609.29140#A14.T19)reports those ratios and the unchanged point\-error comparisons\. Against exact fixed uniform it wins only0,0,0,3,2,10,0,0,3,2,1: six wins and 138 losses overall\. Only five wins have a marginal Monte Carlo 95% difference interval excluding zero; all 138 losses do\. The full\-audit endpoint simply recovers the previously reported Pair result, not a new method gain\. These comparisons show removable conservativeness, not broad certification superiority or the best attainable inference for either design\.

The minimum observed Joint coverage is 0\.994\. The artifact provides every cell’s binomial Wilson coverage interval and all width comparisons, including adverse and zero\-reference cases\. Two thousand bootstrap resamples use seed 20261009, pairing metrics within each policy and independently resampling independently randomized policies\. They quantify marginal replay Monte Carlo uncertainty conditional on fixed banks, not simultaneous testing or future model/task generalization\. Point MSE, budget utilization and the original uniform reference coverage observations are unchanged\.

Numerical checks enumerate 13,004 coverage cases over all small cohorts withM≤4M\\leq 4,3≤L≤63\\leq L\\leq 6, every audit count, and four error levels\. They include 97,948 randomization leaves and 10,574 signed MGF checks; the largest MGF is one\. A broader deterministic grid checks 23,182 local inequalities with no positive excess and 99 unchanged endpoint comparisons\. These finite tests support implementation checking; the proposition, not the enumeration or observed high coverage, establishes mathematical honesty\.

Table 19:All 24 agent panel/event conditions at every audit budget,K=2,L=5K=2,L=5\. Median ratios; parentheses count strict wins out of 24\. Width uses the post\-review Joint refinement versus same\-label Hull or fixed uniform \(UF\); point MSE is unchanged, versus UF or balanced adaptive \(BA\)\. Joint equals the original Hoeffding rule att=0t=0and Pair att=Mt=M\. All denominators are positive\.

## Appendix OFinite\-cohort dominance region and selector

This section turns the Joint boundary into a finite expected\-width comparison\. It is a comparison theorem for two fixed designs, not a new confidence interval and not a claim that the cohort descriptors are known in advance\.

For a fixed cohort, lethi=∑r=1LYi​rh\_\{i\}=\\sum\_\{r=1\}^\{L\}Y\_\{ir\},H=∑ihiH=\\sum\_\{i\}h\_\{i\}, and

q¯=1M​∑i=1M2​hi​\(L−hi\)L⁡\(L−1\)\.\\bar\{q\}=\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\frac\{2h\_\{i\}\(L\-h\_\{i\}\)\}\{L\(L\-1\)\}\.\(80\)Thusq¯\\bar\{q\}is the mean probability that two distinct uniform paths in a task disagree\. Its largest feasible value isqmax,L=max0≤h≤L⁡2​h​\(L−h\)/\{L⁡\(L−1\)\}q\_\{\\max,L\}=\\max\_\{0\\leq h\\leq L\}2h\(L\-h\)/\\\{L\(L\-1\)\\\}\.

Fix0<t<M0<t<Mand the data\-independent Joint mixture in Equation \([76](https://arxiv.org/html/2609.29140#A14.E76)\)\. Extend its boundaryxt​\(d\)x\_\{t\}\(d\)to every reald∈\[0,t\]d\\in\[0,t\]by the same equation\. For pooled uniform sampling, putN=M​LN=ML,n=M\+tn=M\+t, and let\[as,bs\]\[a\_\{s\},b\_\{s\}\]be the integer endpoints returned by the exact equal\-tailed hypergeometric inversion aftersspositives\. Its exact expected interval width on a cohort with totalHHis

WU​\(H\)=∑s=max⁡\{0,n−\(N−H\)\}min⁡\{n,H\}\(Hs\)​\(N−Hn−s\)\(Nn\)​bs−asN\.W\_\{U\}\(H\)=\\sum\_\{s=\\max\\\{0,n\-\(N\-H\)\\\}\}^\{\\min\\\{n,H\\\}\}\\frac\{\\binom\{H\}\{s\}\\binom\{N\-H\}\{n\-s\}\}\{\\binom\{N\}\{n\}\}\\,\\frac\{b\_\{s\}\-a\_\{s\}\}\{N\}\.\(81\)Define the finite Joint region and its boundary by

ℛJ\\displaystyle\\mathcal\{R\}\_\{J\}=\{\(H,q\):0≤H≤N,0≤q≤qmax,L,2​xt​\(t​q\)M≤WU\(H\)\},\\displaystyle=\\left\\\{\(H,q\):0\\leq H\\leq N,\\ 0\\leq q\\leq q\_\{\\max,L\},\\ \\frac\{2x\_\{t\}\(tq\)\}\{M\}\\leq W\_\{U\}\(H\)\\right\\\},\(82\)q⋆​\(H\)\\displaystyle q\_\{\\star\}\(H\)=sup\{q:\(H,q\)∈ℛJ\},\\displaystyle=\\sup\\\{q:\(H,q\)\\in\\mathcal\{R\}\_\{J\}\\\},\(83\)withq⋆​\(H\)=−∞q\_\{\\star\}\(H\)=\-\\inftyif the section is empty\.

###### Proof of Proposition[5](https://arxiv.org/html/2609.29140#Thmtheorem5)\.

Let

G⁡\(x,d\)=log⁡\(w∗\+∑jwj​exp⁡\{uj​x−ψρ,L​\(uj\)​d\}\)\.G\(x,d\)=\\log\\\!\\left\(w\_\{\*\}\+\\sum\_\{j\}w\_\{j\}\\exp\\\{u\_\{j\}x\-\\psi\_\{\\rho,L\}\(u\_\{j\}\)d\\\}\\right\)\.This is a log\-sum\-exp of affine functions, hence is jointly convex in\(x,d\)\(x,d\)\. It is strictly increasing inxx, while every penalty is nonnegative\. The sublevel setG⁡\(x,d\)≤log⁡\(2/α\)G\(x,d\)\\leq\\log\(2/\\alpha\)is therefore exactly the hypographx≤xt​\(d\)x\\leq x\_\{t\}\(d\)\. A function has a convex hypograph precisely when it is concave, soxtx\_\{t\}is concave; nonnegative penalties also make it nondecreasing\.

LetDiD\_\{i\}be the disagreement indicator when taskiiis selected for audit\. The audited set is uniform of sizett, and the two paths are distinct and uniform\. Consequently

𝔼R​d=∑iℙR​\(i∈S\)​𝔼R​\(Di∣i∈S\)=tM​∑i2​hi​\(L−hi\)L⁡\(L−1\)=t​q¯\.\\mathbb\{E\}\_\{R\}d=\\sum\_\{i\}\\mathbb\{P\}\_\{R\}\(i\\in S\)\\mathbb\{E\}\_\{R\}\(D\_\{i\}\\mid i\\in S\)=\\frac\{t\}\{M\}\\sum\_\{i\}\\frac\{2h\_\{i\}\(L\-h\_\{i\}\)\}\{L\(L\-1\)\}=t\\bar\{q\}\.Feasible\-label clipping can only shorten Joint\. Jensen’s inequality now gives

𝔼R​\|IJ\|≤2M​𝔼R​xt​\(d\)≤2​xt​\(𝔼R​d\)M=2​xt​\(t​q¯\)M\.\\mathbb\{E\}\_\{R\}\|I\_\{J\}\|\\leq\\frac\{2\}\{M\}\\mathbb\{E\}\_\{R\}x\_\{t\}\(d\)\\leq\\frac\{2x\_\{t\}\(\\mathbb\{E\}\_\{R\}d\)\}\{M\}=\\frac\{2x\_\{t\}\(t\\bar\{q\}\)\}\{M\}\.Equation \([81](https://arxiv.org/html/2609.29140#A15.E81)\) is the expectation of the exact uniform interval underS∼Hypergeom⁡\(N,H,n\)S\\sim\\operatorname\{Hypergeom\}\(N,H,n\)\. The definition ofq⋆​\(H\)q\_\{\\star\}\(H\)therefore proves the dominance implication\. ∎

#### What a valid selector requires\.

Suppose that, before accessing current\-cohort labels, external information specifies a descriptor set𝒜\\mathcal\{A\}known to contain\(H,q¯\)\(H,\\bar\{q\}\)\. If𝒜⊆ℛJ\\mathcal\{A\}\\subseteq\\mathcal\{R\}\_\{J\}, selecting Joint in advance guarantees no larger expected width than pooled uniform\. Otherwise this theorem recommends neither design\. Because the selection precedes current\-cohort outcome access and both candidate procedures are individually honest for every fixed cohort, the selected interval retains1−α1\-\\alphacoverage\. This is the selector corollary\. Plugging full\-cohortHHandq¯\\bar\{q\}into Equation \([83](https://arxiv.org/html/2609.29140#A15.E83)\) after inspection is instead an oracle diagnostic; it cannot be used to claim prospectively selected coverage or dominance\.

### O\.1Complete post\-hoc LiveCodeBench diagnostic

The analysis follows the declaration inresearch/finite\_regime\_boundary\_v1\.md\. It uses every eligible panel and every interior declared budget, recomputesWU​\(H\)W\_\{U\}\(H\)by exact hypergeometric expectation, and numerically inverts the fixed Joint mixture\. Table[20](https://arxiv.org/html/2609.29140#A15.T20)contains all budget\-level comparisons between the sufficient region and observed mean\-width wins\.

Table 20:Post\-hoc oracle diagnostic on all 16 LiveCodeBench panels\.CCcontains cells satisfying Proposition[5](https://arxiv.org/html/2609.29140#Thmtheorem5);WWcontains locked mean\-width wins\. The last three columns count their overlap, certified non\-wins, and uncertified wins\. OutsideCC, the proposition leaves the width comparison unresolved\.Across 112 panel–budget cells, 73 satisfy the sufficient condition and all 73 are locked mean\-width wins\. Five additional observed wins lie outside the sufficient region: DeepSeek\-V3 and Mistral\-Large att=29t=29, plus those two and GPT\-4\-Turbo\-2024\-04\-09 att=30t=30\. No certified cell is an observed non\-win\. Att=220t=220, the condition holds for the 15 observed wins and does not hold for the sole loss, DeepSeek\-R1\-Lite\-Preview\. Non\-certification leaves the comparison unresolved; the proposition does not imply a loss\.

For continuity with the original diagnostic, imposing the additional convention that non\-certification predicts a non\-win would give 107/112 agreements, with five false negatives and no false positives\. That converse is not a theorem, and this accounting is not predictive validation\. All descriptors use the inspected full cohort\. The diagnostic illustrates how the sufficient region relates to this grid, without supplying a selection rule from budgeted observations or a future\-model accuracy estimate\.

The analytical quantity bounds the true expected Joint length\. Locked widths are 1000\-randomization Monte Carlo means and can fluctuate around that expectation; the largest sample\-mean excess over the bound is9\.17×10−49\.17\\times 10^\{\-4\}at the smallest audit budget\. Machine\-readable output retains all 112 cells, including exact uniform expectations, thresholds, the original diagnostic classifications, and the five additional wins outside the region\.

## Appendix PCertification when tasks may be omitted

This section proves Theorem[4](https://arxiv.org/html/2609.29140#Thmtheorem4)\. It removes the task\-covering restriction from Appendix[K](https://arxiv.org/html/2609.29140#A11); the target, hard\-budget access model, and requirement of honesty over every fixed cohort are unchanged\. Policies have task identities and observed histories but no external information about the latent cohort\. The fixed upper construction uses simple random sampling twice\. The lower bound permits arbitrary adaptive task and path choices, interleaving, stopping, and omission\.

### P\.1A constructive omission\-enabled interval

Choose an integers∈\{0,…,⌊\(M−t\)/2⌋\}s\\in\\\{0,\\ldots,\\lfloor\(M\-t\)/2\\rfloor\\\}before observing labels, and set

n=M−s,q=t\+s≤n\.n=M\-s,\\qquad q=t\+s\\leq n\.Select a uniformnn\-task subsetHH\. In every selected task purchase one uniform path, and in a uniformqq\-task subset ofHHpurchase a second distinct path\. All allocations precede observation\. This reserves exactlyn\+q=M\+tn\+q=M\+tpaths, and hence costs at most\(M\+t\)​K\(M\+t\)Kfor every cohort\. Writepi=L−1​∑rYi​rp\_\{i\}=L^\{\-1\}\\sum\_\{r\}Y\_\{ir\}and letAiA\_\{i\}be the one\- or two\-label task mean\. Then

θ^H=1n​∑i∈HAi,𝔼⁡\(θ^H∣H\)=μH:=1n​∑i∈Hpi,\\widehat\{\\theta\}\_\{H\}=\\frac\{1\}\{n\}\\sum\_\{i\\in H\}A\_\{i\},\\qquad\\mathbb\{E\}\(\\widehat\{\\theta\}\_\{H\}\\mid H\)=\\mu\_\{H\}:=\\frac\{1\}\{n\}\\sum\_\{i\\in H\}p\_\{i\},so simple random task sampling gives𝔼​θ^H=θ𝒞\\mathbb\{E\}\\widehat\{\\theta\}\_\{H\}=\\theta\_\{\\mathcal\{C\}\}\.

Whens\>0s\>0, apply the Audit construction of Appendix[K](https://arxiv.org/html/2609.29140#A11)insideHHat errorα/2\\alpha/2, obtaining radiusrin​\(n,L,q,d,α/2\)r\_\{\\rm in\}\(n,L,q,d,\\alpha/2\)\. IfO=HcO=H^\{c\}andμO=s−1​∑i∈Opi\\mu\_\{O\}=s^\{\-1\}\\sum\_\{i\\in O\}p\_\{i\}, then

μH−θ𝒞=sn​\(θ𝒞−μO\)=sM​\(μH−μO\)\.\\mu\_\{H\}\-\\theta\_\{\\mathcal\{C\}\}=\\frac\{s\}\{n\}\(\\theta\_\{\\mathcal\{C\}\}\-\\mu\_\{O\}\)=\\frac\{s\}\{M\}\(\\mu\_\{H\}\-\\mu\_\{O\}\)\.\(84\)Sampling without replacement from the bounded task means and Equation \([84](https://arxiv.org/html/2609.29140#A16.E84)\) give the outer radius

rout=min⁡\{sM,s​log⁡\(4/α\)/2n\}\.r\_\{\\rm out\}=\\min\\left\\\{\\frac\{s\}\{M\},\\frac\{\\sqrt\{s\\log\(4/\\alpha\)/2\}\}\{n\}\\right\\\}\.\(85\)The first term is deterministic; the second fails with probability at mostα/2\\alpha/2by finite\-population Hoeffding\([Hoeffding, 1963](https://arxiv.org/html/2609.29140#bib.bib18);[Bardenet and Maillard, 2015](https://arxiv.org/html/2609.29140#bib.bib1)\)\. Conditional inner honesty holds for everyHH, so a union bound proves coverage of

\[θ^H−rin−rout,θ^H\+rin\+rout\]\.\[\\widehat\{\\theta\}\_\{H\}\-r\_\{\\rm in\}\-r\_\{\\rm out\},\\widehat\{\\theta\}\_\{H\}\+r\_\{\\rm in\}\+r\_\{\\rm out\}\]\.Intersecting this interval with the compatible\-label range

\[PobsM​L,1−M\+t−PobsM​L\]\\left\[\\frac\{P\_\{\\rm obs\}\}\{ML\},1\-\\frac\{M\+t\-P\_\{\\rm obs\}\}\{ML\}\\right\]preserves coverage\. Fors=0s=0use Audit directly at errorα\\alpha\.

On an internally constant cohortd=0d=0surely, and the untruncated width is

Oα,L​\{1n⁡\(q\+1\)\+1n\+sn\}\.O\_\{\\alpha,L\}\\left\\\{\\frac\{1\}\{\\sqrt\{n\(q\+1\)\}\}\+\\frac\{1\}\{n\}\+\\frac\{\\sqrt\{s\}\}\{n\}\\right\\\}\.\(86\)ForM≥16M\\geq 16andt2<Mt^\{2\}<M, takes=⌊M⌋s=\\lfloor\\sqrt\{M\}\\rfloor; this is feasible,n≥M/2n\\geq M/2, andq\+1≥Mq\+1\\geq\\sqrt\{M\}\. Equation \([86](https://arxiv.org/html/2609.29140#A16.E86)\) is thenOα,L\(M−3/4\)O\_\{\\alpha,L\}\(M^\{\-3/4\}\)\. Ift2≥Mt^\{2\}\\geq M, takes=0s=0and invoke Theorem[3](https://arxiv.org/html/2609.29140#Thmtheorem3)\. The two branches are uniformlyOα,L\(\[M\(t\+M\)\]−1/2\)O\_\{\\alpha,L\}\(\[M\(t\+\\sqrt\{M\}\)\]^\{\-1/2\}\)\. ForM<16M<16, the unit interval absorbs the finite cases\.

The exact MSE identity makes the price of omission explicit\. Define

σp2=1M​∑i\(pi−θ𝒞\)2,V¯=1M​∑ipi​\(1−pi\)\.\\sigma\_\{p\}^\{2\}=\\frac\{1\}\{M\}\\sum\_\{i\}\(p\_\{i\}\-\\theta\_\{\\mathcal\{C\}\}\)^\{2\},\\qquad\\overline\{V\}=\\frac\{1\}\{M\}\\sum\_\{i\}p\_\{i\}\(1\-p\_\{i\}\)\.ForM\>1M\>1, total variance and conditional unbiasedness give

𝔼​\(θ^H−θ𝒞\)2=s​σp2n⁡\(M−1\)\+V¯n​\(1−qn​L2​\(L−1\)\)\.\\mathbb\{E\}\(\\widehat\{\\theta\}\_\{H\}\-\\theta\_\{\\mathcal\{C\}\}\)^\{2\}=\\frac\{s\\sigma\_\{p\}^\{2\}\}\{n\(M\-1\)\}\+\\frac\{\\overline\{V\}\}\{n\}\\left\(1\-\\frac\{q\}\{n\}\\frac\{L\}\{2\(L\-1\)\}\\right\)\.\(87\)The first term is simple\-random\-sample task variance\. Averaging the partial\-audit variance withinHHgives the second; the covariance is zero\. WhenM=1M=1, feasibility forcess=0s=0and only the second term remains\. Thus omission creates point error on heterogeneous pure cohorts even though the within\-task term vanishes\.

### P\.2Lower bound over adaptive omission\-enabled policies

Use the late\-event subexperiment from Appendix[K](https://arxiv.org/html/2609.29140#A11): setTi​r=KT\_\{ir\}=Kfor label zero andTi​r=K\+1T\_\{ir\}=K\+1for label one\. Prefixes belowKKare label\-independent and every distinct terminal label costsKK\. Letrir\_\{i\}be the number of revealed distinct labels in taskiiand letz=\#⁡\{i:ri=0\}z=\\\#\\\{i:r\_\{i\}=0\\\}\. Every pathwise\-\(M\+t\)​K\(M\+t\)Kpolicy satisfies

∑iri≤M\+t,∑i:ri≥1\(ri−1\)≤t\+z\.\\sum\_\{i\}r\_\{i\}\\leq M\+t,\\qquad\\sum\_\{i:r\_\{i\}\\geq 1\}\(r\_\{i\}\-1\)\\leq t\+z\.\(88\)Include all policy and interval random seeds in the full transcriptTT\. Adaptive action factors then cancel when the same compatible transcript is compared under two cohort priors\.

Under prior A, task labels are independent fair bits and every task is pure\. ForM≥256M\\geq 256, puts0=⌈M⌉s\_\{0\}=\\lceil\\sqrt\{M\}\\rceilandF=\{z≤s0\}F=\\\{z\\leq s\_\{0\}\\\}\. IfℙA​\(Fc\)≥1/4\\mathbb\{P\}\_\{A\}\(F^\{c\}\)\\geq 1/4, then conditional on a pure\-compatible transcript thezzomitted bits remain independent fair bits\. An interval of length belowℓ=s0/\(4​M\)\\ell=\\sqrt\{s\_\{0\}\}/\(4M\)contains at mostz/2\\sqrt\{z\}/2possible values of their sum divided byMM\. The largestBin⁡\(z,1/2\)\\operatorname\{Bin\}\(z,1/2\)atom is at mostz−1/2z^\{\-1/2\}\. Consequently its posterior coverage is at most1/21/2onFcF^\{c\}\. Integrated honesty, rather than any pointwise posterior\-coverage assumption, yields

ℙA​\(Fc,\|I\|<ℓ\)≤2​α,𝔼A​\|I\|≥ℓ⁡\(1/4−2​α\)≥s048​M\.\\mathbb\{P\}\_\{A\}\(F^\{c\},\\ \|I\|<\\ell\)\\leq 2\\alpha,\\qquad\\mathbb\{E\}\_\{A\}\|I\|\\geq\\ell\(1/4\-2\\alpha\)\\geq\\frac\{\\sqrt\{s\_\{0\}\}\}\{48M\}\.Forα≤1/12\\alpha\\leq 1/12this lower bounds the target rate\.

It remains to considerℙA​\(F\)\>3/4\\mathbb\{P\}\_\{A\}\(F\)\>3/4\. PutD=t\+s0\+1D=t\+s\_\{0\}\+1,ϵ=1/\(4​D\)\\epsilon=1/\(4D\), and define prior B independently across tasks: with probability1−ϵ1\-\\epsilona task is pure and fair; otherwise it has one orL−1L\-1positive paths, with equal probability and a uniformly located exceptional path\. LetEEbe within\-task agreement of all revealed labels, vacuously true forri=0r\_\{i\}=0\. For every specified A\-compatible transcriptτ\\tau,

d​ℙB​\(T∈d​τ,E\)=w⁡\(τ\)​d​ℙA​\(T∈d​τ\),w⁡\(τ\)=∏ifri,d\\mathbb\{P\}\_\{B\}\(T\\in d\\tau,E\)=w\(\\tau\)d\\mathbb\{P\}\_\{A\}\(T\\in d\\tau\),\\qquad w\(\\tau\)=\\prod\_\{i\}f\_\{r\_\{i\}\},\(89\)wheref0=f1=1f\_\{0\}=f\_\{1\}=1andfr=1−ϵ​r/Lf\_\{r\}=1\-\\epsilon r/Lforr≥2r\\geq 2\. Fromr≤2​\(r−1\)r\\leq 2\(r\-1\)and Equation \([88](https://arxiv.org/html/2609.29140#A16.E88)\), onFF,

1≥w≥1−2​ϵ​\(t\+z\)L≥56\.1\\geq w\\geq 1\-\\frac\{2\\epsilon\(t\+z\)\}\{L\}\\geq\\frac\{5\}\{6\}\.
Construct a subprobability couplingQQby drawing an A transcript, restricting toFF, weighting byww, and then independently drawing latent A and B cohorts from their respective posteriors given the transcript \(andEEfor B\)\. Its mass is greater than5/85/8\. Its A marginal is dominated by the original A joint law, and its B marginal is B restricted toF,EF,E\. Thus honesty gives

Q⁡\(θA∉I\)≤α,Q⁡\(θB∉I\)≤α,𝔼A​\|I\|≥𝔼Q​\|I\|\.Q\(\\theta\_\{A\}\\notin I\)\\leq\\alpha,\\qquad Q\(\\theta\_\{B\}\\notin I\)\\leq\\alpha,\\qquad\\mathbb\{E\}\_\{A\}\|I\|\\geq\\mathbb\{E\}\_\{Q\}\|I\|\.\(90\)The independent A posterior draw is essential because omitted pure bits makeθA\\theta\_\{A\}no longer transcript\-measurable\.

Given a transcript inFF, the number of observed but noncensused tasks is at least

M−z−t\+zL−1≥M2−3​s02≥M4\.M\-z\-\\frac\{t\+z\}\{L\-1\}\\geq\\frac\{M\}\{2\}\-\\frac\{3s\_\{0\}\}\{2\}\\geq\\frac\{M\}\{4\}\.Each such task has B posterior variance at leastϵ/\(2​L3\)\\epsilon/\(2L^\{3\}\)\. Unobserved A and B task means add nonnegative variance\. ForH=M⁡\(θB−θA\)H=M\(\\theta\_\{B\}\-\\theta\_\{A\}\), conditional meanμ\\mu, and conditional variancevv, therefore

v≥aL​MD,aL=132​L3\.v\\geq a\_\{L\}\\frac\{M\}\{D\},\\qquad a\_\{L\}=\\frac\{1\}\{32L^\{3\}\}\.\(91\)DecomposeHHinto the independent B task means and the negative omitted A bits\. Each centered summand is bounded by one\. WithS=𝔼⁡\(H2\)=μ2\+vS=\\mathbb\{E\}\(H^\{2\}\)=\\mu^\{2\}\+v,

𝔼​H4≤3​S2\+4​\|μ\|​v\+v\.\\mathbb\{E\}H^\{4\}\\leq 3S^\{2\}\+4\|\\mu\|v\+v\.Ifv≥16v\\geq 16, the right side is below\(7/2\)​S2\(7/2\)S^\{2\}\. Paley–Zygmund applied toH2H^\{2\}gives

Q⁡\{\|H\|≥S/100∣T\}≥980135000\.Q\\\{\|H\|\\geq\\sqrt\{S/100\}\\mid T\\\}\\geq\\frac\{9801\}\{35000\}\.Equations \([90](https://arxiv.org/html/2609.29140#A16.E90)\)–\([91](https://arxiv.org/html/2609.29140#A16.E91)\) then imply

𝔼A​\|I\|≥\{58​980135000−2​α\}​aL/100M​D\.\\mathbb\{E\}\_\{A\}\|I\|\\geq\\left\\\{\\frac\{5\}\{8\}\\frac\{9801\}\{35000\}\-2\\alpha\\right\\\}\\frac\{\\sqrt\{a\_\{L\}/100\}\}\{\\sqrt\{MD\}\}\.The bracket is positive throughα=1/12\\alpha=1/12\.

IfaL​M/D<16a\_\{L\}M/D<16, compare the all\-zero cohort with a prior placing one positive path uniformly among theM​LMLpositions\. Any all\-zero\-compatible transcript reveals at mostM\+tM\+tpositions, so its no\-hit likelihood ratio is at least1−\(M\+t\)/\(M​L\)≥1/31\-\(M\+t\)/\(ML\)\\geq 1/3\. The same two\-target honesty argument used in Appendix[K](https://arxiv.org/html/2609.29140#A11)yields

𝔼0​\|I\|≥1−4​αM​L≥\(1−4​α\)​aL4​L​M​D\.\\mathbb\{E\}\_\{0\}\|I\|\\geq\\frac\{1\-4\\alpha\}\{ML\}\\geq\\frac\{\(1\-4\\alpha\)\\sqrt\{a\_\{L\}\}\}\{4L\\sqrt\{MD\}\}\.ForM<256M<256, this single\-positive bound is at least\(1−4​α\)/\(4​L\)\(1\-4\\alpha\)/\(4L\)times the target rate\. Finally, forM≥256M\\geq 256,t\+M≤D≤\(9/8\)​\(t\+M\)t\+\\sqrt\{M\}\\leq D\\leq\(9/8\)\(t\+\\sqrt\{M\}\)\. Taking the smallest positive constant from the branches proves the lower half of Equation \([5](https://arxiv.org/html/2609.29140#S4.E5)\)\.

#### Scope\.

The theorem concerns expected interval length on the worst pure fixed cohort subject to honesty over all fixed cohorts\. It does not establish pointwise adaptation, optimal finite constants, or a general benefit from task omission\. The upper rule was chosen to attain the order, not after observing empirical outcomes\. Its complete finite experiment in Appendix[Q](https://arxiv.org/html/2609.29140#A17)is adverse\.

## Appendix QProspective LiveCodeBench replication

#### Chronology and source\.

This extension was declared only after the omission theorem and its independent mathematical audit\. Before downloading or inspecting model correctness, a local prospective lock fixed the source revision, file digests, eligibility rule, methods, budgets, seeds, comparisons and analysis code\. This is not a public preregistration or independently witnessed timestamp\. The source is the official[Jain et al\. \(2025\)](https://arxiv.org/html/2609.29140#bib.bib5)code\-generation sample Space at immutable revision79837278b7c58c64a17c90936207db25f977f414\. Itsall\_outputs\.jsonfile has 284,487,753 bytes and SHA\-256b8fa8293294ed03c607e4c0d861b50203bd6aee395414d8f5d9d7e9a7853acc9\. The adapter verifies this digest and the pinned 880\-problem metadata\.

A model is eligible exactly when it has one source\-ordered record for every problem, and every record contains at least five binarypass1\_listentries\. We retain the first five\. Sixteen panels pass\. Six N=1 panels—the three O1\-2024\-12\-17 effort levels, O1\-Mini, O1\-Preview and QwQ\-32B\-Preview—fail the repetition rule and are listed as structural exclusions\. No model, task or output is filtered by accuracy, purity, interval width or result\. The adapter retains task identifiers and binary correctness only; it neither executes nor retains generated code\.

#### Frozen designs and analysis\.

HereM=880,L=5,K=1M=880,L=5,K=1, and

t∈\{0,8,15,29,30,88,220,440,880\}\.t\\in\\\{0,8,15,29,30,88,220,440,880\\\}\.At everytt, each method purchases exactlyM\+tM\+tlabels\. Task\-covering Joint uses one uniform label per task and a second distinct label in a uniformtt\-task subset, with Proposition[9](https://arxiv.org/html/2609.29140#Thmtheorem9)\. Exact pooled uniform samplesM\+tM\+tof the5​M5Mlabels without replacement and inverts the hypergeometric law\.

Omission\-Joint usess=⌊M⌋=29s=\\lfloor\\sqrt\{M\}\\rfloor=29whent2<Mt^\{2\}<Mand zero otherwise\. Fors\>0s\>0, selectn=M−sn=M\-stasks and auditq=t\+sq=t\+sof them\. Apply Joint inside the selected tasks at errorα/2\\alpha/2, add Equation \([85](https://arxiv.org/html/2609.29140#A16.E85)\) at errorα/2\\alpha/2, and intersect with the full compatible\-label range\. This replaces generic Audit only inside the same valid union\-bound construction; no interval is selected after seeing outcomes\.

Every panel/design/budget has 1000 independent evaluator randomizations from stable SHA\-256\-derived streams with base seed 20261010\. The complete grid has 432,000 runs\. We report conditional bias, MSE, mean width, coverage and charged labels\. Binomial intervals quantify coverage Monte Carlo error\. For every omission\-active comparison, 2000 independently resampled bootstrap draws quantify only randomization error in mean\-width differences\. Models share tasks and providers; panels are not independent draws from a model population, and no across\-modelpp\-value is used\.

#### Complete budget summary\.

Table[21](https://arxiv.org/html/2609.29140#A17.T21)reports all nine task\-covering comparisons\. Joint point MSE is lower than exact uniform in all 144 panel–budget cells\. Width crosses near the smallest declared audits: Joint wins 9/16 att=15t=15and 11/16 att=29t=29\. It wins 15/16 att=220t=220, with median width ratio 0\.694 and median MSE ratio 0\.130\. Table[22](https://arxiv.org/html/2609.29140#A17.T22)shows every model att=220t=220and full audit\. DeepSeek\-R1\-Lite\-Preview is the only width loss att=220t=220; DeepSeek\-R1\-Preview also loses at full audit\. Both retain substantial MSE gains\. Thus the conclusion is a cohort\-conditional policy–interval result, not uniform dominance\.

The post\-hoc finite\-region calculation in Appendix[O](https://arxiv.org/html/2609.29140#A15)uses the exact cohort total and mean pair disagreement to test Proposition[5](https://arxiv.org/html/2609.29140#Thmtheorem5)\. It retains all seven interior budgets and all 16 panels\. Its 73 sufficient certificates are all observed wins; five additional observed wins lie outside the sufficient region and are reported by model and budget\. The calculation uses the inspected full cohort and does not validate prospective design selection\.

Table 21:Separately protocol\-locked LiveCodeBench panels \(M=880,L=5,K=1M=880,L=5,K=1\)\. J/U is task\-covering Joint / exact fixed uniform; J/H is Joint / same\-observation count\-only Hull\. Ratios are medians across the 16 panels, using each panel’s mean width or MSE\. Below one favors Joint; parentheses give strict panel wins\. Coverage is the minimum–maximum across panels\. Every method usesM\+tM\+tlabels and 1000 randomizations per panel\. Hull is a conservative local\-law Chernoff comparator, computed post hoc on the locked Joint observations\.Table 22:Every eligible model at two declared budgets\. Entries are task\-covering Joint / exact fixed uniform; below one favors Joint\.
#### The adverse omission result\.

At the four active budgetst=0,8,15,29t=0,8,15,29, Omission\-Joint loses width to exact uniform in every panel\. Median ratios are respectively1\.346,1\.328,1\.324,1\.2871\.346,1\.328,1\.324,1\.287\. Att=0t=0it is narrower than task\-covering Joint in 15/16 panels, but at each positive active audit count it loses to task\-covering Joint in all 16\. All bootstrap intervals place the Omission\-Joint minus uniform mean\-width difference above zero throught=15t=15; 15/16 do so att=29t=29\. This does not contradict Theorem[4](https://arxiv.org/html/2609.29140#Thmtheorem4): the branch rule proves an asymptotic order, not finite dominance or optimized constants\. No post\-outcome omission count is substituted\.

#### Cohort structure, coverage, and checks\.

Panel accuracies range from 0\.222 to 0\.777\. Pure\-task fraction has median 0\.893 and range 0\.718–0\.959; mean random\-pair disagreement has median 0\.0518 and range 0\.0202–0\.1357\. These post\-lock descriptors help explain why task stratification is valuable but did not select the experiment\.

Across all 144 cells per method, empirical coverage ranges from 0\.992 to one for task\-covering Joint, 0\.994 to one for Omission\-Joint, and 0\.938 to 0\.972 for exact uniform\. The first two ranges indicate conservative certificates; the exact analytical arguments establish coverage\. There are no budget violations\. Before outcome access, exhaustive enumeration of the Omission\-Joint wrapper covered 8,797 small finite designs, 82,354 exact probability states and 26,391 interval conditions at three error levels, with no undercoverage\. After the run, every locked code digest still matches, all 16 panel summaries contain 27,000 rows, and the analyzer retains all 432 metric cells and 288 declared panel comparisons\. These computational checks are not formal floating\-point enclosures or human proof verification\.

## Appendix RPost\-hoc finite\-optimum and decision calibration

#### Chronology and scope\.

Every analysis in this section was declared after all LiveCodeBench outcomes and the primary comparisons had been inspected\. The declaration fixes all cells, width targets, omission\-selection criterion, randomization count and seed\. These results strengthen finite interpretation but are not held\-out confirmation\. Machine\-readable outputs retain 288,000 analyzed or newly sampled rows, every model and budget, and the complete declaration\.

### R\.1Same\-observation comparator and radius\-tuned omission

The mixed\-count Hull in Appendix[L](https://arxiv.org/html/2609.29140#A12)uses the same preselected audited\-task subset and the same one\-or\-two observed labels as Joint\. It relaxes unknown task compositions through exact finite local laws and concave envelopes, but does not use the disagreement statistic\. The local laws are exact; the resulting Chernoff interval is conservative and is not an optimal exact interval over all rules for the observation design\. We recompute it on all 144,000 locked task\-covering observations\. Table[23](https://arxiv.org/html/2609.29140#A18.T23)reports every budget\. Joint loses att=0t=0, where it falls back to its one\-draw rule, then wins in every panel for everyt≥8t\\geq 8\. Thus the favorable Joint/uniform comparison is not explained only by different observations\. Hull coverage is one in all 144 panel–budget cells; its analytical construction, not this conservative replay value, establishes validity\.

For the unrestricted policy class, forcings=⌊M⌋s=\\lfloor\\sqrt\{M\}\\rflooris only an order argument\. We instead choosesswithout outcomes by minimizing the pure\-cohort radius

rJoint​\(M−s,L,t\+s,0,αs\)\\displaystyle r\_\{\\rm Joint\}\(M\-s,L,t\+s,0;\\alpha\_\{s\}\)\+min\{sM,s​log⁡\(4/α\)/2M−s\},αs=α𝟏\{s=0\}\+\(α/2\)𝟏\{s\>0\}\.\\displaystyle\+\\min\\left\\\{\\frac\{s\}\{M\},\\frac\{\\sqrt\{s\\log\(4/\\alpha\)/2\}\}\{M\-s\}\\right\\\},\\qquad\\alpha\_\{s\}=\\alpha\\mathbf\{1\}\\\{s=0\\\}\+\(\\alpha/2\)\\mathbf\{1\}\\\{s\>0\\\}\.over every feasible integer0≤s≤⌊\(M−t\)/2⌋0\\leq s\\leq\\lfloor\(M\-t\)/2\\rfloor, breaking ties toward smallerss\. This selector uses onlyM,L,t,αM,L,t,\\alphaand the fixed certificate formulas\. It choosess=39,28s=39,28att=0,8t=0,8and zero thereafter\. Fresh 1000\-randomization evaluation per cell uses seed 20261020\. It still loses to uniform in all panels att=0,8t=0,8, then matches the task\-covering distribution\. Minimum observed coverage is 0\.993 and there are no budget violations\. This is a valid fallback, not evidence that omission has favorable finite constants\.

Table 23:Complete post\-hoc budget summary\. Opt/U is the radius\-selected unrestricted rule / exact uniform; J/H is locked task\-covering Joint / same\-observation Hull\. Parentheses are strict panel wins out of 16\.
### R\.2Finite confidence\-rule program on tiny designs

The complete grid isL=3L=3,M∈\{2,3,4\}M\\in\\\{2,3,4\\\}, andL=5L=5,M∈\{2,3\}M\\in\\\{2,3\\\}, with every integer0≤t≤M0\\leq t\\leq M: 19 cells in total\. NoM=1M=1cell is included\. The optimization concerns this fixed sampling design and the worst pure cohort, subject to coverage over every cohort\.

For a fixed task\-covering design, letxxrecord the audited subset and every observed within\-task success count\. A cohort is a vectorh∈\{0,…,L\}Mh\\in\\\{0,\\ldots,L\\\}^\{M\}, with target\-grid indexH=∑ihiH=\\sum\_\{i\}h\_\{i\}, andPh​\(x\)P\_\{h\}\(x\)is its finite randomization law\. For every observationxxand target grid interval\[a,b\]\[a,b\], the program selects probabilityqx,a,bq\_\{x,a,b\}\. It solves

minq\\displaystyle\\min\_\{q\}\\quadW,\\displaystyle W,\(92\)s\.t\.∑xPh​\(x\)​∑a≤H≤bqx,a,b≥1−α\\displaystyle\\sum\_\{x\}P\_\{h\}\(x\)\\sum\_\{a\\leq H\\leq b\}q\_\{x,a,b\}\\geq 1\-\\alphafor every​h,\\displaystyle\\text\{for every \}h,\(93\)∑a≤bqx,a,b=1\\displaystyle\\sum\_\{a\\leq b\}q\_\{x,a,b\}=1for every​x,\\displaystyle\\text\{for every \}x,\(94\)∑xPp​\(x\)​∑a≤bqx,a,b​b−aM​L≤W\\displaystyle\\sum\_\{x\}P\_\{p\}\(x\)\\sum\_\{a\\leq b\}q\_\{x,a,b\}\\frac\{b\-a\}\{ML\}\\leq Wfor every pure​p\.\\displaystyle\\text\{for every pure \}p\.\(95\)Restricting endpoints to the target grid loses nothing: an endpoint between adjacent targets can move inward without changing coverage\. The program is the finite optimum among randomized interval rules for this fixed design and is a lower benchmark for deterministic rules\. It does not optimize sampling policies\. We solve every declared cell in double precision with HiGHS; all minimum reconstructed coverages are within1\.2×10−141\.2\\times 10^\{\-14\}of 0\.95 or above, and maximum state\-probability residual is1\.4×10−131\.4\\times 10^\{\-13\}\. These residuals are numerical checks, not rational dual certificates\.

Table[24](https://arxiv.org/html/2609.29140#A18.T24)retains all 19 cells\. Joint is within a factor 1\.116–1\.818 of the numerical randomized optimum on this grid\. The displayed Joint and Hull ratios coincide in 18 cells, so these tiny designs provide little evidence separating their finite efficiency\. The reported factors apply only to the checked designs\. They neither establish a constant\-gap guarantee atM=880M=880nor a lower benchmark for the best adaptive sampling policy; the program holds sampling fixed\.

Table 24:All 19 tiny\-design program cells, withM≥2M\\geq 2\.W∗W^\{\*\}is the double\-precision numerical randomized\-rule optimum for the fixed sampling design; the last columns are worst\-pure expected width divided byW∗W^\{\*\}\.
### R\.3Width targets and computation

For each method and panel, take the monotone envelope of mean width over the declared budget grid and report the first grid point at or below each fixed target\. There is no interpolation\. These are retrospective comparisons of Monte Carlo mean widths on a fixed grid, not guarantees for each realized interval or an outcome\-dependent stopping procedure\. Table[25](https://arxiv.org/html/2609.29140#A18.T25)shows that the useful regime is a precision requirement, not every possible target\. At width 0\.04, Joint saves a median 660 labels with no panel\-level loss; at 0\.05 it saves 352 in the median but loses in three panels\. At 0\.06, the initial uniform design already suffices and is uniformly cheaper on this grid\.

Table 25:Labels required on the declared LiveCodeBench grid\. Medians use resolved panels only;\+\+/==/−\-compare Joint savings with uniform\.The complete Hull recomputation plus 144,000 new optimized\-policy runs takes 64 seconds on the analysis host, with peak resident memory about 106 MiB\. An independent checker reaggregates every row, recomputes 1,440 Hull endpoints, freshly replays 1,056 optimized allocations, and reruns all 19 finite programs\. It finds minimum coverage 1\.000 for Hull and 0\.993 for the optimized rule, with no budget or summary discrepancy\. More evaluator randomizations would reduce Monte Carlo error but would not address model\- or task\-population generalization\.