更多程序还是更多次数？在 LLM 调优框架中区分覆盖面与专业化

# More Programs or More Rolls?Separating Coverage from Specialization in LLM Harnesses
Source: [https://arxiv.org/html/2609.35873](https://arxiv.org/html/2609.35873)
## More Programs or More Rolls? Separating Coverage from Specialization in LLM HarnessesThanks:Code is available at:[https://github\.com/StatXzy7/harness\-eval](https://github.com/StatXzy7/harness-eval)\.

Ziyang XuAffiliation:Department of Mathematics, The Chinese University of Hong KongHong Kong, ChinaEmail:[ziyang\.xu@link\.cuhk\.edu\.hk](mailto:[email protected])Haitian ZhongAffiliation:New Laboratory of Pattern Recognition \(NLPR\)State Key Laboratory of Multimodal Artificial Intelligence Systems \(MAIS\),Institute of Automation, Chinese Academy of SciencesAffiliation:Zhongguancun AcademyEmail:[tieyongzeng@bnbu\.edu\.cn](mailto:[email protected])Hao ZhouHao QinAffiliation:Department of Mathematics, The Chinese University of Hong KongHong Kong, ChinaChenhan JinAffiliation:Department of Mathematics, The Chinese University of Hong KongHong Kong, ChinaTe QiAffiliation:Department of Basic ResearchSINOPEC Research Institute of Petroleum ProcessingShengze XuAffiliation:Department of Mathematics, The Chinese University of Hong KongHong Kong, ChinaTieyong Zeng††thanks:Corresponding author\.Affiliation:Institute for Advanced StudyBeijing Normal\-Hong Kong Baptist University, Zhuhai, ChinaAffiliation:School of Mathematics and Statistics, Guangzhou Nanfang CollegeGuangzhou, China

###### Abstract

Automated generation of LLM harnesses promises to improve inference through task specialization\. Yet additional answer coverage can arise from repeated execution of the same program, making specialization difficult to identify\. We introduce a controlled evaluation that separates answer coverage, repeatable task advantages, and gains from pre\-execution selection\. On386386MATH\-500 tasks, we compare eight generated harnesses plus a baseline with nine byte\-identical baseline copies, using three executions per member\. Identical programs yield2\.162\.16percentage points of repeat\-averaged oracle headroom\. Generated programs exhibit substantially more repeatable score patterns, but these chiefly reveal persistent weaknesses: losses relative to the baseline persist across all three repeats on100100tasks, while persistent wins occur on only one task and are sensitive to answer extraction\. The frozen selector gains0\.000\.00percentage points, and both populations reach98\.70%98\.70\\%oracle coverage at2727harness executions\. Stable complementarity remains unresolved at three repeats\. Supporting BIRD traces locate failures in mechanism implementation, activation, and output validity\. Together, these findings establish why coverage and repeatability alone cannot justify claims of useful specialization\. They motivate an evaluation standard for harness diversity: task advantages should persist across executions, guide usable decisions, and improve on additional fixed\-program executions under matched inference budgets\.

## 1Introduction

Automated agent design allows language models to write the programs that organize a solver’s inference\. These LLM harnesses arrange prompts, tools, and control flow around a fixed solver\([Hu et al\., 2024](https://arxiv.org/html/2609.35873#bib.bib31);[Nie et al\., 2026](https://arxiv.org/html/2609.35873#bib.bib1);[Zhang et al\., 2026b](https://arxiv.org/html/2609.35873#bib.bib2)\)\. A population of such programs offers an attractive possibility: a division of labor in which different members address different task\-specific failures\. Realizing this possibility would let a system allocate inference to the program best suited to each task\. It raises a concrete question: when should a fixed budget buy more programs, and when would further executions of one program suffice?

The most immediate evidence for this division of labor is also ambiguous\. A population can contain more correct answers than any single member simply because it provides more attempts\. Pass@kkmeasures the probability of at least one correct answer amongkksamples; self\-consistency and sampling\-and\-voting ensembles turn repeated attempts into an aggregated answer\([Chen et al\., 2021](https://arxiv.org/html/2609.35873#bib.bib26);[Wang et al\., 2022](https://arxiv.org/html/2609.35873#bib.bib16);[Li et al\., 2024](https://arxiv.org/html/2609.35873#bib.bib30)\)\. An oracle that selects after observing correctness can exploit complementary successes even when every member has identical code\. Additional coverage therefore leaves open how much the population benefits from having different programs\.

Repeating the evaluation introduces a second distinction\. With few repeats, oracle estimates can still select favorable estimation errors; even strongly repeatable score patterns can reflect a program’s persistent weaknesses\. Exploiting specialization before execution requires task advantages that recur and can be recognized from the task\. We therefore distinguish*answer coverage*,*repeat\-stable task advantages*, and*usable selection*\. These distinctions turn the promise of program specialization into an evaluation problem: identify which advantages persist, which can guide a choice, and what they add beyond repeated execution of a fixed program\.

![Refer to caption](https://arxiv.org/html/2609.35873v1/teaser_illustrated.png)Figure 1:Coverage, repeatability, and selection on386386complete MATH tasks\.Oracle coverage matches executions, not tokens/calls; headroom uses repeat means\. Persistent task counts require at least one generated member to win/lose against baseline in all three repeats; the sole win is extraction\-sensitive\. HereGGis held\-out task\-wise selection gain over the discovery\-selected fixed member; panel \(b\) labels residual Pearson correlation asrr\. R1–R3 are repeat labels; pp denotes percentage points and CI denotes confidence interval\. Intervals are 95% bootstrap CIs\. Program drawings are schematic\.We investigate this problem on MATH\-500, starting with the coverage offered by a generated population\. We then compare a selected low\-call panel with byte\-identical baseline copies, holding the solver, population size, and repeat count fixed\. Choices learned from earlier executions are tested on held\-out repeats\. A selector frozen on development data tests pre\-execution choice, and oracle replay compares coverage at equal harness execution counts\. Separate BIRD traces examine whether proposed mechanisms are implemented, triggered, and able to return valid answers\. This sequence follows the evidence needed to move from additional answers to useful specialization\.

The central empirical finding is that the generated panel’s repeatable differences predominantly identify weaknesses \(Figure[1](https://arxiv.org/html/2609.35873#S1.F1)\)\. Persistent losses span100100tasks, while persistent wins occur on one task and are sensitive to answer extraction\. Identical programs already yield positive oracle headroom\. The clone\-adjusted held\-out gain is−0\.26\-0\.26pp, leaving stable complementarity unresolved at three repeats\. The frozen selector gains0\.000\.00pp, and both arms reach98\.70%98\.70\\%oracle coverage at2727executions\. Thus, substantial repeatability coexists with zero gain for the tested selector and matching full\-budget coverage from further baseline executions\.

The study provides a controlled basis for attributing population gains: a same\-code sampling reference, a directional analysis of repeated wins and losses, and separate evaluations of pre\-execution choice and post\-execution coverage\. Together with the execution traces, these analyses connect program differences to the behavior and decisions through which they could become useful\. The resulting design target is a task advantage that persists, guides a usable decision, and adds value beyond further attempts with a fixed program\.

## 2Related Work

#### Automated agent and harness design\.

Executable programs provide a flexible space for adapting LLM behavior\. ADAS searches this space directly\([Hu et al\., 2024](https://arxiv.org/html/2609.35873#bib.bib31)\); Self\-Harness uses failure traces to propose edits and regression tests to retain them\([Zhang et al\., 2026b](https://arxiv.org/html/2609.35873#bib.bib2)\); TTHE adapts the harness during evaluation from unlabeled traces\([Nie et al\., 2026](https://arxiv.org/html/2609.35873#bib.bib1)\)\. Related systems improve task adaptation and validation allocation\([Zhang et al\., 2026a](https://arxiv.org/html/2609.35873#bib.bib3);[Miyai et al\., 2026](https://arxiv.org/html/2609.35873#bib.bib4)\), or expose harness behavior through verification, diagnosis, localization, and policy evolution\([Xu et al\., 2026](https://arxiv.org/html/2609.35873#bib.bib5);[Chen et al\., 2026a](https://arxiv.org/html/2609.35873#bib.bib6);[Lin et al\., 2026](https://arxiv.org/html/2609.35873#bib.bib7);[Luo et al\., 2026](https://arxiv.org/html/2609.35873#bib.bib8);[Wang et al\., 2026](https://arxiv.org/html/2609.35873#bib.bib9);[Chen et al\., 2026b](https://arxiv.org/html/2609.35873#bib.bib27)\)\.[El et al\. \(2025\)](https://arxiv.org/html/2609.35873#bib.bib29)report low diversity in task\-level score patterns among the automatically designed agents they study\. We examine program populations through both execution traces and repeated outcomes: traces reveal which mechanisms execute, and same\-code repeats measure coverage from sampling\. This perspective connects program equivalence and mutation testing\([Budd and Angluin, 1982](https://arxiv.org/html/2609.35873#bib.bib24);[Papadakis et al\., 2019](https://arxiv.org/html/2609.35873#bib.bib25)\)with performance attribution in generated harness populations\.

#### Repeated sampling and ensembles\.

Multiple executions can improve answer coverage even with a fixed program\. Pass@kkmeasures success withinkksamples; self\-consistency and identical\-agent ensembles aggregate attempts\([Chen et al\., 2021](https://arxiv.org/html/2609.35873#bib.bib26);[Wang et al\., 2022](https://arxiv.org/html/2609.35873#bib.bib16);[Li et al\., 2024](https://arxiv.org/html/2609.35873#bib.bib30)\), while nominally temperature\-00calls can also vary\([Ouyang et al\., 2023](https://arxiv.org/html/2609.35873#bib.bib33)\)\. These gains provide the sampling reference in our execution\-matched comparison\. A second distinction concerns when selection occurs\. Post\-execution verifiers rank candidate solutions\([Kwok et al\., 2026](https://arxiv.org/html/2609.35873#bib.bib28)\); ensemble selection chooses a model combination on validation data and averages predictions at inference\([Caruana et al\., 2004](https://arxiv.org/html/2609.35873#bib.bib15)\)\. Our frozen selector chooses a program before execution, while oracle replay measures coverage across recorded outputs\.

#### Selection and resource allocation\.

Turning a population into a useful choice connects harness design to algorithm selection and portfolios\([Rice, 1976](https://arxiv.org/html/2609.35873#bib.bib13);[Xu et al\., 2008](https://arxiv.org/html/2609.35873#bib.bib17);[Kotthoff, 2014](https://arxiv.org/html/2609.35873#bib.bib18)\)\. HELIX uses complementary sibling trajectories to expand verified coverage and supply data for model improvement\([Fan and Huang, 2026](https://arxiv.org/html/2609.35873#bib.bib12)\)\. STS compares six reasoning paradigms and trains an embedding\-based router to recover part of their oracle gap\([Zhou et al\., 2026](https://arxiv.org/html/2609.35873#bib.bib10)\)\. Harness sensitivity across model–problem pairs\([Gupta et al\., 2026](https://arxiv.org/html/2609.35873#bib.bib20)\)and evaluated models\([Cho, 2026](https://arxiv.org/html/2609.35873#bib.bib21)\), search routing\([Kim, 2026](https://arxiv.org/html/2609.35873#bib.bib19)\), and preference\-trained model routing\([Ong et al\., 2024](https://arxiv.org/html/2609.35873#bib.bib32)\)further motivate task\-dependent choices\. GRASP selects edits to a bounded skill library under regression constraints\([Moll et al\., 2026](https://arxiv.org/html/2609.35873#bib.bib11)\)\. We hold the target model fixed and measure oracle coverage, repeat\-stable task advantages, and realized selector gain in the same program population\. The distinction between quantitative and qualitative interaction\([Gail and Simon, 1985](https://arxiv.org/html/2609.35873#bib.bib14)\)clarifies why varying effect sizes can coexist with an unchanged best action\.

## 3Problem Formulation

We distinguish the answers a population provides, the task advantages that persist across executions, and the gains a selector realizes\. LetPPbe a population of harness members containing the single\-call baselineH0H\_\{0\}\. Writeh∈Ph\\in Pfor a member,xix\_\{i\}for taskiiamongnnevaluated tasks, andr∈\{1,…,R\}r\\in\\\{1,\\ldots,R\\\}for one ofRRexecution repeats per member–task pair\. The scoreY⁡\(h,x,r\)∈\{0,1\}Y\(h,x,r\)\\in\\\{0,1\\\}is one exactly when memberhhanswers taskxxcorrectly in repeatrr\. Equations use proportions; multiplying by100100gives accuracies in % and gains or headroom in percentage points \(pp\)\. Appendix[A](https://arxiv.org/html/2609.35873#A1)summarizes the notation\.

#### Coverage: which additional answers are available?

A one\-run oracle selects a member after seeing correctness\. For a fixed repeatrr, its headroom over the best fixed member in the recorded matrix is

H⁡\(P\)=1n​∑i=1nmaxh∈P⁡Y⁡\(h,xi,r\)−maxh∈P⁡1n​∑i=1nY⁡\(h,xi,r\)\.H\(P\)=\\tfrac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\max\_\{h\\in P\}Y\(h,x\_\{i\},r\)\-\\max\_\{h\\in P\}\\tfrac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}Y\(h,x\_\{i\},r\)\.\(1\)This quantity captures available correct answers, including those obtained through repeated sampling\. The repeat\-averaged pluginH^\\widehat\{H\}replacesY⁡\(h,x,r\)Y\(h,x,r\)in Equation[1](https://arxiv.org/html/2609.35873#S3.E1)withp^h​\(x\)=R−1​∑r=1RY⁡\(h,x,r\)\\hat\{p\}\_\{h\}\(x\)=R^\{\-1\}\\sum\_\{r=1\}^\{R\}Y\(h,x,r\)on complete observations\. With few repeats, maximizing those means can select favorable estimation errors, which motivates the same\-code control\.

#### Stable complementarity: which member is best for each task?

Writeph\(x\)=Pr\{Y\(h,x,r\)=1\}p\_\{h\}\(x\)=\\Pr\\\{Y\(h,x,r\)=1\\\}for memberhh’s success probability over fresh executions under a fixed execution condition\. The corresponding population headroom isHstable=𝔼x​maxh​ph​\(x\)−maxh⁡𝔼x​ph​\(x\)H\_\{\\rm stable\}=\\mathbb\{E\}\_\{x\}\\max\_\{h\}p\_\{h\}\(x\)\-\\max\_\{h\}\\mathbb\{E\}\_\{x\}p\_\{h\}\(x\), where𝔼x\\mathbb\{E\}\_\{x\}averages over the target task distribution and all member maxima range overPP\. ForP=\{H0,H1\}P=\\\{H\_\{0\},H\_\{1\}\\\}, abbreviatepHj​\(x\)p\_\{H\_\{j\}\}\(x\)aspj​\(x\)p\_\{j\}\(x\)forj∈\{0,1\}j\\in\\\{0,1\\\}and defineΔ⁡\(x\)=p1​\(x\)−p0​\(x\)\\Delta\(x\)=p\_\{1\}\(x\)\-p\_\{0\}\(x\)\. Then

Hstable=12​\(𝔼x​\|Δ⁡\(x\)\|−\|𝔼x​Δ​\(x\)\|\)\.H\_\{\\rm stable\}=\\tfrac\{1\}\{2\}\\bigl\(\\mathbb\{E\}\_\{x\}\|\\Delta\(x\)\|\-\|\\mathbb\{E\}\_\{x\}\\Delta\(x\)\|\\bigr\)\.\(2\)Positive complementarity requires task\-wise crossover: the better member changes with the task\. If one member is better everywhere, headroom is zero even when its advantage varies in size\. Identical programs under the same condition have equalph​\(x\)p\_\{h\}\(x\), giving a zero\-complementarity reference\. This definition concerns population probabilities; the repeat diagnostic below tests choices learned from finite executions\.

For a concrete example, consider two equally common task types with success probabilities\(0\.90,0\.60\)\(0\.90,0\.60\)forH0H\_\{0\}and\(0\.80,0\.20\)\(0\.80,0\.20\)forH1H\_\{1\}\. The performance gap is task\-dependent, butH0H\_\{0\}remains preferable on both types\. The task\-wise oracle and the best fixed member therefore both achieve0\.750\.75, leaving zero stable headroom\. This constructed example separates variation in the size of an advantage from a change in the preferred action\. It motivates checking the direction of repeated differences, alongside their repeatability\.

#### Usable selection: which advantage can guide a decision?

A selectorπZ\\pi\_\{Z\}maps the pre\-execution featuresZ⁡\(x\)Z\(x\)of taskxxto a memberπZ​\(x\)∈P\\pi\_\{Z\}\(x\)\\in P\. Its usefulness depends on whether those features predict the relevant advantage\. We evaluate this decision separately from choices learned from earlier outcomes on the same task, which test repeat generalization\. An execution\-matched oracle replay then compares the coverage offered by a portfolio with additional fixed\-program executions\. Its budget counts harness executions; each execution may contain multiple model calls and tokens\.

## 4Experimental Design

### 4\.1Harness Populations and Evaluation Setup

We hold the target solver, GLM\-5\.3\-Flash, fixed and vary the Python programs that call it\. A*builder*writes each harness, the*solver*answers its model calls, and an external*judge*scores the final output\. Gold answers are available only to the judge\. Each harness receives a question and can query the fixed solver\. BIRD harnesses also have access to database schemas and SQL execution feedback\. The single\-call baseline makes one temperature\-00call\. Generated programs can change prompts, decoding, sample count, and control flow; interface and model details appear in Appendix[B](https://arxiv.org/html/2609.35873#A2)\.

MATH\-500 tests whether additional coverage survives repetition and supports useful program choices\. Table[1](https://arxiv.org/html/2609.35873#S4.T1)maps these questions to their comparisons\. BIRD separately examines how proposed mechanisms execute and what admission screening changes\.

Table 1:Comparisons separating coverage from usable specialization\.Oracle comparisons use correctness after execution; the frozen selector chooses from the question before execution\.Dev\-fixeddenotes the development\-selected fixed member\.DDis generated\-panel minus clone held\-out selection gain \(Equation[3](https://arxiv.org/html/2609.35873#S4.E3)\);S⁡\(πZ\)S\(\\pi\_\{Z\}\)is the frozen selector’s accuracy gain over dev\-fixed\.#### MATH tasks and scoring\.

MATH\-500\([Lightman et al\., 2023](https://arxiv.org/html/2609.35873#bib.bib23)\)is split into100100development and400400evaluation problems\. All primary MATH comparisons use the same386386evaluation problems, selected after collection by requiring complete observations in both repeat arms and the single\-run population\. This excludes1414problems with incomplete repeat observations; Appendix[C](https://arxiv.org/html/2609.35873#A3)gives completeness counts, exclusions, and conflict sensitivity\. The judge extracts the final answer, preferring the designated final\-answer line, and compares numeric values when both parse or normalized LaTeX otherwise \(Appendix[B\.1](https://arxiv.org/html/2609.35873#A2.SS1)\)\.

#### MATH generation and repeat panel\.

Six builders—GLM\-5\.3, Qwen3\.8\-Max, DeepSeek\-V4\-Pro, Kimi\-K3, MiniMax\-M3, and ERNIE\-5\.0\-Thinking\-Preview—each receive eight slots under generation seed00\. A slot is an independent opportunity to improve on a single greedy call: the builder chooses the strategy, with no cross\-slot diversity constraint\. Each slot allows up to three attempts at temperature0\.70\.7and32,76832\{,\}768output tokens, using the same prompt without feedback from earlier attempts\. Generation stops at the first program that runs successfully and returns nonempty answers on two development questions\. The4848slots yield3535frozen programs \(builder instructions: Appendix[B\.1](https://arxiv.org/html/2609.35873#A2.SS1.SSS0.Px1)\)\.

The repeat panel samples eight of these programs\. A historical mean\-call ceiling of3\.053\.05leaves2121eligible members; a fixed salted hash selects one per builder and two at\-large seats\. The resulting low\-call panel includes draft–check–revise, agreement checking, and sample voting \(Appendix[B\.2](https://arxiv.org/html/2609.35873#A2.SS2)\)\. Adding single\-call gives nine members\. This panel and nine identical baseline slots each receive three fresh, cache\-disabled repeats, with decoding fixed by the source\.

The selected programs differ in how they obtain and resolve additional answers\. Some request several samples and vote on normalized answers; others verify a draft, obtain an alternative solution, or request adjudication when answers disagree\. One uses differently prompted solutions and stops when three agree\. Verification and adjudication are additional prompts to the same solver, without access to gold answers\. A single logical call can request five samples, and the3\.053\.05eligibility threshold is a historical mean rather than a per\-task maximum\. The inventory describes implemented operations; their presence alone does not show that each branch activates or improves an answer\.

#### BIRD discovery and admission\.

BIRD\([Li et al\., 2023](https://arxiv.org/html/2609.35873#bib.bib22)\)requires SQL for a question and database\. In the discovery study, TTHE uses a GLM\-5\.3\-Flash builder to produce1212candidates that pass basic execution checks\. These programs and two supplied baselines are evaluated on6060questions\. The admission study crosses free versus assigned \(“forced”\) strategies with ungated versus conformance\-gated admission: six builders, three seeds, and eight slots per builder–seed cell in each arm, with up to three attempts per slot \(temperature0\.70\.7;16,38416\{,\}384output tokens\)\. Forced strategies include execution repair, three\-sample voting, and schema linking\. All arms require executable programs and nonempty SQL; gated arms also check mechanism conformance and exclude some structures\.

The admission split uses365365questions from two databases for development and1,1691\{,\}169from nine databases for evaluation, with a pre\-drawn400400\-question core shared by all four arms\. An1818\-item smoke test is a development\-exposure exception\. The official set\-of\-rows scorer compares predicted and gold query results, with a3030\-second timeout and failures scored zero\. Discovery uses an earlier stringified\-row proxy with a row cap\. Admission uses shared response caching, whereas the MATH repeats disable caching\. Full split, scoring, and bundled gating specifications appear in Appendices[F\.1](https://arxiv.org/html/2609.35873#A6.SS1)and[G](https://arxiv.org/html/2609.35873#A7)\.

### 4\.2Same\-Code Controls and Repeat Validation

#### Learning a choice on two repeats, testing it on the third\.

Each fold uses two discovery repeats to select a member for every task and a single best fixed member\. The third repeat measures their accuracy difference\. Averaging over tasks and the three folds gives gainGG:GrealG\_\{\\rm real\}for the generated panel plus baseline andGcloneG\_\{\\rm clone\}for the same\-code arm\. We compare paired task\-level gains on the common complete tasks through

D=Greal−Gclone\.D=G\_\{\\rm real\}\-G\_\{\\rm clone\}\.\(3\)A positive difference means that choices learned from past outcomes transfer better for distinct programs than for identical ones\. Both the task\-wise choice and the fixed comparator are determined entirely by the discovery repeats\. Thus,GGevaluates a choice learned from finite observations: it depends on both the underlying task advantages and how accurately the discovery repeats identify them\.DDcompares this learning procedure across the two arms\. It is distinct fromHstableH\_\{\\rm stable\}, which assumes access to the success probabilities themselves\. Its confidence interval is therefore not an interval forHstableH\_\{\\rm stable\}; a near\-zeroDDneed not imply a small population\-level opportunity\.

Positive support requiresD\>1\.0D\>1\.0pp, positive 95% bootstrap lower bounds for bothDDandGrealG\_\{\\rm real\}, and randomizationp<0\.05p<0\.05\. Before subsetting, incomplete tasks must account for at most10%10\\%of either arm or their paired intersection, with at least3030paired tasks retained\. These conditions combine effect size, uncertainty, and observation completeness; failure to satisfy them yieldsInsufficient Evidence\.

#### Distinguishing repeatability from advantage\.

We denote byρres\\rho\_\{\\rm res\}the mean held\-out\-versus\-discovery Pearson correlation of score residuals across repeats, after removing additive member and task effects\. This exploratory measure describes repeatability, whileGGdescribes held\-out selection gain\. Known\-truth simulations characterize the diagnostic’s sensitivity\. Appendices[C\.1](https://arxiv.org/html/2609.35873#A3.SS1)and[D](https://arxiv.org/html/2609.35873#A4)specify these calculations, tie handling, uncertainty estimates, and decision rules\. The acquisition protocol preceded collection; the repeat diagnostic and common\-complete\-set analysis were developed afterward\. Recovery retained the programs, solver, tasks, and repeat design \(Appendix[C](https://arxiv.org/html/2609.35873#A3)\)\.

#### Checking the diagnostic against known truth\.

Calibration includes equal\-ability and global\-dominance settings withHstable=0H\_\{\\rm stable\}=0, alongside alternatives in which the best member changes with the task\. In the strong two\-specialist case,400400tasks form two equal halves\. Each specialist succeeds with probability0\.950\.95on its own half and0\.600\.60on the other; the remaining seven members have probability0\.600\.60throughout\. The oracle accuracy is0\.950\.95and the best fixed accuracy is0\.7750\.775, giving17\.517\.5pp of true headroom\. These settings distinguish false support from failure to detect an existing advantage\. The simulation variant uses fewer resampling draws and checks completeness in the real arm; the empirical rule also checks clones and the paired intersection\. Appendix[D](https://arxiv.org/html/2609.35873#A4)specifies both rules and their tie handling\.

### 4\.3Frozen Selection and Execution\-Matched Replay

#### Selecting a program from the question\.

Using only development data, we fitπZ\\pi\_\{Z\}to predict the member with the highest available\-repeat mean correctness on each task; training\-label ties follow member\-ID order\. Five task folds keep each task’s repeats together\. The fitted models share a term\-frequency–inverse\-document\-frequency \(TF\-IDF\) vocabulary learned from development questions and use training\-fold member accuracies as additional inputs\. At inference, we average the fitted fold models’ probability vectors and select the top member, falling back to single\-call when the top\-two gap is below0\.150\.15, including a tie for the highest probability\. The policy is frozen before evaluation; Appendix[E](https://arxiv.org/html/2609.35873#A5)specifies fitting and fallback details\. Comparators include the development\-selected fixed memberhdev​\-​fixedh\_\{\\rm dev\\text\{\-\}fixed\}and one randomly drawn fixed member\. We writeS⁡\(πZ\)S\(\\pi\_\{Z\}\)for the policy accuracy minus that ofhdev​\-​fixedh\_\{\\rm dev\\text\{\-\}fixed\}, averaging over evaluation tasks and repeats\.

#### Allocating executions to a portfolio\.

An offline oracle replay compares a development\-ranked portfolio with same\-code executions at per\-task budgetsb∈\{3,9,15,27\}b\\in\\\{3,9,15,27\\\}harness executions\. Each included member contributes three repeats\. For clones, we compute exact mean coverage over uniform size\-bbsubsets of the2727recorded executions\. Both arms use post\-execution correctness to measure available answers\. The recorded selector choices and portfolio order stay fixed for the recovery analysis\. The budget unit is a harness execution, with model\-call and token use determined by the program\. Appendix[E](https://arxiv.org/html/2609.35873#A5)gives the policy, replay formula, and accounting details\.

## 5Results

### 5\.1The Population Covers More Tasks Than Any Single Member

The full generated population plus single\-call reaches99\.22%99\.22\\%oracle coverage, compared with95\.08%95\.08\\%for the best fixed member, single\-call\. Every generated member is individually worse \(69\.6969\.69–92\.49%92\.49\\%\), yet their union adds4\.154\.15pp of coverage available to an oracle after execution\.

This coverage comes with substantial variation in scored outcomes:2929of the3636correctness vectors are unique, mean pairwise disagreement is10\.94%10\.94\\%\(maximum33\.16%33\.16\\%\), and the union repairs84\.21%84\.21\\%of baseline errors\. The next question is how much of this variation survives repetition\. The following comparisons use the selected panel and its fresh executions, with baseline accuracy95\.85%95\.85\\%, to distinguish repeatable advantages from sampling\. Their population and executions differ from the full\-pool single\-run coverage reported here \(§[4\.1](https://arxiv.org/html/2609.35873#S4.SS1.SSS0.Px1)\)\.

### 5\.2Identical Programs Also Produce Positive Oracle Estimates

Identical programs produce a2\.162\.16pp repeat\-averaged oracle estimate, compared with2\.332\.33pp for the generated panel \(Table[2](https://arxiv.org/html/2609.35873#S5.T2)\)\. The nine clone slots implement the same program, so their population\-level stable complementarity is zero by construction\. Thus, even after averaging three repeats, a positive oracle estimate alone does not identify specialization\.

Table 2:Coverage, repeatability, and selection gain separate\.MATH\-500, selected panel versus the same\-code control \(§[4\.1](https://arxiv.org/html/2609.35873#S4.SS1.SSS0.Px1)\)\.H^\\widehat\{H\}uses repeat means;GGtests choices on held\-out repeats\.DDis their paired, unrounded between\-arm difference, so displayed means may not subtract exactly\.
### 5\.3Repeatable Score Patterns Are Dominated by Losses

Generated programs have more repeatable scored\-outcome patterns than clones\. Their mean residual correlation is0\.7890\.789, compared with0\.0060\.006for clones; the paired\-bootstrap difference is0\.7830\.783\(95% interval\[0\.69,0\.87\]\[0\.69,0\.87\]\)\. The correlation measures repeatability after removing additive member and task effects \(§[4\.2](https://arxiv.org/html/2609.35873#S4.SS2)\)\.

Persistent differences overwhelmingly favor single\-call\. All\-three\-repeat losses span100100tasks \(570570member–task pairs\), whereas all\-three\-repeat wins occur on one task shared by all eight generated members\. That apparent repair is sensitive to answer\-format extraction \(Appendix[C\.1](https://arxiv.org/html/2609.35873#A3.SS1)\)\. This directional asymmetry explains how strong score repeatability can coexist with few observed wins\.

The winning task illustrates why the scoring path matters\. Its gold answer is12\. The baseline ends with\#\#\#\# 12th grade, whose complete final\-answer payload is retained and scored incorrect\. A generated member returns12th grade, which instead reaches numeric fallback and is scored correct\. The recorded advantage therefore reflects an answer\-format difference\. We retain the original scores; this example explains the sole persistent winning task, without assigning a common cause to the persistent losses\.

Held\-out selection provides no clear positive evidence beyond clones\. The clone\-adjusted mean isD=−0\.26D=\-0\.26pp \(Table[2](https://arxiv.org/html/2609.35873#S5.T2)\), with randomizationpp\-valuep=0\.8127p=0\.8127\. Its interval includes zero and the positive\-support conditions are not met, givingInsufficient Evidencerather than an equivalence conclusion\. Excluding the three tasks containing conflicting scores givesD=−0\.61D=\-0\.61pp on383383paired tasks and the same decision \(Appendix[C](https://arxiv.org/html/2609.35873#A3)\)\.

#### Three repeats have limited power for crossovers\.

In the two\-specialist setting described in §[4\.2](https://arxiv.org/html/2609.35873#S4.SS2), the simulated diagnostic has0%0\\%power atR=3R=3and100%100\\%atR=10,20R=10,20, despite trueHstable=17\.5H\_\{\\rm stable\}=17\.5pp\. These are observed detection rates over300300simulated datasets atR=3R=3and150150at each higher repeat count\. Smaller cyclic crossovers remain largely undetected even atR=20R=20\(Appendix Table[8](https://arxiv.org/html/2609.35873#A4.T8)\)\. Increasing repeats therefore helps in the strong positive case without ensuring sensitivity to every crossover\. The result characterizes this diagnostic and these settings, rather than all procedures using three repeats\. It explains why the empirical nonpositive result leaves stable complementarity unresolved\. We next evaluate the tested pre\-execution decision rule directly, whose realized gain is a separate question\.

### 5\.4The Tested Selector Adds No Gain and Full\-Budget Coverage Ties

#### The frozen selector matches the development\-selected baseline\.

The tested feature policy yields0\.000\.00pp gain\. Its frozen classifier directly selects single\-call on every evaluation task; the confidence\-based fallback is never triggered\. These choices give95\.85%95\.85\\%accuracy over three repeats, identical to the development\-selected fixed member\. Appendix[E](https://arxiv.org/html/2609.35873#A5)gives its construction and the same\-set comparator results\.

#### At the full execution budget, the oracle coverages tie\.

Atb=27b=27executions, the development\-ranked portfolio and same\-code controls both reach98\.70%98\.70\\%oracle coverage \(Table[3](https://arxiv.org/html/2609.35873#S5.T3)\)\. Atb=9b=9and1515, the portfolio\-minus\-clone differences are\+0\.08\+0\.08pp and\+0\.02\+0\.02pp\. Atb=3b=3, the portfolio contains only single\-call, so the\+0\.20\+0\.20pp difference reflects execution variation between identical programs\.

Harness executions per taskbb391527Panel members, including single\-call1359Panel oracle coverage \(%\)98\.4598\.7098\.7098\.70Clone oracle coverage \(%\)98\.2498\.6398\.6998\.70Panel minus clones \(pp\)\+0\.20\+0\.20\+0\.08\+0\.08\+0\.02\+0\.02\+0\.00\+0\.00
Differences are computed before rounding the displayed coverage values\.

Table 3:Full\-budget oracle coverage ties in the recorded executions\.MATH\-500 coverage from the development\-ranked panel and uniform same\-code sampling\. Clone values are exact subset means \(§[4\.3](https://arxiv.org/html/2609.35873#S4.SS3)\)\. Budgets count harness executions; model\-call and token costs vary by program\.Together, the selector and replay results separate two operational questions: the tested pre\-execution policy yields zero gain, while post\-execution oracle coverage ties at the full recorded budget\.

### 5\.5BIRD Traces Locate Breaks Between Code and Execution

Source variation can disappear before it produces a valid final answer\. The supporting BIRD study follows this process directly in independent program populations, using its task\-specific judging and protocols\. Here a mechanism is an implemented prompt transformation, tool use, or control\-flow operation\. This execution view complements the MATH controls by examining what the programs actually do\.

#### Different source code often produces identical scored outcomes\.

All1212discovery candidates pass a code\-difference check, yet21/6621/66generated\-candidate pairs have identical correctness vectors \(36/9136/91including baselines\)\. Source similarity has essentially no Spearman rank correlation with correctness disagreement \(ρ=−0\.010\\rho=\-0\.010\)\. This agreement on6060tasks motivates examining the traces \(Appendix Figure[2](https://arxiv.org/html/2609.35873#A6.F2)\)\.

#### Traces distinguish absent, inactive, and broken mechanisms\.

T1: described but absent\.Six of twelve candidates describe retrieval or self\-checking but make a single model call without an implemented retrieval or self\-checking step\. This classification covers all twelve candidates; the five\-harness trace audit supplies worked examples\.

T2: present but untriggered\.For the execution\-repair baseline, the first SQL executes successfully on all ten audited tasks, so the repair branch never fires and final SQL equals single\-call\. A successful execution can therefore leave a semantically wrong answer unchanged\.

T3: executed but broken\.One candidate runs multi\-turn repair but returns prose as final SQL, losing tasks that the baseline answers correctly\. The three cases locate distinct checks: implementation, branch activation, and valid output\. Appendix[F](https://arxiv.org/html/2609.35873#A6)provides worked traces and hand\-written controls, with their development\-exposure limitations\.

The separate BIRD admission study tests whether prescribing strategies and screening programs increases headroom\. Its implemented gate combines structural exclusion and conformance verification\. Forced/gated minus free/ungated generation gives−0\.16\-0\.16pp, with no increase in observed mean headroom \(Appendix[G](https://arxiv.org/html/2609.35873#A7)\)\. The admission comparison measures what screening changes; the traces locate where proposed operations fail to deliver useful answers\. Together, these BIRD studies locate execution failures that complement the MATH evidence about persistence and selection\.

## 6Discussion and Limitations

A program population can provide additional answers and highly repeatable score patterns while leaving useful specialization unestablished\. Our same\-code control exposes the first gap: byte\-identical programs have zero stable complementarity by construction, yet produce positive oracle headroom even after averaging repeats\. This headroom captures a real opportunity to obtain correct answers\. Attributing that opportunity to program identity, however, requires comparison with what further executions of a fixed program already provide\. The central evaluation question is therefore what changing programs adds when additional sampling is also available\.

The generated panel’s strong residual correlation chiefly identifies recurrent weaknesses in the recorded scores\. Equation[2](https://arxiv.org/html/2609.35873#S3.E2)clarifies why the direction matters: variation in the size of a disadvantage can be highly repeatable without changing which member is preferable\. Repeatability is informative about where a population fails; demonstrating a division of labor additionally requires different members to retain advantages on different tasks\.

#### Mechanisms should target identifiable failures\.

The separate BIRD traces make the design problem concrete\. An operation must be implemented, activated on the relevant input, and produce a valid answer before it can contribute to performance\. The inactive repair branch illustrates a mismatch between a mechanism and the failure it needs to address: SQL can execute successfully while answering the question incorrectly, leaving an execution\-error trigger unable to address that failure\. Designing a mechanism therefore also means specifying the task conditions under which it should change an answer\. A targeted follow\-up would compare enabled and disabled versions of the same mechanism on tasks that exercise its intended trigger, then test any advantage on fresh executions\. Such interventions would connect intended behavior to a causal contribution and test whether that contribution persists\.

#### Population value depends on a usable decision\.

For pre\-execution routing, useful task advantages must be recognizable from information available before the answer is known\. A portfolio that runs several programs instead needs an answer\-selection or aggregation procedure that can turn its outputs into a better final answer\. Our frozen selector and oracle replay address different parts of this problem: the former retains single\-call on every task, while the latter measures coverage after correctness is known\. Their respective zero gain and full\-budget coverage tie make additional fixed\-program executions an essential comparator\. Future evaluations should combine this comparator with repeat counts calibrated to detect task\-wise crossovers and an implementable decision rule under a matched resource budget\. This would establish whether the advantages associated with program identity improve inference beyond the sampling opportunities a larger population creates\.

#### Limitations and unresolved questions\.

The MATH evidence concerns one solver, a selected low\-call panel, and three repeats\. Known\-truth calibration shows that this repeat count can miss substantial task\-wise crossovers for the tested diagnostic\. TheInsufficient Evidencedecision leaves stable complementarity unresolved and does not establish equivalence\. The repeat diagnostic and common\-complete\-set analysis were developed after collection; recovery retained the programs, tasks, solver, and repeat count\. The1414excluded tasks are harder than the386386included tasks: baseline repeat\-mean accuracy is71\.43%71\.43\\%on excluded tasks and95\.85%95\.85\\%on included tasks \(Appendix[C](https://arxiv.org/html/2609.35873#A3)\)\. Recorded scores include answer\-format effects\. The selector result concerns the tested policy and features\. Replay matches harness executions, with model calls and tokens varying by program, so the observed tie does not establish equal\-cost utility\. BIRD uses separate populations and scoring conditions; its traces identify execution failures without establishing the causes of the MATH losses\.

## 7Conclusion

The value of a harness population depends on how its differences improve inference\. Our same\-code control and repeated executions show why coverage, repeatability, and usable specialization require separate evidence: identical programs yield positive oracle headroom, while generated programs exhibit persistent differences dominated by losses\. On the tested MATH panel, the frozen selector gains0\.000\.00pp, and repeated baseline executions match the portfolio’s98\.70%98\.70\\%oracle coverage at2727harness executions\. Stable complementarity remains unresolved at three repeats\.

These findings give automated harness design a concrete objective: develop reproducible task advantages that a usable decision rule can exploit\. To justify additional programs through specialization, evaluation must show what those decisions gain over spending the relevant inference budget on further executions of a fixed program\.

### Ethics Statement

All experiments use public BIRD and MATH\-500 benchmark data and generated program artifacts; no human participants, private records, or personally identifiable information are collected\. Model calls are reported as part of the experimental provenance, and the paper does not release credentials or provider responses that would expose private data\. The results distinguish recorded answer coverage from the performance of a selector operating before execution\.

### Reproducibility Statement

The manuscript specifies the populations, task splits, execution conditions, scoring rules, statistical procedures, and sensitivity analyses used for the reported results\. The appendices provide additional protocol details, calibration results, and worked examples\.

### AI use statement

Generative AI tools assisted with research questions, experimental and statistical design, counterexample analysis, code development, model\-API workflows, data analysis, figures, and manuscript preparation\. Versioned artifacts record the analyses and revisions; automated tests, local runtime checks, and same\-model subagents supported mechanical verification\. The human authors set the research direction and retain responsibility for scientific decisions, verification, the final content, and submission\.

## References

- Budd and Angluin \(1982\)T\. A\. Budd and D\. AngluinTwo notions of correctness and their relation to testing\.Acta Informatica18\(1\),pp\. 31–45\.External Links:[Document](https://dx.doi.org/10.1007/bf00625279)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px1.p1.1)\.
- Caruanaet al\.\(2004\)R\. Caruana, A\. Niculescu\-Mizil, G\. Crew, and A\. KsikesEnsemble selection from libraries of models\.InProceedings of the 21st International Conference on Machine Learning,External Links:[Document](https://dx.doi.org/10.1145/1015330.1015432)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2107.03374),[Link](https://arxiv.org/abs/2107.03374)Cited by:[§1](https://arxiv.org/html/2609.35873#S1.p2.1),[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2026a\)M\. Chen, J\. Wang, Z\. Liu, Y\. Wang, H\. Zheng, and Q\. WangFrom failed trajectories to reliable LLM agents: diagnosing and repairing harness flaws\.arXiv preprint arXiv:2606\.06324\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2606.06324),[Link](https://arxiv.org/abs/2606.06324)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2026b\)M\. Chen, C\. Lv, G\. Zhang, H\. Chang, and S\. ZhouHarnessForge: joint harness and policy evolution for adaptive agent systems\.arXiv preprint arXiv:2606\.01779\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2606.01779),[Link](https://arxiv.org/abs/2606.01779)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px1.p1.1)\.
- Cho \(2026\)Y\. ChoIt’s not the capability: harness sensitivity is non\-monotone across LLM agent tiers\.arXiv preprint arXiv:2605\.26731\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2605.26731),[Link](https://arxiv.org/abs/2605.26731)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px3.p1.1)\.
- Elet al\.\(2025\)B\. El, M\. Yuksekgonul, and J\. ZouInefficiencies of meta agents for agent design\.arXiv preprint arXiv:2510\.06711\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2510.06711),[Link](https://arxiv.org/abs/2510.06711)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px1.p1.1)\.
- Fan and Huang \(2026\)T\. Fan and C\. HuangHELIX: model\-harness co\-evolution for recursive self\-improvement\.arXiv preprint arXiv:2608\.13951\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2608.13951),[Link](https://arxiv.org/abs/2608.13951)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px3.p1.1)\.
- Gail and Simon \(1985\)M\. H\. Gail and R\. SimonTesting for qualitative interactions between treatment effects and patient subsets\.Biometrics41\(2\),pp\. 361–372\.External Links:[Document](https://dx.doi.org/10.2307/2530862)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px3.p1.1)\.
- Guptaet al\.\(2026\)A\. Gupta, J\. Lei, A\. Lu, G\. Anumanchipalli, and L\. ChoshenAutomated discovery has no universally superior harness\.arXiv preprint arXiv:2607\.18235\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2607.18235),[Link](https://arxiv.org/abs/2607.18235)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px3.p1.1)\.
- Huet al\.\(2024\)S\. Hu, C\. Lu, and J\. CluneAutomated design of agentic systems\.arXiv preprint arXiv:2408\.08435\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2408.08435),[Link](https://arxiv.org/abs/2408.08435)Cited by:[§1](https://arxiv.org/html/2609.35873#S1.p1.1),[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px1.p1.1)\.
- Kim \(2026\)M\. KimWhen should LLMs search? counterfactual supervision for search routing\.arXiv preprint arXiv:2607\.05752\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2607.05752),[Link](https://arxiv.org/abs/2607.05752)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px3.p1.1)\.
- Kotthoff \(2014\)L\. KotthoffAlgorithm selection for combinatorial search problems: a survey\.AI Magazine35\(3\),pp\. 48–60\.External Links:[Document](https://dx.doi.org/10.1609/aimag.v35i3.2460)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px3.p1.1)\.
- Kwoket al\.\(2026\)J\. Kwok, S\. Li, P\. Atreya, Y\. Liu, Y\. Jiang, C\. Finn, M\. Pavone, I\. Stoica, and A\. MirhoseiniLLM\-as\-a\-verifier: a general\-purpose verification framework\.arXiv preprint arXiv:2607\.05391\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2607.05391),[Link](https://arxiv.org/abs/2607.05391)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2023\)J\. Li, B\. Hui, G\. Qu, J\. Yang, B\. Li, B\. Li, B\. Wang, B\. Qin, R\. Geng, N\. Huo, X\. Zhou, C\. Ma, G\. Li, K\. Chang, F\. Huang, R\. Cheng, and Y\. LiCan LLM already serve as a database interface? a BIg bench for large\-scale database grounded text\-to\-SQLs\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track,Vol\.36,pp\. 42330–42357\.External Links:[Document](https://dx.doi.org/10.52202/075280-1835),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/83fc8fab1710363050bbd1d4b8cc0021-Abstract-Datasets_and_Benchmarks.html)Cited by:[§4\.1](https://arxiv.org/html/2609.35873#S4.SS1.SSS0.Px3.p1.1)\.
- Liet al\.\(2024\)J\. Li, Q\. Zhang, Y\. Yu, Q\. Fu, and D\. YeMore agents is all you need\.arXiv preprint arXiv:2402\.05120\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2402.05120),[Link](https://arxiv.org/abs/2402.05120)Cited by:[§1](https://arxiv.org/html/2609.35873#S1.p2.1),[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px2.p1.1)\.
- Lightmanet al\.\(2023\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.arXiv preprint arXiv:2305\.20050\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2305.20050),[Link](https://arxiv.org/abs/2305.20050)Cited by:[§4\.1](https://arxiv.org/html/2609.35873#S4.SS1.SSS0.Px1.p1.1)\.
- Linet al\.\(2026\)J\. Lin, S\. Liu, C\. Pan, L\. Lin, S\. Dou, Z\. Xi, X\. Huang, H\. Yan, Z\. Han, T\. Gui, and Y\. JiangAgentic harness engineering: observability\-driven automatic evolution of coding\-agent harnesses\.arXiv preprint arXiv:2604\.25850\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2604.25850),[Link](https://arxiv.org/abs/2604.25850)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px1.p1.1)\.
- Luoet al\.\(2026\)X\. Luo, D\. Xue, F\. Wang, C\. Hu, and Y\. DengHarnessBank: semantic gene\-bank search with gated verification for agent\-harness self\-evolution\.arXiv preprint arXiv:2607\.13683\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2607.13683),[Link](https://arxiv.org/abs/2607.13683)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px1.p1.1)\.
- Miyaiet al\.\(2026\)A\. Miyai, K\. Aizawa, and T\. YamasakiTask\-CoEvolve: efficient harness optimization via adaptive validation task selection\.arXiv preprint arXiv:2608\.20169\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2608.20169),[Link](https://arxiv.org/abs/2608.20169)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px1.p1.1)\.
- Mollet al\.\(2026\)J\. Moll, J\. Corbeil, J\. Pan, M\. Hadamitzky, D\. Rueckert, L\. Adams, and K\. BressemGRASP: gated regression\-aware skill proposer for self\-improving LLM agents\.arXiv preprint arXiv:2605\.29668\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2605.29668),[Link](https://arxiv.org/abs/2605.29668)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px3.p1.1)\.
- Nieet al\.\(2026\)J\. Nie, Y\. Zhang, J\. Song, Q\. Cai, D\. Yu, Y\. Guo, X\. Tian, and B\. HanTTHE: test\-time harness evolution\.arXiv preprint arXiv:2607\.08124\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2607.08124),[Link](https://arxiv.org/abs/2607.08124)Cited by:[§1](https://arxiv.org/html/2609.35873#S1.p1.1),[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px1.p1.1)\.
- Onget al\.\(2024\)I\. Ong, A\. Almahairi, V\. Wu, W\. Chiang, T\. Wu, J\. E\. Gonzalez, M\. W\. Kadous, and I\. StoicaRouteLLM: learning to route llms with preference data\.arXiv preprint arXiv:2406\.18665\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2406.18665),[Link](https://arxiv.org/abs/2406.18665)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px3.p1.1)\.
- Ouyanget al\.\(2023\)S\. Ouyang, J\. M\. Zhang, M\. Harman, and M\. WangAn empirical study of the non\-determinism of ChatGPT in code generation\.arXiv preprint arXiv:2308\.02828\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2308.02828),[Link](https://arxiv.org/abs/2308.02828)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px2.p1.1)\.
- Papadakiset al\.\(2019\)M\. Papadakis, M\. Kintis, J\. Zhang, Y\. Jia, Y\. Le Traon, and M\. HarmanMutation testing advances: an analysis and survey\.InAdvances in Computers,Vol\.112,pp\. 275–378\.External Links:[Document](https://dx.doi.org/10.1016/bs.adcom.2018.03.015)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px1.p1.1)\.
- Rice \(1976\)J\. R\. RiceThe algorithm selection problem\.InAdvances in Computers,Vol\.15,pp\. 65–118\.External Links:[Document](https://dx.doi.org/10.1016/s0065-2458%2808%2960520-3)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px3.p1.1)\.
- Wanget al\.\(2026\)R\. Wang, Y\. Shi, Z\. Li, Z\. Li, Y\. Yu, J\. Yang, K\. Panaganti, H\. Mi, D\. Zhou, and LeoweiliangHarness handbook: making evolving agent harnesses readable, navigable, and editable\.arXiv preprint arXiv:2607\.13285\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2607.13285),[Link](https://arxiv.org/abs/2607.13285)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2022\)X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. ZhouSelf\-consistency improves chain of thought reasoning in language models\.arXiv preprint arXiv:2203\.11171\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2203.11171),[Link](https://arxiv.org/abs/2203.11171)Cited by:[§1](https://arxiv.org/html/2609.35873#S1.p2.1),[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px2.p1.1)\.
- Xuet al\.\(2026\)J\. Xu, Y\. Zhang, A\. Chen, W\. Li, J\. Liang, and D\. YangVerify smarter, evolve further: efficient harness evolution through behavior\-aware verification\.arXiv preprint arXiv:2608\.27311\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2608.27311),[Link](https://arxiv.org/abs/2608.27311)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px1.p1.1)\.
- Xuet al\.\(2008\)L\. Xu, F\. Hutter, H\. H\. Hoos, and K\. Leyton\-BrownSATzilla: portfolio\-based algorithm selection for SAT\.Journal of Artificial Intelligence Research32,pp\. 565–606\.External Links:[Document](https://dx.doi.org/10.1613/jair.2490)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px3.p1.1)\.
- Zhanget al\.\(2026a\)G\. Zhang, L\. Lu, F\. Xie, K\. Zhu, J\. Wang, Z\. Xie, Z\. Yu, Z\. Liu, Z\. Sun, Q\. Li, Y\. Liao, H\. Chang, X\. Hu, Q\. Ren, W\. Zhou, C\. Hu, Y\. Deng, and S\. YanJIT\-Agent: scaling harness intelligence via just\-in\-time harness evolution\.arXiv preprint arXiv:2608\.25593\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2608.25593),[Link](https://arxiv.org/abs/2608.25593)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026b\)H\. Zhang, S\. Zhang, K\. Li, C\. Zhang, Y\. Chen, Y\. Zhang, L\. Bai, and S\. HuSelf\-harness: harnesses that improve themselves\.arXiv preprint arXiv:2606\.09498\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2606.09498),[Link](https://arxiv.org/abs/2606.09498)Cited by:[§1](https://arxiv.org/html/2609.35873#S1.p1.1),[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px1.p1.1)\.
- Zhouet al\.\(2026\)H\. Zhou, Z\. Tan, Z\. Zhang, Y\. Fan, Y\. Lin, L\. Kang, X\. Song, R\. Li, S\. Huang, A\. Yu, Y\. Fan, Y\. Chen, K\. Xu, X\. Liu, Y\. Qin, P\. Torr, C\. Zhang, and Z\. YinSelect\-then\-solve: paradigm routing as inference\-time optimization for LLM agents\.arXiv preprint arXiv:2604\.06753\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2604.06753),[Link](https://arxiv.org/abs/2604.06753)Cited by:[§2](https://arxiv.org/html/2609.35873#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix Guide

The appendices document program generation, repeated execution, selection, calibration, and the supporting BIRD studies\.

[A](https://arxiv.org/html/2609.35873#A1)[B](https://arxiv.org/html/2609.35873#A2)[C](https://arxiv.org/html/2609.35873#A3)[D](https://arxiv.org/html/2609.35873#A4)[E](https://arxiv.org/html/2609.35873#A5)[F](https://arxiv.org/html/2609.35873#A6)[G](https://arxiv.org/html/2609.35873#A7)

## Appendix ANotation Summary

Table[4](https://arxiv.org/html/2609.35873#A1.T4)collects the notation used in the main text and the analysis appendices\. Mathematical scores and gains use proportions; reported accuracies use % and reported gains use percentage points \(pp\)\. Thus a gain of0\.010\.01is11pp\. Population expectations, finite\-sample estimates, and held\-out gains are distinct quantities\.

Table 4:Symbols, definitions, and scope\.NotationMeaningMembers, tasks, and outcomes \(§[3](https://arxiv.org/html/2609.35873#S3)\)PP,hh,MMHarness population, an indexed memberh∈Ph\\in P, and population sizeM=\|P\|M=\|P\|\. Identical clone slots have distinct member identities\.H0H\_\{0\},H1H\_\{1\}Single\-call baseline and the other member in the two\-member identity\.xx,xix\_\{i\},ii,nnA task, evaluated taskii, its index, and the number of evaluated tasks\.rr,RRExecution\-repeat index and number of repeats per member–task pair; empirical repeat comparisons useR=3R=3\.Y⁡\(h,x,r\)Y\(h,x,r\)Binary correctness of memberhhon taskxxin repeatrr\.ph​\(x\)p\_\{h\}\(x\);p0​\(x\),p1​\(x\)p\_\{0\}\(x\),p\_\{1\}\(x\)Success probability over fresh executions at fixed conditions\. For two members,pj​\(x\)=pHj​\(x\)p\_\{j\}\(x\)=p\_\{H\_\{j\}\}\(x\),j∈\{0,1\}j\\in\\\{0,1\\\}\.p^h​\(x\)\\hat\{p\}\_\{h\}\(x\)Complete\-repeat estimateR−1​∑r=1RY⁡\(h,x,r\)R^\{\-1\}\\sum\_\{r=1\}^\{R\}Y\(h,x,r\)\.𝔼x\\mathbb\{E\}\_\{x\},Pr\\PrExpectation over the target task distribution and probability\. In the Jensen argument, unsubscripted𝔼\\mathbb\{E\}instead averages execution sampling at a fixed task\.H⁡\(P\)H\(P\)Single\-run oracle accuracy minus the best fixed member’s accuracy on the same tasks and repeat \(Equation[1](https://arxiv.org/html/2609.35873#S3.E1)\)\.H^\\widehat\{H\}Repeat\-averaged plug\-in headroom: replaceYYinH⁡\(P\)H\(P\)byp^h​\(x\)\\hat\{p\}\_\{h\}\(x\)before taking either maximum\.HstableH\_\{\\rm stable\}Population headroom computed from the success probabilitiesph​\(x\)p\_\{h\}\(x\)\.Δ⁡\(x\)\\Delta\(x\)Two\-member success\-probability differencep1​\(x\)−p0​\(x\)p\_\{1\}\(x\)\-p\_\{0\}\(x\)\(Equation[2](https://arxiv.org/html/2609.35873#S3.E2)\)\.Repeat diagnostics, selection, and budgetsGG;Greal,GcloneG\_\{\\rm real\},G\_\{\\rm clone\}Mean held\-out accuracy gain of the discovery\-selected task\-wise mapping over the discovery\-selected fixed member; subscripts identify the generated\-plus\-baseline and same\-code arms \(§[4\.2](https://arxiv.org/html/2609.35873#S4.SS2)\)\.DDPaired clone\-adjusted gainGreal−GcloneG\_\{\\rm real\}\-G\_\{\\rm clone\}, a finite\-repeat diagnostic \(Equation[3](https://arxiv.org/html/2609.35873#S4.E3)\)\.ρres\\rho\_\{\\rm res\}Mean Pearson correlation of residual score patterns between each held\-out repeat and the other repeats’ mean \(Appendix[C\.1](https://arxiv.org/html/2609.35873#A3.SS1)\); labeled residualrrin Figure[1](https://arxiv.org/html/2609.35873#S1.F1)\.ρ\\rhoBIRD Spearman rank correlation between source similarity and correctness disagreement \(§[5\.5](https://arxiv.org/html/2609.35873#S5.SS5)\)\.pp; CIRandomizationpp\-value for the MATH diagnostic; confidence interval\. A scalarpp\-value is distinct from success probabilityph​\(x\)p\_\{h\}\(x\)\.Z⁡\(x\)Z\(x\),πZ​\(x\)\\pi\_\{Z\}\(x\)Pre\-execution task features and the member selected from them; the implemented features use question text and training\-fold member accuracies\.hdev​\-​fixedh\_\{\\rm dev\\text\{\-\}fixed\}Fixed member chosen by development accuracy, before evaluation\.S⁡\(πZ\)S\(\\pi\_\{Z\}\)Evaluation\-task mean of repeat\-mean accuracy differences between the frozen feature selector andhdev​\-​fixedh\_\{\\rm dev\\text\{\-\}fixed\}\(Appendix[E](https://arxiv.org/html/2609.35873#A5)\)\.bbNumber of harness executions per task in oracle replay,b∈\{3,9,15,27\}b\\in\\\{3,9,15,27\\\}; distinct from model\-call and token counts\.kkin Pass@kkNumber of sampled answers; Pass@kkmeasures the probability that at least one is correct\.Additional notation in the analysis appendicesh0h\_\{0\},Ar​\(h,x\)A\_\{r\}\(h,x\)Arm\-specific reference \(single\-call or clone\-c1\) and baseline\-relative scoreY⁡\(h,x,r\)−Y⁡\(h0,x,r\)Y\(h,x,r\)\-Y\(h\_\{0\},x,r\)\(Appendix[C\.1](https://arxiv.org/html/2609.35873#A3.SS1)\)\.TTNumber of simulated tasks in calibration;MMandRRretain their member\-count and repeat\-count meanings \(Appendix[D](https://arxiv.org/html/2609.35873#A4)\)\.U⁡\(0,10−9\)U\(0,10^\{\-9\}\)Uniform distribution used for independent random tie\-breaking jitter\.θ∗\\theta^\{\*\}A given selector’s population gain over the population\-best fixed member; bounded above byHstableH\_\{\\rm stable\}\.ℛh​x\\mathcal\{R\}\_\{hx\},Y¯h​\(x\)\\bar\{Y\}\_\{h\}\(x\)Observed development\-repeat indices and their mean score\. In evaluation,Y¯h​\(x\)\\bar\{Y\}\_\{h\}\(x\)averages all three repeats \(Appendix[E](https://arxiv.org/html/2609.35873#A5)\)\.y⁡\(x\)y\(x\)Member\-ID training label maximizingY¯h​\(x\)\\bar\{Y\}\_\{h\}\(x\); distinct from binary correctnessY⁡\(h,x,r\)Y\(h,x,r\)\.CCInverse regularization strength of the logistic classifier, fixed at11\.KK,kk,qk​\(h∣x\)q\_\{k\}\(h\\mid x\)Number of fitted fold models \(1≤K≤51\\leq K\\leq 5\), model index, and modelkk’s predicted member\-class probability\. Herekkindexes models, rather than answer samples\.q¯​\(h∣x\)\\bar\{q\}\(h\\mid x\);h\(1\),h\(2\)h\_\{\(1\)\},h\_\{\(2\)\}Mean of fitted models’ class probabilities; the two highest\-probability members\. A gap below0\.150\.15triggers baseline fallback\.ff,\(fb\)\\binom\{f\}\{b\}Incorrect clone executions among2727for a task, and the number of size\-bbsubsets containing only incorrect executions; zero whenb\>fb\>f\.KcandK\_\{\\rm cand\},KtotalK\_\{\\rm total\}Admitted BIRD candidate count without and with the baseline:Ktotal=Kcand\+1K\_\{\\rm total\}=K\_\{\\rm cand\}\+1\. K\-matching matches candidate counts \(Appendix[G](https://arxiv.org/html/2609.35873#A7)\)\.II\-A–II\-D; A–DBIRD free/ungated, free/gated, forced/ungated, and forced/gated arms\. A–D denote arm headrooms in factorial formulas; their D−\-A contrast differs from the MATH statisticDD\.
## Appendix BHarness Generation, Program Inventory, and Data Splits

This appendix describes the MATH population, splits, frozen panel, and model versions\. BIRD protocols appear in Appendices[F](https://arxiv.org/html/2609.35873#A6)and[G](https://arxiv.org/html/2609.35873#A7)\.

#### Execution interface\.

Harnesses implementsolve\(question\)and access the frozen solver throughself\.llm\. BIRD additionally provides the database schema asself\.schema;self\.execute\(sql\)returns query results or execution errors\. Gold answers remain external to these interfaces\. The execution\-repair baseline is labeledreactin archived programs and traces\.

### B\.1MATH population, common analysis set, and coverage

The single\-run study supplies coverage measurements and programs for the repeat panel, applying BIRD’s free, ungated generation procedure to competition mathematics with a domain\-specific comparator\.

#### Protocol and analysis set\.

The builder instruction, summarized rather than quoted, requests a Pythonsolve\(question\)harness that improves on one greedy call throughself\.llm, with freely chosen prompting, decoding, sampling, and control flow and a final\#\#\#\#answer\. Gold answers remain external\. Slots receive no assigned mechanism or diversity target\. Attempts reuse the instruction without prior code or performance feedback; admission requires a valid interface and nonempty output\.

Six builders each receive eight slots under one seed, with at most three attempts per slot\. Parsing, import, interface, and nonempty answers are checked on two development questions\. The target is GLM\-5\.3\-Flash, with member\-specific decoding and temperature00for single\-call\. The pre\-generation split reserves100100development problems\. Single\-run, repeat, and policy analyses share386386fully observed tasks; the post\-collection completeness rule excludes1414problems \(Appendix[C](https://arxiv.org/html/2609.35873#A3)\)\. Development tasks and the selector stay fixed\.

The judge compares parsed numbers, including fractions and decimals, or otherwise normalized LaTeX\. Normalization removes units, base subscripts, and degree symbols, retains text\-command content, and maps matrices to tuples\. Gold answers prefixed with\#\#\#\#pass386/386386/386self\-comparisons\. Single\-run baseline accuracy on this set is95\.08%95\.08\\%\.

The gate admits35/4835/48slots\. GLM\-5\.3 and Qwen3\.8\-Max exhaust the3232k\-token generation budget before emitting code in most slots, leaving their styles under\-represented in this fixed\-budget population\.

#### Results on the common complete set\.

All3636members, including single\-call, have one score per task:13,896/13,89613\{,\}896/13\{,\}896cells over386386tasks\. Mean pairwise disagreement is10\.94%10\.94\\%\(maximum33\.16%33\.16\\%\), with29/3629/36unique outcome vectors \(28/3528/35generated\) and2\.22%2\.22\\%identical pairs\. The generated union repairs84\.21%84\.21\\%of baseline errors; all generated members fail on0\.82%0\.82\\%of baseline\-correct tasks\. Oracle coverage of99\.22%99\.22\\%exceeds the best fixed member single\-call \(95\.08%95\.08\\%\) by4\.154\.15pp\. Generated accuracies span69\.69%69\.69\\%–92\.49%92\.49\\%\.

The repeat panel compares eight generated members plus single\-call with nine same\-code slots on identical tasks\. Its held\-out diagnostic concerns this panel; single\-run headroom concerns the full population\. Both the archived judge and the versioned endpoint\-sensitive judge reproduce all13,89613\{,\}896included scores\.

#### Scope\.

These measurements describe one seed and one target on the completeness\-selected MATH set; BIRD uses the separate protocols below\.

### B\.2Selected MATH Program Inventory

Table[5](https://arxiv.org/html/2609.35873#A2.T5)describes the nine frozen members selected for the repeat study\. H1–H8 are neutral labels for generated members, all constructed under seed00\. The listed operations describe what each program implements, not whether every branch fires or improves correctness\. The mean\-call eligibility ceiling is not a per\-execution maximum, and one logical call may request several samples\.

Table 5:What was selected for the repeat study\.All calls use the same frozen solver; verification and adjudication are solver prompts, not access to benchmark labels\. Six seats are builder\-stratified; H5 \(Kimi\) and H7 \(MiniMax\) are the two at\-large seats\.
### B\.3Models and versions

Table[6](https://arxiv.org/html/2609.35873#A2.T6)gives the exact API model identifiers\. All roles use one commercial OpenAI\-compatible aggregator\. Target identity is fixed; decoding follows each harness, with temperature00for the single\-call baseline\. Phase\-II generation uses temperature 0\.7 and a 16,384\-token budget; Phase\-I conditions appear in Appendix[F\.1](https://arxiv.org/html/2609.35873#A6.SS1)\. Phase I uses a proxy execution judge, Phase II the official BIRD scorer \(30 s timeout\) with a secondary legacy\-judge audit, and MATH\-500 the answer comparator in Appendix[B\.1](https://arxiv.org/html/2609.35873#A2.SS1)\. Reported accuracy uses no LLM judge\.

Table 6:Model inventory for the reported benchmark studies\. The target is held frozen within each phase; Phase\-II builders come from six independent labs\.

## Appendix CSame\-Code Controls and Repeated\-Execution Protocol

All MATH primary estimates use the same386386fully observed tasks\. The protocol below specifies the panel, inclusion rule, and repeat analysis\.

#### Panel and inclusion\.

Programs, solver, judge, development split, and three repeat identities are fixed\. A salted hash selects one low\-call member per builder plus two at\-large members and single\-call; the control contains nine identical baseline slots\. The September 26, 2026, 05:15 UTC snapshot combines generated continuation 10 and clone continuation 20\. After outcome inspection, we retained tasks with a recorded binary score in every required single\-run and repeat cell\. These386386task identities define the common reanalysis set\.

#### Completeness and exclusions\.

The single\-run matrix contains36×386=13,89636\\times 386=13\{,\}896observed cells and each repeat arm9×3×386=10,4229\\times 3\\times 386=10\{,\}422, giving100%100\\%completeness for all three matrices\. Policy evaluation and oracle replay use the same task identities\.

The1414excluded problems contain6161unscored generated cells, distributed as DeepSeek 5, ERNIE 4, GLM 9, Kimi 30, MiniMax 9, and Qwen 4; baseline and clone records are complete\. Their pre\-subsetting fraction,3\.50%3\.50\\%, is below the frozen10%10\\%gate\. Excluded tasks are harder and longer, with lower baseline accuracy \(Table[7](https://arxiv.org/html/2609.35873#A3.T7)\)\. Unknown scores and all incurred acquisition costs remain in the source records\.

Table 7:Observed characteristics of included and excluded tasks\.Excluded tasks have higher benchmark difficulty, longer questions, and lower baseline accuracy\.
#### Provenance and conflicts\.

Manifests are checked on 13 fields covering execution conditions, programs, task/split hashes, panel draw, and schedule; snapshots, matrices, and task lists are hash\-bound\. Completed scores replace unknowns, with first\-completed precedence on disagreement\. Five generated cell identities conflict, including three on three retained tasks; clones have none\. Collector and concurrency changes are logged\.

#### Estimates and sensitivity\.

Recorded binary scores give plug\-in headroom of2\.332\.33pp \(generated\) and2\.162\.16pp \(clone\), andD=−0\.26D=\-0\.26pp \(95% interval\[−1\.45,0\.95\]\[\-1\.45,0\.95\]pp;Insufficient Evidence\)\. Removing the three conflict\-affected tasks from both arms and reselecting on the remaining383383tasks givesD=−0\.61D=\-0\.61pp \(interval\[−1\.96,0\.64\]\[\-1\.96,0\.64\]pp\), with the same decision\.

### C\.1Exploratory Repeatability and Score Sensitivity

For non\-reference members, defineAr​\(h,x\)=Y⁡\(h,x,r\)−Y⁡\(h0,x,r\)A\_\{r\}\(h,x\)=Y\(h,x,r\)\-Y\(h\_\{0\},x,r\), whereh0h\_\{0\}is single\-call or clone\-c1\. Within each repeat, subtract row and column means and add the grand mean\. Correlate each held\-out residual matrix with the residual of the other two repeats’ mean, then average the three Pearson correlations to obtainρres\\rho\_\{\\rm res\}\. A paired task bootstrap uses2,0002\{,\}000shared draws\. Correlations are0\.7890\.789\(\[0\.75,0\.83\]\[0\.75,0\.83\]\) for generated members and0\.0060\.006\(\[−0\.07,0\.08\]\[\-0\.07,0\.08\]\) for clones, with difference0\.7830\.783\(\[0\.69,0\.87\]\[0\.69,0\.87\]\)\. This post\-collection diagnostic measures repeatable score patterns after removing additive effects\.

Losses in all three repeats span570570member–task pairs on100100tasks\. Wins in all three comprise eight pairs on one task, shared by all eight generated programs\. That task is format\-sensitive: the gold answer is12, baseline outputs end with\#\#\#\# 12th grade, and a generated member returns12th grade\. The extractor retains the former’s complete\#\#\#\#payload and scores it zero, but uses numeric fallback for the latter and scores it one\. Recorded scores are retained; the repeated repair here arises from answer formatting\.

## Appendix DStatistical Analysis, Calibration, and Sensitivity

Empirical diagnostics use386386complete MATH tasks\. Synthetic calibration uses the sample sizes specified below\.

### D\.1Held\-out\-repeat statistic and decision rule

In each of three folds, two discovery repeats select a member for each task and a single best fixed member by mean discovery accuracy\. The third repeat scores the task\-wise mapping against that fixed member\. Averaging task\-level gains over folds givesGG; the paired difference between arms isD=Greal−GcloneD=G\_\{\\rm real\}\-G\_\{\\rm clone\}\(§[4\.2](https://arxiv.org/html/2609.35873#S4.SS2.SSS0.Px1)\)\. The repeat\-averaged oracle plug\-inH^\\widehat\{H\}is descriptive and does not determine the decision\.

Discovery scores lie on\{0,12,1\}\\\{0,\\tfrac\{1\}\{2\},1\\\}, so ties are common\. Selection uses independent, identically distributed uniform jitterU⁡\(0,10−9\)U\(0,10^\{\-9\}\)to break member ties symmetrically, averaging ten frozen tie realizations for the empirical report\. Each calibration dataset uses one tie realization\. For the empirical randomization reference, member labels are permuted independently within each task and each repeat in each arm\. Both selections, both gains, andDDare recomputed for each of10,00010\{,\}000draws, with the\+1\+1Monte Carlo correction\. A paired task bootstrap uses2,0002\{,\}000draws with a fixed random seed and the same resampled task identities in both arms to obtain the 95% interval\.

Positive support requires: at most10%10\\%incomplete tasks in either arm or their intersection before subsetting; at least3030paired tasks; randomizationp<0\.05p<0\.05; positive bootstrap lower bounds forDDandGrealG\_\{\\rm real\}; andD\>1\.0D\>1\.0pp\. Repeats must have matched conditions and verified independent execution provenance\. The current decision isInsufficient Evidence, rather than an equivalence finding\.

#### Analysis specification\.

The acquisition protocol and initial stopping rule preceded collection; the v3 diagnostic, recovery, and completeness\-selected reanalysis are post\-collection amendments\. Estimates use the September 26 snapshot; inclusion, exclusions, and conflict sensitivity are specified in Appendix[C](https://arxiv.org/html/2609.35873#A3)\.

### D\.2Population target and limits of the diagnostic

For member success probabilitiesph​\(x\)p\_\{h\}\(x\), stable complementarity is

Hstable=𝔼x​maxh​ph​\(x\)−maxh⁡𝔼x​ph​\(x\)\.H\_\{\\rm stable\}=\\mathbb\{E\}\_\{x\}\\max\_\{h\}p\_\{h\}\(x\)\-\\max\_\{h\}\\mathbb\{E\}\_\{x\}p\_\{h\}\(x\)\.It is zero when some fixed member is expectation\-best on almost every task\. For a given selector, its population gainθ∗\\theta^\{\*\}relative to the*population\-best fixed*member satisfiesθ∗≤Hstable\\theta^\{\*\}\\leq H\_\{\\rm stable\}, because its success probability on each task is at mostmaxh⁡ph​\(x\)\\max\_\{h\}p\_\{h\}\(x\)\.

EmpiricalGGuses a fixed member selected from finite discovery data\. Clone subtraction provides a sampling reference, whileDDand its interval remain distinct fromHstableH\_\{\\rm stable\}: a near\-zeroDDdoes not bound the population target near zero\.

The randomization reference is exact under equal\-ability member\-label exchangeability\. For the broader global\-dominance null, error control is empirical over the calibration grid\.

#### Finite\-repeat oracle estimates\.

For repeat meansp^h​\(x\)\\hat\{p\}\_\{h\}\(x\), with𝔼\\mathbb\{E\}here taken over repeated\-execution sampling at fixedxx, Jensen’s inequality gives𝔼​maxh​p^h​\(x\)≥maxh⁡𝔼​p^h​\(x\)\\mathbb\{E\}\\max\_\{h\}\\hat\{p\}\_\{h\}\(x\)\\geq\\max\_\{h\}\\mathbb\{E\}\\hat\{p\}\_\{h\}\(x\)on each task\. Headroom subtracts a second maximum, so this inequality does not imply upward bias of the difference in general\. In a known\-truth equal\-ability simulation withph​\(x\)=0\.9p\_\{h\}\(x\)=0\.9,M=9M=9members,T=400T=400tasks, andR=3R=3, the mean plug\-in is8\.698\.69pp over300300datasets despiteHstable=0H\_\{\\rm stable\}=0\. The empirical same\-code arm likewise gives2\.162\.16pp\. These examples show that positive plug\-in headroom is insufficient evidence of specialization\.

### D\.3Known\-truth calibration and statistical power

Binary tensors are sampled from specified member–task probability matrices\. Null settings include equal ability, shared task difficulty, global dominance, the0\.5/0\.90\.5/0\.9dominance configuration, and heterogeneous dominance\. Alternatives include crossovers over latent task types and difficulty\-related partial crossover\. The grid spansM∈\{3,9,20\}M\\in\\\{3,9,20\\\},T∈\{100,400,1000\}T\\in\\\{100,400,1000\\\},R∈\{3,5,10,20\}R\\in\\\{3,5,10,20\\\}, complete data, 5% missing completely at random \(MCAR\), and 3% hardness\-related missingness\.

For every repeat countRR, the calibrated statistic usesRRfolds: each fold learns both choices from the otherR−1R\-1repeats and scores them on its single held\-out repeat\. The table’s*plug*comparator declares a positive result when the lower endpoint of its percentile 95% task\-bootstrap interval exceeds zero \(1,0001\{,\}000draws\)\. It fixes the task\-wise maxima and sample\-best member before resampling\. The simulation variant, labeled v3∗, uses the positive\-support conditions above with300300bootstrap draws and300300randomizations, but checks the completeness gate only on the real arm\. The empirical rule additionally gates the clone arm and paired intersection\. Simulation abstention denotes failure to obtain positive support\.

In the two\-specialist setting, the400400tasks form two equal halves\. Each of two specialists succeeds with probability0\.950\.95on its own half and0\.600\.60on the other; the remaining seven members have probability0\.600\.60throughout\. Outcomes are independent Bernoulli draws\. The oracle mean is0\.950\.95and the best fixed mean is0\.7750\.775, soHstable=17\.5H\_\{\\rm stable\}=17\.5pp\. All nine clone slots share the first specialist’s probability profile, with fresh independent draws\.

Table[8](https://arxiv.org/html/2609.35873#A4.T8)reportsM=9,T=400M=9,T=400, using300300Monte Carlo datasets per cell \(150150in sweeps\) and300300randomizations per dataset\. Across the100100\-cell grid, the largest observed null false\-support rate is2%2\\%\. A two\-specialist alternative withHstable=17\.5H\_\{\\rm stable\}=17\.5pp has power0%0\\%atR=3R=3, approximately0\.7%0\.7\\%atR=5R=5, and100%100\\%atR=10,20R=10,20; smaller cyclic crossovers remain largely undetected atR=20R=20\. These setting\-specific results show why three repeats leave stable complementarity unresolved\.

†Null \(H=0H\{=\}0\): false\-support %; otherwise power %\.M=9,T=400M\{=\}9,T\{=\}400,300300datasets atR=3R\{=\}3\(150150in sweeps\)\.*plug*: positive 95% bootstrap lower bound\.∗v3 simulation variant: real\-arm completeness gate,300300bootstrap draws and300300randomizations\. The empirical rule additionally gates clones and their paired intersection; details are given above\.

Table 8:Three repeats can miss even a strong positive case\.AtM=9,T=400M=9,T=400, null rows report false\-support % and alternative rows report power %; the final column givesR=3R=3abstention\. The simulated clone\-calibrated rule \(v3∗\) rarely declares positive support when stable complementarity is zero, but has little sensitivity to the tested small and moderate crossovers\.

## Appendix EFrozen Selector and Execution\-Matched Replay

This appendix specifies how development data determine the frozen program choices and portfolio ranking, and how each is evaluated\.

### E\.1Frozen feature selector

The reported selectorπZ\\pi\_\{Z\}uses multinomial logistic regression with five task folds on the100100development tasks\. The score summary for memberhhon development taskxxis the available\-repeat mean,

Y¯h​\(x\)=1\|ℛh​x\|​∑r∈ℛh​xY⁡\(h,x,r\),\\bar\{Y\}\_\{h\}\(x\)=\\frac\{1\}\{\|\\mathcal\{R\}\_\{hx\}\|\}\\sum\_\{r\\in\\mathcal\{R\}\_\{hx\}\}Y\(h,x,r\),whereℛh​x\\mathcal\{R\}\_\{hx\}contains its observed development repeats\. A task is eligible for fitting only when every member has at least one observed repeat\. Its training label isy⁡\(x\)=arg⁡maxh​Y¯h​\(x\)y\(x\)=\\arg\\max\_\{h\}\\bar\{Y\}\_\{h\}\(x\), with ties resolved by ascending member ID\. Eligible tasks, ordered by task ID, are split into five consecutive folds without shuffling; all repeats of a task stay together\.

The TF\-IDF vocabulary and inverse\-document frequencies are fitted once on eligible development questions, using English stop\-word removal and at most300300features\. Each fold model appends the vector of member accuracies computed on its four training folds to every question’s features\. This vector is constant across questions within that model; only the text features vary by task\. Logistic regression uses inverse regularization strengthC=1C=1and at most1,0001\{,\}000iterations\. A training fold with only one label is omitted; absent member classes receive zero probability\. If1≤K≤51\\leq K\\leq 5models are fitted andqk​\(h∣x\)q\_\{k\}\(h\\mid x\)is modelkk’s predicted class probability, inference usesq¯​\(h∣x\)=K−1​∑k=1Kqk​\(h∣x\)\\bar\{q\}\(h\\mid x\)=K^\{\-1\}\\sum\_\{k=1\}^\{K\}q\_\{k\}\(h\\mid x\)\. Leth\(1\)h\_\{\(1\)\}andh\(2\)h\_\{\(2\)\}have the two highest averaged probabilities\. The final frozen choice is

πZ​\(x\)=\{single\-call,q¯​\(h\(1\)∣x\)−q¯​\(h\(2\)∣x\)<0\.15,h\(1\),otherwise\.\\pi\_\{Z\}\(x\)=\\begin\{cases\}\\text\{single\-call\{\}\},&\\bar\{q\}\(h\_\{\(1\)\}\\mid x\)\-\\bar\{q\}\(h\_\{\(2\)\}\\mid x\)<0\.15,\\\\ h\_\{\(1\)\},&\\text\{otherwise\}\.\\end\{cases\}The threshold is applied after averaging; a tied top probability therefore triggers the baseline fallback\. The policy table evaluates these final choices, including the baseline fallback\. It reports repeat\-mean accuracy, with harness executions as the unit of evaluation; a selected harness can make multiple model calls\.

Comparators are single\-call, the development\-selected fixed member, one fixed member drawn once with seed2026091520260915\(gsm\_kimi\_s0\_g3\), and the development\-selected member with baseline fallback on missing cells\. All policies are evaluated on the common386386tasks, with three scores per selected member–task pair\. Paired differences against dev\-fixed use2,0002\{,\}000task\-bootstrap draws\. The feature, dev\-fixed, and fallback policies retain their recorded single\-call choices\.

Table 9:Frozen selection on MATH\-500\.Accuracy is repeat\-mean correctness on fresh executions;SSis the paired gain over the development\-selected fixed member, with 95% task\-bootstrap intervals\.
### E\.2Selection gain and the scope of budget utility

For the implemented selector,Z⁡\(x\)Z\(x\)consists only of question features and training\-fold member accuracies\. Its gain over the development\-selected fixed member is

S⁡\(πZ\)=1n​∑i=1n\[Y¯πZ​\(xi\)​\(xi\)−Y¯hdev​\-​fixed​\(xi\)\],S\(\\pi\_\{Z\}\)=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\left\[\\bar\{Y\}\_\{\\pi\_\{Z\}\(x\_\{i\}\)\}\(x\_\{i\}\)\-\\bar\{Y\}\_\{h\_\{\\rm dev\\text\{\-\}fixed\}\}\(x\_\{i\}\)\\right\],whereY¯h​\(x\)\\bar\{Y\}\_\{h\}\(x\)now averages the three evaluation repeats,nnis the number of evaluation tasks, andhdev​\-​fixedh\_\{\\rm dev\\text\{\-\}fixed\}is the fixed member selected using development data\. We useSSfor policy gain and reserveDDfor the real\-minus\-clone repeat diagnostic\. Mechanism labels are not inputs to this feature selector\.

Replay budgetbbcounts*harness executions*, each potentially containing multiple model calls; coverage uses a post\-execution oracle\.

### E\.3Execution\-matched replay and collection accounting

Each arm contains nine members/slots with three repeats\. Replay uses the common386386tasks and frozen development ranking \(§[5\.4](https://arxiv.org/html/2609.35873#S5.SS4.SSS0.Px2)\), taking all three repeats of the firstb/3b/3members forb∈\{3,9,15,27\}b\\in\\\{3,9,15,27\\\}\. Withffincorrect clone executions among2727, expected coverage of a uniform size\-bbsubset is1−\(fb\)/\(27b\)1\-\\binom\{f\}\{b\}/\\binom\{27\}\{b\}, where\(fb\)=0\\binom\{f\}\{b\}=0forb\>fb\>f\. Table[3](https://arxiv.org/html/2609.35873#S5.T3)compares this exact expectation with generated\-panel coverage at matched execution counts\.

Acquisition ledgers retain all attempts and known usage, including excluded tasks and recovery; outcome tables use the common complete set\. Calls and tokens follow each program and acquisition record\.

## Appendix FBIRD Execution Traces and Supporting Analyses

Phase I studies BIRD discovery; Phase II studies admission\. This appendix details the discovery programs and their execution paths\.

### F\.1Discovery populations and conditions

#### Population and judging\.

Discovery evaluates twelve TTHE\-proposer candidates from GLM\-5\.3\-Flash, plus single\-call andreact, on6060tasks \(§[5\.5](https://arxiv.org/html/2609.35873#S5.SS5)\)\. Admission checks import and smoke execution\. The baseline flips on three of6060tasks between discovery runs\. The set\-of\-rows proxy judge agrees with the official BIRD scorer on 0\.97–0\.99 of outcomes in two matrices with retained SQL, with no proxy\-correct/official\-incorrect cases\.

#### Development exposure\.

Hand\-written controls were designed after trace inspection, and24/15124/151evaluation items overlap protocol development\. Their headroom is descriptive\. Equal generation budgets cannot be reconstructed from the early records, which lack generation\-tool and per\-attempt logs\.

Figure 2:Source differences need not change observed correctness\.BIRD discovery:1414harnesses,6060tasks, one execution each\. Across9191pairs including baselines, token\-sequence source similarity hasρ=−0\.010\\rho=\-0\.010rank association with correctness disagreement;36/9136/91pairs have identical correctness vectors\. This is an outcome comparison on the observed tasks, not a test of program equivalence\.

### F\.2Discovery measurements and hand\-written controls

All candidates pass the code\-difference check, but21/6621/66generated pairs share correctness vectors \(36/9136/91with baselines\)\. Source similarity and correctness disagreement haveρ=−0\.010\\rho=\-0\.010across9191pairs \(Figure[2](https://arxiv.org/html/2609.35873#A6.F2)\)\. On these6060tasks, mean disagreement is3\.3%3\.3\\%overall and3\.8%3\.8\\%among generated candidates, union repair is7\.7%7\.7\\%, and single\-run headroom is3\.33\.3pp\. Traces explain how distinct programs produce these closely aligned outcomes\.

### F\.3Traces distinguish absence, non\-activation, and failure

T1: described but absent\.Six of twelve candidates claim retrieval or self\-checking but make one model call without implementing either step\.T2: present but untriggered\.On all ten auditedreacttasks, the first SQL executes without error, leaving the repair branch idle and final SQL identical to single\-call; execution success can coexist with semantic error\.T3: executed but broken\.One of twelve candidates executes multi\-turn repair but returns prose as SQL, losing baseline\-correct tasks\.

The three cases separate implementation, branch activation, and output validity\. Four hand\-written controls with distinct paths yield6\.676\.67pp headroom on6060tasks and7\.287\.28pp on151151items\. These post\-inspection controls lack matched repeats and clones; their24/15124/151development overlap is documented in Appendix[F\.1](https://arxiv.org/html/2609.35873#A6.SS1)\. Worked traces follow in Appendix[F\.4](https://arxiv.org/html/2609.35873#A6.SS4)\.

### F\.4Worked examples from the trace audit

The six\-of\-twelve T1 count covers the full discovery population\. The separate trace audit follows five harnesses—the single\-call baseline,react, a generated single\-call candidate, a timeout\-retry candidate, and a multi\-turn repair candidate—on ten tasks \(5×\\times10\)\. Below, one task illustrates these paths\. Logged SQL excerpts are capped at 120 characters; truncation is marked*\[trunc\.\]*\.

#### The task\.

BIRD dev\-set question 352 \(card\_gamesdatabase; question text verbatim, the typo is the dataset’s\):

> Calculate the percentage of the cards availabe in Chinese Simplified\.

Its BIRD hint:

> ’Chinese Simplified’ is the language; percentage = Divide\(Sum\(id where language = ’Chinese Simplified’\), Count\(id\)\) \*100

Gold SQL:

> SELECT CAST\(SUM\(CASE WHEN T2\.language = ’Chinese Simplified’ THEN 1 ELSE 0 END\) AS REAL\) \* 100 / COUNT\(T1\.id\) FROM cards AS T1 INNER JOIN foreign\_data AS T2 ON T1\.uuid = T2\.uuid

The single\-call harness answers:

> SELECT SUM\(CASE WHEN language = ’Chinese Simplified’ THEN 1 ELSE 0 END\) \* 100\.0 / COUNT\(id\) AS percentage FROM foreign\_data;

On the released database, the gold query yields 8\.7734 and the baseline query yields 8\.7728\. The official scorer marks the baseline query wrong: it counts directly overforeign\_data, whereas the gold query counts after joining tocards; the hint does not spell out that join\.

#### react\(T2: untriggered repair\)\.

The first prompt is byte\-identical to single\-call’s\. SQL executes successfully, so the database\-error retry does not fire\. All 10 audited tasks follow this path, with SQL execution but 10/10 final queries byte\-identical to single\-call’s\.

#### Single\-call candidate \(T1: absent repair\)\.

The docstring’s*conservative repair*specifies an output\-token cap and answer\-only block\. Execution consists of one model call followed by SQL extraction\. The cap is an environment default \(2,048 versus 32,000 at runtime\), has no harness control\-flow implementation, and never binds on these short answers\. The worked task returns the same wrongforeign\_dataquery, modulo whitespace, with one call and zero SQL executions\. Across all 10 tasks, SQL executions remain zero and 6 final queries are byte\-identical to single\-call’s\. The remaining differences reflect a reworded prompt, without retrieval, self\-check, or repair\.

#### Timeout\-retry candidate \(untriggered watchdog\)\.

A daemon\-thread watchdog wraps the coder call and issues one hedged duplicate after a 40 s stall\. No stall occurs in the audit: SQL executions are zero and final SQL equals single\-call’s on all 10 tasks\.

#### Multi\-turn repair candidate \(T3: a broken multi\-turn path\)\.

The one audited candidate implementing genuine execute\-inspect\-repair returns the model’s*prose*as final SQL\. On a task single\-call answered correctly \(“State the alternative languages available for card named Annul numbered 29\.”\), its logged final SQL begins, verbatim:

> with "only a single read\-only SELECT is allowed"\. The previous "query" was actually just prose text \(the model wrote rea*\[trunc\.\]*\)

Under this repair re\-prompt, the output contract fails again: asked to fix a query that “was actually just prose”, the frozen target responds with more prose reasoning, no SQL block is found, and the reasoning text is submitted as the query\. The item flips from baseline\-correct to harness\-wrong\.

## Appendix GBIRD Admission\-Policy Study

The2×22\{\\times\}2study uses1818builder–seed cells per arm \(six builders and three seeds\); II\-E is development\-only\. Each cell has eight generation slots\. Each slot allows three generation attempts and stops at first admission without top\-up\. Two development databases are excluded\. Evaluation uses1,1691\{,\}169questions from nine databases and a pre\-drawn400400\-item core, with one disclosed1818\-item smoke\-test exposure\. The official BIRD judge uses a3030s timeout\. Model\-specific response caching uses cross\-process locks after correction of a concurrency defect\.

The primary D−\-A contrast \(forced/gated minus free/ungated headroom\) pairs builder–seed cells with hierarchical bootstrap and permutation inference; core analyses use Holm\-corrected factorial contrasts and K\-matched sensitivity\. Instrument fixes preceded confirmatory outcomes\. Subsequent analysis corrections share baseline records and database/task resampling\. Canonical records use first\-write\-wins deduplication; conflicts and operational deviations remain in the source audit\. Generation costs vary across arms\.

Figure 3:BIRD admission design\.Left: data splits and disclosed development exposure\. Middle: the2×22\{\\times\}2four\-arm comparison with a common attempt cap\. Right: neutral validity and the implemented structural\-screening/conformance bundle in gated arms\. The additional open\-mechanism arm II\-E is development\-only\.### G\.1Design and primary contrasts

The four arms cross strategy forcing and gating: II\-A \(free, ungated\), II\-B \(free, gated\), II\-C \(forced, ungated\), and II\-D \(forced, gated\)\. Forced strategies come from a fixed set; free builders select a declaration within the vocabulary in Appendix[G\.4](https://arxiv.org/html/2609.35873#A7.SS4)\. Results below use the corrected reanalysis of the pre\-outcome specification\.

Table 10:BIRD admission: full\-set headroom\.Means over builder–seed cells on1,1691\{,\}169questions; single\-call is included in both the oracle and best\-fixed comparators\.The forced/gated minus free/ungated contrast is−0\.16\-0\.16pp in baseline\-inclusive oracle headroom, with cell contrasts−4\.36​to\+2\.74\-4\.36\\text\{ to \}\+2\.74pp\. The implemented bundle does not improve observed mean headroom; Table[10](https://arxiv.org/html/2609.35873#A7.T10)reports uncertainty\.

Figure 4:BIRD admission contrasts\.Left: full\-set D−\-A cell contrasts with the pooled 95% interval\. Right: four\-arm headroom on the shared core\-400400tasks with a common single\-call baseline\. Headroom and contrast axes show percentage points\.On core\-400400, the gate contrast is−0\.69\-0\.69pp, with inconclusive strategy and interaction effects \(Table[11](https://arxiv.org/html/2609.35873#A7.T11)\)\. The candidate\-count\-matched contrast is−0\.31\-0\.31pp\. II\-E remains development\-only\. The R2/R3 audits \(Appendix[G\.3](https://arxiv.org/html/2609.35873#A7.SS3.SSS0.Px1)\) place these single\-run coverage measurements alongside8\.54%8\.54\\%R2 flips,14\.5%14\.5\\%of baseline tasks changing across three runs, and a6\.506\.50pp same\-code oracle gap\.

Table 11:Corrected core\-400400factorial reanalysis\.Here A–D denote arm mean headrooms\. Gate is\[\(B−A\)\+\(D−C\)\]/2\[\(B\{\-\}A\)\+\(D\{\-\}C\)\]/2, strategy\[\(C−A\)\+\(D−B\)\]/2\[\(C\{\-\}A\)\+\(D\{\-\}B\)\]/2, interaction\(D−C\)−\(B−A\)\(D\{\-\}C\)\-\(B\{\-\}A\); raw and Holm\-adjustedpp\-values\.
### G\.2Same\-core headroom decomposition

Table[12](https://arxiv.org/html/2609.35873#A7.T12)separates coverage, best\-fixed accuracy, and headroom using the same core tasks and baseline records in every arm\.

Table 12:Same\-core headroom decomposition\.All arms use identical 400 tasks and identical single\-call outcomes\.KcandK\_\{\\rm cand\}counts admitted candidates, excluding single\-call, andKtotal=Kcand\+1K\_\{\\rm total\}=K\_\{\\rm cand\}\+1; counts and accuracies are cell means\. Headroom is oracle minus best\-fixed accuracy, not realized routing gain\.
### G\.3Sensitivity, repeat audits, and recorded cost

#### K\-matched and R2 replays\.

Replays use deterministic record ordering, shared database/within\-database task draws, and seed resampling within fixed builders \(10,000 replicates\)\. K\-matching uses the smaller candidate count in each A/B and C/D pair, exactly enumerates uniform subsets, retains single\-call, and reselects best\-fixed within each subset and task draw\. It measures population\-size sensitivity\. R2, described below, retains all 18 cells, including two singleton A subsets; its D−\-A changes from \+0\.42 to \+0\.81 pp \(change \+0\.39 pp, 95% CI \[\-0\.95, \+1\.63\]\)\.

R2 covers 70 harnesses on 400 tasks, taking the first two callable admitted A/D candidates where available\. Two cache\-off acquisition groups remain separate despite their duplicated repeat label, with program identity checked against cached runs\. Without matched single\-call acquisitions, both R2 conditions exclude that baseline; singleton headroom is zero across the 18 retained pairs\. Recorded flips are 2391/28,000 against cached outcomes \(8\.54%\) and 1245/14,400 between D cache\-off passes \(8\.65%\)\. These describe observed instability: acquisition time and caching are confounded, and the rates are not standard errors or effect\-size floors\.

#### R3 single\-call\-only repeat audit\.

Three independent cache\-off runs evaluate only the single\-call baseline on core\-400 with GLM\-5\.3\-Flash at temperature 0\. Correctness changes on 58/400 tasks \(14\.5%\); pairwise flips are 39/400, 40/400, and 37/400 \(9\.75%, 10\.0%, 9\.25%\)\. Treating runs as three pseudo\-members, one execution each, gives official\-scorer oracle coverage72\.25%72\.25\\%and best\-repeat accuracy65\.75%65\.75\\%: a6\.506\.50pp same\-code gap\. This records baseline execution variability at temperature 0 and motivates the repeat controls used in the main study\.

Table 13:BIRD sensitivity on core\-400\.CIs are unadjusted\. K\-matched comparisons always retain single\-call; R2 comparisons exclude single\-call and are conditional on the observed acquisition groups\.
#### Recorded cost\.

The core records contain 259,304 logical solver calls across 153,600 candidate–task records \(Table[14](https://arxiv.org/html/2609.35873#A7.T14)\)\. These include cache\-served requests and count logical rather than billed requests; token usage is unavailable\. Generation attempts, admissions, and execution calls are reported separately\. Population cost averages weight builder–seed cells equally, exclude single\-call, and assign zero calls to zero\-candidate populations\.

Table 14:BIRD generation and execution cost on core\-400\.Logged logical calls are not billable requests or token counts\. The last column averages candidate\-population calls per task over cells\.

### G\.4Admission scope and implementation

Free\-arm builders choose among repair, vote, twostage, and plain\. Vote requires multiple samples and selection, including but not limited to three\-way majority voting\. Twostage consumes an artifact without a prescribed response interface\. Plain declares no control\-flow mechanism despite the prompt requesting a change\. Thus strategy choice operates within this four\-declaration vocabulary\.

The evaluated gate combines structural screening with conformance verification; valid prompt\-only transformations can fail the structural screen\. Development checks found false rejections, request\-attribution and response\-interface defects, and unresolved judgments\. Later revisions produced no replacement benchmark outcomes, so results describe the implemented bundle\.

#### Generation interface\.

Format\-guard is one of eight fixed strategies, each assigned one slot per builder–seed cell in both forced arms \(C and D\)\. It requests schema fidelity and a final SQL code block with no extra text\. All108108format\-guard attempts across its3636slots fail neutral validity:7171logs report unterminated strings,3636no Python fence, and one a provider timeout\. Synthetic examples reproduce truncation in the archived non\-greedy fence extractor\. Full builder messages and exact historical extractor versions are unavailable, limiting attribution of these upstream validity failures\.