Metrics That Write Themselves: Evolving an Evaluator from Its Own Blind Spots

# Metrics That Write Themselves:Evolving an Evaluator from Its Own Blind Spots
Source: [https://arxiv.org/html/2608.18744](https://arxiv.org/html/2608.18744)
Yanwei CuiGuanghui WangZhihao LinPeiyang HeThanks:Corresponding author:peiyan@amazon\.comAffiliation:\[3pt\] AWS Generative AI Innovation Center

###### Abstract

Agents improve quickly against a reliable automatic metric and stall without one, and the applications that need them most, report generation among them, are the ones nobody knows how to score\. Can the metric write itself? Saying what makes an answer good is hard; pointing at something wrong with one is easier, so the metric we evolve is a pool of small Python*operators*that each flag a candidate for one named defect, or abstain, and vote\. Asking a model for operators directly does not work: 183 candidates realise only 96 distinct behaviours, from one narrow region of an enormous space\. EvalCEGAR instead borrows counterexample\-guided abstraction refinement from program verification\. It reads the pool as an abstraction and searches for a*collision*, two answers the operators score identically, one correct and one not\. That pair, not a prompt, is the authoring request, and when a collision defeats every attempt the loop widens what an operator may read rather than resampling\. On MBPP\+ and HumanEval\+, a sandbox whose hidden unit tests give exact ground truth, the loop writes a 55\-line operator that closes15\.4%15\.4\\%of the gap between flagging nothing and a perfect filter on 428 unseen tasks \(\+0\.0065\+0\.0065,p=0\.0010p\{=\}0\.0010\) at a quarter of our best hand\-written operator’s flags\. On the benchmark it never saw it matches that operator’s effect exactly on a third of the flags\. Six of eight runs admit such an operator and all six help out of sample; our 15 hand\-written operators applied together as one filter*lose*accuracy\. An LLM judge on the same information ties that delta on a nearly disjoint set of candidates, and charges a model call per candidate forever where the operator charges none\.

## 1Introduction

A self\-improving system needs a metric before it needs a policy\. Given a reliable automatic metric, LLM agents improve rapidly against it: self\-refinement, self\-debugging and reflective loops all assume a signal separating better answers from worse\([madaan2023selfrefine](https://arxiv.org/html/2608.18744#bib.bib13);[shinn2023reflexion](https://arxiv.org/html/2608.18744#bib.bib20);[chen2024selfdebug](https://arxiv.org/html/2608.18744#bib.bib5)\)\. Without one, progress stalls, and the substitutes are weak: reference overlap misses semantics by construction\([papineni2002bleu](https://arxiv.org/html/2608.18744#bib.bib16);[lin2004rouge](https://arxiv.org/html/2608.18744#bib.bib10)\), and an LLM judge carries position, verbosity and self\-preference biases that belong to the judge, not the answer\([zheng2023judging](https://arxiv.org/html/2608.18744#bib.bib24);[wang2023unfair](https://arxiv.org/html/2608.18744#bib.bib21)\)\. This bites hardest on open\-ended output, a report or a plan, where there is neither a reference answer nor a settled account of what a good one looks like\.

This paper asks whether the*metric*can be evolved automatically\. What we evolve is not a score but a pool of*operators*: small Python functions that each read a task and one candidate answer and either*flag*it for one specific defect, pass it as*clean*, or*abstain*\. A pool becomes a metric by voting: a candidate is rejected when enough members flag it\. This form follows an asymmetry that survives the absence of a reference: saying how good an answer is needs a standard nobody has written down; pointing at something wrong with it usually does not\. Each operator states one such objection, so it can be read, run and falsified on its own, and the pool’s validity is a measured property rather than an assumed one: every member had to earn its place on a downstream decision before it joined\.

#### What our first runs showed\.

We first ran the obvious loop: ask the model for an operator, keep it if a gate accepts it, repeat\. It stalls in two ways, and we take both as the problem statement\.*Obstacle 1: the space of operators is enormous, and a model asked for “an evaluation operator” samples a tiny, fixed region of it\.*Resampling does not move that region: scoring all 295 operators the study authored at that narrow interface yields no discovery at all \([Section5\.2](https://arxiv.org/html/2608.18744#S5.SS2)\)\.*Obstacle 2: what it does write repeats itself behaviourally, and most of it is rejected\.*Five runs produced 183 operators that flag anything, realising only 96 distinct flag sets \([Section6](https://arxiv.org/html/2608.18744#S6)\)\. A rejection teaches nothing either: the loop learns that a candidate failed, not which distinction it failed to draw\.

#### Our approach\.

We borrow a discipline from program verification: counterexample\-guided abstraction refinement sharpens an abstraction only when a concrete counterexample proves the current one too coarse\([clarke2000cegar](https://arxiv.org/html/2608.18744#bib.bib7)\)\. The operator pool is that abstraction, mapping each candidate answer to a*signature*, the vector of verdicts the pool returns for it\. Two answers with the same signature are indistinguishable to the metric, so a pair sharing one while disagreeing on ground truth proves the metric cannot express the distinction\. EvalCEGAR hands that pair to the model as the specification\. Each admission re\-partitions the signatures, so the next specification differs by construction: the target, not the sampler, supplies the diversity pressure \(Obstacle 2\)\. When a pair defeats every attempt, the loop widens the*interface*an operator may read rather than sampling again, since a distinction the interface cannot see is unreachable at any sample count \(Obstacle 1\)\. An operator is admitted only if it improves the decision the metric is deployed to make, not if it covers more known faults, and admitted operators vote\.

#### Why we validate on code\.

The motivation is domains*without*a usable metric, yet every number here comes from a domain that has an exact one: Python problems from MBPP\+ and HumanEval\+\([austin2021mbpp](https://arxiv.org/html/2608.18744#bib.bib1);[chen2021codex](https://arxiv.org/html/2608.18744#bib.bib4);[liu2023evalplus](https://arxiv.org/html/2608.18744#bib.bib11)\), whose hidden unit\-test suites settle correctness outright\. A method that writes metrics cannot be evaluated where the truth is unknown, so we build it where the truth is known and withheld\. The operators are domain\-specific by construction, so a new domain inherits the loop, not this pool\. What transfers is narrow but checkable: the operators never read the oracle, four screens enforce that they do not reconstruct one \([AppendixA](https://arxiv.org/html/2608.18744#A1)\), and oracle labels enter only on the training split, at a price a new domain can pay: of our 75 training tasks only 10 carry both a correct and an incorrect candidate that survive the visible checks, and that was enough to admit operators that transfer \([AppendixB](https://arxiv.org/html/2608.18744#A2)\)\.

#### Contributions\.

- •EvalCEGAR\.A loop that evolves an evaluation metric from its own blind spots, the authoring request being a counterexample it finds for itself \([Section4](https://arxiv.org/html/2608.18744#S4)\)\.
- •Operators that hold up out of sample\.Six of eight runs admit an operator and all six help on 428 unseen tasks; the best is 55 lines of Python and matches our best hand\-written operator on a second benchmark at a third of its flags, where a model judge on the same information ties it while flagging a near\-disjoint set \([Section5\.1](https://arxiv.org/html/2608.18744#S5.SS1)\)\.
- •What each mechanism is worth\.Ablated inside the loop on identical information: the admission question alone moves admissions from zero to one, and the narrow interface admits nothing in 336 attempts \([Section5\.2](https://arxiv.org/html/2608.18744#S5.SS2)\)\.
- •Composition solved exactly, not searched\.Scoring all 2\.3M subsets the loop can compose ranks the objective it used to combine them as low as the 5th percentile of that space, and an untuned class\-weighted one at the 91st or above \([Section5\.3](https://arxiv.org/html/2608.18744#S5.SS3)\)\.

## 2Related work

#### Automatic evaluation of open\-ended output\.

Both substitutes above have been sharpened without changing shape: learned metrics replace overlap counting with a model’s own similarity judgement\([zhang2020bertscore](https://arxiv.org/html/2608.18744#bib.bib23)\), and judge frameworks hand the judge explicit criteria to fill in\([liu2023geval](https://arxiv.org/html/2608.18744#bib.bib12)\), refined by decomposing coarse ones and filtering the redundant\([shen2026rrd](https://arxiv.org/html/2608.18744#bib.bib19)\)\. The output is still a single scalar whose parts a reader cannot separate\. EvalCEGAR produces a different object, a pool of small, executable, individually falsifiable*objections*\. A pool can gain or lose one without retraining, which is what makes a metric of this form evolvable\. We compare against both kinds: the judge ties us on the endpoint while dissenting on which candidates to drop \([Section5\.1](https://arxiv.org/html/2608.18744#S5.SS1)\), so the claim is about how a checkable metric is produced, not about beating a scalar\.

#### LLM\-driven program search and CEGAR\.

FunSearch evolves programs against a*given*evaluator\([romera2024funsearch](https://arxiv.org/html/2608.18744#bib.bib18)\), and automated agent design searches agent code against a*given*benchmark\([hu2024adas](https://arxiv.org/html/2608.18744#bib.bib9)\)\. Ours is the complement: with the evaluator itself as the artefact there is no fixed fitness function, so the design problem moves from how candidates are mutated to what a candidate must prove before it is kept\. Closest in output are methods that induce executable checks with an LLM and combine them, by a compact set of Python verifiers whose joint satisfaction approximates a labelled objective\([pezeshkpour2026autopyverifier](https://arxiv.org/html/2608.18744#bib.bib17)\)or an unweighted vote over model\-written yes/no questions\([griffin2025rrf](https://arxiv.org/html/2608.18744#bib.bib8)\)\. Those search for checks that predict a label; we search for the counterexample showing the current ones cannot, and admit on the deployed decision instead\. We run that recipe as a baseline \([Section5\.2](https://arxiv.org/html/2608.18744#S5.SS2)\)\. CEGAR is a verification technique\([clarke2000cegar](https://arxiv.org/html/2608.18744#bib.bib7)\); to our knowledge its transfer to*metric*synthesis is new \([Section4](https://arxiv.org/html/2608.18744#S4)\)\.

#### Diversity and test generation\.

Quality\-diversity search keeps a population from collapsing by maintaining an explicit archive of behaviours\([mouret2015mapelites](https://arxiv.org/html/2608.18744#bib.bib15)\); our diversity pressure comes instead from re\-partitioning the target after each admission, and[Section6](https://arxiv.org/html/2608.18744#S6)measures what that does not buy\. Nor is this test generation: a property\-based tester needs a property to falsify\([claessen2000quickcheck](https://arxiv.org/html/2608.18744#bib.bib6)\)and differential testing needs a second implementation\([mckeeman1998differential](https://arxiv.org/html/2608.18744#bib.bib14)\), and our setting supplies neither\.

## 3Preliminaries

#### Tasks and data\.

Every task is a Python programming problem from MBPP\+ or HumanEval\+ as distributed by EvalPlus\([liu2023evalplus](https://arxiv.org/html/2608.18744#bib.bib11)\), whose hidden test suite is the exact oracle\. It labels the training samples the loop targets and admits on, and nothing else reads it but the scoring script\. A statement also shows a few example calls with their expected results, which the solver sees too; we call these the*visible checks*, and they are not held\-out information\. We drew 150 MBPP\+ tasks, split 75 to run the loop on and 75 held out, and added the 228 remaining MBPP\+ tasks and 125 of the 164 HumanEval\+ tasks to the held\-out side\.111HumanEval\+ prints no asserts, so its visible checks are synthesised from the docstring examples; we keep the 125 tasks that yield at least one such check the reference solution passes\.Results are reported on those 428 unseen tasks as*MBPP\+ held\-out*\(75\),*MBPP\+ additional*\(228\) and*HumanEval\+*\(125\), the only group from a different task distribution\.

#### Selection accuracy\.

For each taskttwe sample many model solutions and letV⁡\(t\)V\(t\)be those that pass the visible checks, duplicates included: the candidates a deployed system would plausibly accept, so the faults that matter are the ones surviving them\. A metric must choose among them\. It drops fromV⁡\(t\)V\(t\)what it flags, leaving the kept setK⁡\(t\)K\(t\), and one sample is drawn uniformly fromK⁡\(t\)K\(t\), or fromV⁡\(t\)V\(t\)if the metric flagged everything\.*Selection accuracy*is the probability that this sample is correct, averaged over tasks;Δ\\Deltaalways means its change against the metric that flags nothing\.

#### Decidability and headroom\.

A task is*decidable*only ifV⁡\(t\)V\(t\)holds both a correct and an incorrect sample; on any other task the drawn sample’s correctness is settled whatever the metric does\. Only 58 of the 428 held\-out tasks are decidable at our sampling depth, so selection accuracy over all 428 runs from0\.71700\.7170when nothing is flagged up to0\.75930\.7593for a perfect operator\. AbsoluteΔ\\Deltavalues are small for that reason, so we also report each as a fraction of that reachable range, the*headroom*\. The endpoint is also*recall\-dominated*: a flagged correct sample merely leaves the kept set, while a caught incorrect one changes what remains to draw from\.

#### What we compare against\.

Before building the loop we wrote 15 operators by hand, ten static checks on the candidate’s syntax tree and five that run it\. They are the pool the loop starts from and, applied together as one filter, our human baseline\. Our strongest single hand\-written operator, the*comparator*throughout, flags a candidate whose behaviour disagrees with the plurality of the other samples for its task\. The second baseline is an LLM judge, asked one question per candidate under exactly a level\-1 operator’s information, in four configurations over two models and two prompts \([AppendixI](https://arxiv.org/html/2608.18744#A9)\)\.

#### The model that writes operators\.

One frozen model \(Claude Opus 4\.7, no fine\-tuning and no gradient update anywhere in this work\) emits operator source code, 47–48 calls per run under a 60\-call ceiling across 8 rounds\. The faulty candidates its operators are judged on come from weaker models \(Llama 3\.1 8B Instruct and Claude Haiku 4\.5, with Amazon Nova Micro and Mistral 7B Instruct added for the deeper draw of[Section6](https://arxiv.org/html/2608.18744#S6)\), so the model that writes bad code never wrote the operators that judge it\. What EvalCEGAR ships, the operator pool and its vote, is pure Python with*no*inference\-time model call\.

#### Keeping the loop honest\.

Four screens reject an authored operator before the gate sees it, each closing a way an earlier round of this work reconstructed the oracle by accident: deciding from the candidate’s syntax without running it, keying on a constant the fault generator happens to emit, branching on prompt keywords, and reaching a flag by text match while executing the candidate elsewhere\. None fires on any of the 15 hand\-written operators, so they are not a ban on ordinary detectors \([AppendixA](https://arxiv.org/html/2608.18744#A1)\)\. EveryΔ\\Deltais tested against a size\-matched shuffle null: per task the operator’s own flag count is redrawn from that task’s own samples, so the null fixes how much it flags and randomises only where the flags land\. That licenses “this rule beats flagging nothing” but not “ruleAAbeats ruleBB”, the claim[Section6](https://arxiv.org/html/2608.18744#S6)reports on, and multiple comparisons carry a Bonferroni correction within a family of related tests and a Benjamini–Hochberg one across a screen of many\([benjamini1995controlling](https://arxiv.org/html/2608.18744#bib.bib2)\)\. And*registered*means the prediction was written down before the script that tests it existed\.

## 4Method

Figure 1:EvalCEGAR\. The authoring request is a*blind spot*: answers the pool maps to one signature while ground truth disagrees \(left\)\. Admission re\-partitions the signatures, so the next request has moved, and a blind spot that resists level 1 escalates \(right\) rather than being resampled\.#### In one sentence\.

Find two candidate answers the*current*pool scores identically, one correct and one incorrect; ask the model for an operator that catches faults of that kind; widen what an operator may read when the request proves unreachable; and admit on whether the deployed decision improves\.[Figure1](https://arxiv.org/html/2608.18744#S4.F1)is the loop and[Algorithm1](https://arxiv.org/html/2608.18744#alg1)one round of it\.

Algorithm 1EvalCEGAR, one round on one target\.ΔD\\Delta\_\{D\}is the change in selection accuracy on the training tasksDD\. Ground\-truth labelsyyare read on the training split only, at lines 2 and 10;Authorsees the specification and the interface, neveryy\.1:operator pool

PP, training tasks

DDwith candidate sets

V⁡\(t\)V\(t\)and labels

yy, level\-1 retry budget

rmaxr\_\{\\max\}, total attempt budget

RR
2:

σP​\(c\)←\(o⁡\(t,c\)\)o∈P\\sigma\_\{P\}\(c\)\\leftarrow\\big\(o\(t,c\)\\big\)\_\{o\\in P\}for every

c∈V⁡\(t\)c\\in V\(t\),

t∈Dt\\in D⊳\\trianglerightthe pool*is*the abstraction

3:

𝒞←\{\\mathcal\{C\}\\leftarrow\\\{\\,signature classes containing both a

y=1y\{=\}1and a

y=0y\{=\}0candidate

\}\\,\\\}⊳\\trianglerightthe blind spots

4:if

𝒞=∅\\mathcal\{C\}=\\emptysetthenreturn

PP⊳\\trianglerightthe pool already separates every labelled pair

5:

κ←arg⁡maxclass∈𝒞​\|class\|\\kappa\\leftarrow\\arg\\max\_\{\\text\{class\}\\in\\mathcal\{C\}\}\|\\text\{class\}\|⊳\\trianglerightthe target

6:

spec←\\mathrm\{spec\}\\leftarrowevery incorrect training sample that fails as

κ\\kappa’s do, and every correct one⊳\\trianglerightnot just the pair

7:

ℓ←1\\ell\\leftarrow 1⊳\\trianglerightinterface level:op\(task, code\)

8:for

r=1​…​Rr=1\\dots Rdo

9:

o←Author​\(spec,ℓ\)o\\leftarrow\\textsc\{Author\}\(\\mathrm\{spec\},\\ell\)⊳\\trianglerightone LLM call; a stated request, not “write a metric”

10:if

oofails a leakage screen, does not shrink

κ\\kappa, or is a function of

σP\\sigma\_\{P\}thencontinue

11:if

ΔD​\(P∪\{o\}\)\>0∧helped≥3∧helped\>hurt\\Delta\_\{D\}\(P\\cup\\\{o\\\}\)\>0\\;\\wedge\\;\\mathrm\{helped\}\\geq 3\\;\\wedge\\;\\mathrm\{helped\}\>\\mathrm\{hurt\}then

12:return

P∪\{o\}P\\cup\\\{o\\\}⊳\\trianglerightadmission re\-partitionsσP\\sigma\_\{P\}, so the nextκ\\kappahas moved

13:if

r=rmaxr=r\_\{\\max\}then

14:

ℓ←2\\ell\\leftarrow 2⊳\\triangleright*same*κ\\kappa, wider interfaceop\(task, code, ctx\)

15:return

PP⊳\\trianglerightκ\\kappaabandoned: unreachable at either interface

#### Collision targeting\.

A*collision*, or blind spot, is a signature class holding both correct and incorrect candidates, and the largest one in the pool becomes the next target\. The colliding pair locates the request but does not state it: the request covers every training sample that fails in the same way, because a specification naming two samples is satisfiable by a lookup table \([AppendixA](https://arxiv.org/html/2608.18744#A1)\)\. A target that survives every attempt at both interfaces is abandoned, and once every target has been abandoned the loop returns an*insufficiency certificate*for the operator vocabulary: the metric*language*, not the author, is the limit\. That is what makes a failed run informative: it names the distinction the vocabulary could not express\.

#### Interface escalation\.

Level 1 isop\(task, code\)\. Whenrmax=3r\_\{\\max\}\{=\}3level\-1 attempts fail on a target, the*same*target escalates to level 2,op\(task, code, ctx\), wherectxlets the operator see the candidate’s*peers*and run any of them on inputs of its own choosing, under a hard budget of 600 observations per operator call over at most 16 peers \([Figure1](https://arxiv.org/html/2608.18744#S4.F1), right, and[AppendixA](https://arxiv.org/html/2608.18744#A1)\)\. This changes the coordinates of the search instead of resampling within them: if the useful operator is not expressible at level 1, no amount of sampling finds it\.

#### Admission on the deployed decision\.

The final gate is a floor on the*decision*, evaluated on the training split:Δ\>0\\Delta\>0, at least 3 tasks helped, helped\>\>hurt\. It is*not*a floor on fault\-class recall, which matters more than it sounds: a sparse, precise operator fails a recall floor by construction and is exactly what the endpoint wants\. The recall\-gated arm rejected our best operator for covering none of the 140 known faults the pool still missed \([AppendixD](https://arxiv.org/html/2608.18744#A4)\)\.

#### Composition by error independence\.

Admitted operators are composed by a vote: a candidate is flagged when at leastmmof thekkoperators in a chosen subsetSSflag it, withm=2m\{=\}2andk=\|S\|≤Kmax=6k=\|S\|\\leq K\_\{\\max\}\{=\}6unless stated otherwise\. The subset is the one maximising an objective computed on the labelled training samples alone, with no development split, no tuned threshold and no tie\-break, and unit weights rather than learned ones, which are hard to beat when the positive class is scarce\([griffin2025rrf](https://arxiv.org/html/2608.18744#bib.bib8)\)\. The objective was originallyJ=TP−FPJ=\\mathrm\{TP\}\-\\mathrm\{FP\}over*corroborated*flags, those on which the vote fires;[Section5\.3](https://arxiv.org/html/2608.18744#S5.SS3)showsJJworks*against*deployment and replaces it withTP−λ​FP\\mathrm\{TP\}\-\\lambda\\,\\mathrm\{FP\}atλ=nTP/nFP\\lambda=n\_\{\\mathrm\{TP\}\}/n\_\{\\mathrm\{FP\}\}read off the training split\.

## 5Results and analysis

Three questions: does the metric work on unseen tasks \([Section5\.1](https://arxiv.org/html/2608.18744#S5.SS1)\), what does each mechanism contribute when it is removed \([Section5\.2](https://arxiv.org/html/2608.18744#S5.SS2)\), and how good the composition step is once its whole search space can be scored exactly \([Section5\.3](https://arxiv.org/html/2608.18744#S5.SS3)\)\.

### 5\.1Main result

Table 1:The authored operator against the strongest hand\-written one and an LLM judge, on the 428 held\-out tasks and their 2592 samples\. % of headroom isΔ\\Deltaas a fraction of the0\.04240\.0424a perfect operator could add \([Section3](https://arxiv.org/html/2608.18744#S3)\);*calls*is model requests per candidate scored\. The three positive rows lie within0\.00050\.0005of one another onΔ\\Delta: what separates them is flags and cost\.[Table1](https://arxiv.org/html/2608.18744#S5.T1)compares single operators\. The loop’s is 55 lines of Python and reaches94\.2%94\.2\\%of the comparator’s effect with a quarter of its flags\. Its\+0\.0065\+0\.0065rises to\+0\.0481\+0\.0481over the 58 decidable tasks alone \([Table2](https://arxiv.org/html/2608.18744#S5.T2)\), and its flags overlap the closest hand\-written operator at Jaccard0\.2450\.245, so it is no rediscovery\. What it does reads in one sentence: keep the peers that pass the prompt\-visible asserts, run them and the candidate on perturbed inputs, and flag disagreement with the surviving plurality\. Every admitted level\-2 operator is a variant of that algorithm, since the loop rewards a new distinction rather than a new way to draw one \([Section6](https://arxiv.org/html/2608.18744#S6)\)\. The same table holds the contrast that motivates the loop: in a domain people find*easy*, the 15 hand\-written operators as one filter*lose*accuracy\.

#### A model judge on the same information\.

Asked one question per candidate under a level\-1 operator’s exact interface, and scored as a metric on the identical endpoint and null, the best of four judge configurations reaches\+0\.0070\+0\.0070\([Table1](https://arxiv.org/html/2608.18744#S5.T1)\), so we do not claim to beat it\. It is neither the same verdict nor the same cost\. The two flag sets meet at Jaccard0\.1050\.105, 27 flags of 257; each side’s exclusive flags are null alone \(\+0\.0010\+0\.0010atp=0\.068p\{=\}0\.068, and−0\.0007\-0\.0007\); and the 27 they agree on are worth\+13/−1\+13/\{\-\}1tasks, with the union above both parts at\+0\.0073\+0\.0073\. A checkable operator and a model judgement fail on different inputs, so their agreement is a high\-precision region neither reaches alone\. Cost separates them permanently: 47 authoring calls and nothing per candidate, against one call per candidate forever, 1762 for a single pass here\. Two asymmetries belong with that comparison: the author model judges at\+0\.0040\+0\.0040, below what it authors, and both benchmarks are public, so a judge may have memorised solutions where an operator cannot \([AppendixI](https://arxiv.org/html/2608.18744#A9)\)\.

#### A pool, not one operator\.

Eight runs differ only in the model’s sampling, so their spread is EvalCEGAR’s own sampling distribution: 6 of 8 admit an operator, all 6 are positive out of sample \(median\+0\.0029\+0\.0029, range\+0\.0009\+0\.0009to\+0\.0065\+0\.0065, 4 of 6 individually significant\), all 6 are at level 2, and all four fitting screens are clean on all 6\. Across the whole study the gate has admitted 26 operators, the smaller of the two pools[Section5\.3](https://arxiv.org/html/2608.18744#S5.SS3)composes over: on the held\-out tasks 25 are non\-harmful and 17 are individually significant against their own nulls, where chance predicts1\.31\.3, at a median of\+0\.0036\+0\.0036\. The 15 hand\-written operators are not such a pool: 9 flag anything at all, 5 are non\-harmful, and 1 is significant\. Admission reads the training split alone, so this is transfer\.

#### Transfer to an unseen benchmark\.

Two facts stand out \([Table2](https://arxiv.org/html/2608.18744#S5.T2)\)\. Of 13 rules measured on all three pools, 12 loop\-authored operators and the comparator, this one alone is positive*and*clears its own per\-pool null on all three, where the comparator reachesp<0\.05p<0\.05on HumanEval\+ alone\. And on HumanEval\+, the pool from a different task distribution, it matches the comparator exactly and beats it on parsimony: the same 10 helped tasks, the same single hurt task and the same\+0\.0125\+0\.0125, which is28\.8%28\.8\\%of that pool’s headroom, with 36 flags against 103\. Each pool was first checked to hold enough decidable tasks for a perfect operator to register, so a zero would mean absence, not no power\. Transfer also holds along a second axis, to 9320 candidates from three generators the composed rule never saw, at22\.1%22\.1\\%of that draw’s headroom against18\.9%18\.9\\%here \([AppendixG](https://arxiv.org/html/2608.18744#A7)\)\.

Table 2:Per\-pool transfer of the authored operator, each against its own shuffle null \(b=1000b\{=\}1000resamples\)\. HumanEval\+ differs from the authoring pool in oracle module, prompt style and difficulty, and nothing was re\-tuned\.\+/−\+/\-counts tasks helped and hurt; false alarms is the rate at which*correct*samples are flagged, authored / hand\-written\. Per\-pool headroom:0\.03920\.0392,0\.04290\.0429,0\.04330\.0433\.

### 5\.2Mechanism ablations

Table 3:The two mechanism ablations, each run inside the loop on its own candidates, over the same three seeds and the same training\-split labels, with only the named component changed\.*Calls*is the model’s authoring calls per seed, so neither ablation is a starved run\. No level\-1 attempt was ever admitted, in either arm or in the study’s 336 \([AppendixD](https://arxiv.org/html/2608.18744#A4)\)\.Removing either mechanism stops admission outright \([Table3](https://arxiv.org/html/2608.18744#S5.T3)\): each is necessary, not merely helpful\. The narrow\-only arm’s failure is not a budget artefact, because level 1’s own ceiling is below chance: scoring all 295 level\-1 operators the study authored, on the pooled 503 tasks with*no split and no selection cost*gives 9 atp<0\.05p<0\.05against 14\.75 expected by chance, so Benjamini–Hochberg returns*no discoveries*at a 10% false\-discovery rate\.

An independent search agrees on where that wall is\. Commissioning checks against a balanced labelled sample and admitting them for*predicting the label*, as the induction methods of[Section2](https://arxiv.org/html/2608.18744#S2)do, admits 2 of 48 candidates at level 1, whose\+0\.0016\+0\.0016its own null does not reject, and 6 of 16 at level 2 for\+0\.0054\+0\.0054under a training\-only combiner \([AppendixI](https://arxiv.org/html/2608.18744#A9)\)\. Two searches sharing nothing but the interface both fail at level 1 and both work at level 2, which also bounds what the recall\-floor row claims: an admission objective must not reward coverage, but predictive fit does not fail the way recall gating does\.

#### The gate ledger\.

Recording*every*gate consultation locates that wall exactly\. Admission needs three helped tasks; the narrow interface reaches two and stops there 55 times with no task hurt, and none of its 124 consultations reaches three, while the wider interface reaches three 6 times and every one was admitted \([Figure3](https://arxiv.org/html/2608.18744#A5.F3)\)\. The extension does not buy the*direction*, which the narrow interface finds easily; it buys*generality*, the third distinct task\. Targeting itself buys candidate viability rather than admissions: an operator abstains everywhere 34\.5% of the time undirected against 2\.1% directed \([AppendixC](https://arxiv.org/html/2608.18744#A3)\)\.

### 5\.3Composition, enumerated

Composition chooses a subset of size at most 6 from the loop’s admitted operators, with no hand\-written operator in either pool\. Both pools collect the gate’s admissions across the study’s authoring arms and seeds, 26 operators and 35, the larger containing the smaller, and they realise 17 and 20 distinct flag behaviours, giving 313,911 and 2,007,327 subsets, small enough to*enumerate*\. We scored every one on the held\-out endpoint through the same code used everywhere else \([Figure2](https://arxiv.org/html/2608.18744#S5.F2)\), so questions previously answered by inference become arithmetic and nopp\-value appears in this subsection\. Two terms recur: a subset is*vacuous*when its vote flags nothing, and the*oracle argmax*is the best held\-outΔ\\Deltain the enumeration, an upper bound no training\-data rule should reach\. Every result spans four*cells*: each pool at each of two pinned settings of the peer set a level\-2 operator reads\.

Figure 2:The composition search space, enumerated\.\(a,b\)Held\-outΔ\\Deltaof every≥2\\geq 2\-of\-kksubset \(k≤6k\\leq 6\) of the 26\- and 35\-operator pools, as a density, with C1–C7 of[Table4](https://arxiv.org/html/2608.18744#A6.T4), the frozen rule and the oracle argmax each at its exact percentile\. C7 was added after the fact\.\(c\)Spearman correlation betweenJJand the endpoint*within*each subset size on the larger pool: positive only atk=2k\{=\}2, so the misalignment is no size artefact\.#### The defect\.

JJranks its own argmax anywhere from the 4\.9th to the 85\.7th percentile of the space it searches across the four cells \(C1 in[Table4](https://arxiv.org/html/2608.18744#A6.T4)\), against an oracle argmax of roughly*twice*its delta \(\+0\.0300/\+0\.0314\+0\.0300/\{\+\}0\.0314againstJJ’s\+0\.0155/\+0\.0140\+0\.0155/\{\+\}0\.0140\), and its within\-kkcorrelation with the endpoint is positive only atk=2k\{=\}2, falling to−0\.338\-0\.338byk=6k\{=\}6\([Figure2](https://arxiv.org/html/2608.18744#S5.F2)c\)\. Enumeration also shows the cause is*bias, not variance*: refitting on more labelled data converges \(16–18 distinct selections atn=210n\{=\}210to 3–5 atn=1195n\{=\}1195\), and it converges*below*the*frozen rule*, the subset pinned under an earlier rule of thumb\. So freeze the selection rather than refitting it\.

#### The repair\.

We registered six selection criteria before running, each computed from the training split alone \(nTP=222n\_\{\\mathrm\{TP\}\}\{=\}222true andnFP=1183n\_\{\\mathrm\{FP\}\}\{=\}1183false flags\), ranked by the unchanged≥2\\geq 2\-of\-kkrule and scored against the exact landscape \([Table4](https://arxiv.org/html/2608.18744#A6.T4)\)\. Balanced accuracy is best of the six in all four cells, closing 39\.8–90\.2% of the gap to the oracle argmax, and it also beats label\-predictionF1F\_\{1\}, the criterion the induction baseline admits on, in all four\.

#### Why it works\.

Our registered prediction blamed aggregation, thatJJcounts per*sample*while the endpoint averages per*task*; but C2, the endpoint’s own formula, never reaches the 85th percentile, so that was not the defect\. What matters is the class weight: balanced accuracy is monotone inTP−0\.188​FP\\mathrm\{TP\}\-0\.188\\,\\mathrm\{FP\}, penalising a false alarm5\.33×5\.33\\timesless thanJJ’s 1:1, and C4 falls to the 1\.5th percentile or below\. That is the recall dominance of[Section3](https://arxiv.org/html/2608.18744#S3)resurfacing as a mis\-specified objective, and it means*refitting is safe once the objective is right*, which a self\-evolving library needs\. Two registered checks hold: a split\-half protocol re\-selects C6 in 4 of 4 cells, and on a 12\-point sweep HumanEval\+’s optimalλ\\lambda\-plateau contains the MBPP training ratio in 4 of 4 cells whileλ=1\\lambda\{=\}1never does, so the weight is a property of selection endpoints, not of those labels \([AppendixF](https://arxiv.org/html/2608.18744#A6)\)\.

#### Safety over the whole closure\.

Of the2,007,2922\{,\}007\{,\}292non\-vacuous subsets of the larger pool,*zero*harm the endpoint \(minimum\+0\.0007\+0\.0007\): admission makes composition safe over its*entire*combinatorial closure, exactly and with no power argument\. The converse holds on the 15 hand\-written operators, which admission accepts*none*of: the same combiner converges there to flagging nothing\.

## 6Negative results and limitations

#### Behavioural duplication\.

The pool repeats itself behaviourally rather than textually: no source\-text pair among 187 candidates reaches Jaccard0\.80\.8, yet the 183 that flag anything realise only 96 distinct flag sets \([Figure5](https://arxiv.org/html/2608.18744#A8.F5)\)\. Neither cure worked\. Targeting made it worse, the directed arm being the*least*diverse of three, and a novelty term at admission recovers a genuinely new operator in only 2 of the 11 candidates it blocks; handed the incumbent it duplicated, the model duplicates it again\. So the restriction sits in the operator prior, not at the gate, which argues for an explicit behavioural archive\([mouret2015mapelites](https://arxiv.org/html/2608.18744#bib.bib15)\)\.

#### Waste, not exhaustion\.

Nine of 14 admissions on three fresh seeds rediscovered an operator already selected, and a composition headline of\+0\.0076\+0\.0076on that pool did not survive the rediscovery screen we then added \([AppendixH](https://arxiv.org/html/2608.18744#A8)\)\. Rarefaction over behavioural classes\([chao1987estimating](https://arxiv.org/html/2608.18744#bib.bib3)\)still yields 2\.00 new classes at the sixth seed: wasteful, not exhausted\.

#### What the numbers do not show\.

EveryΔ\\Deltais measured against flagging nothing, never against another rule: no two rules separated in 9 paired tests\([wilcoxon1945individual](https://arxiv.org/html/2608.18744#bib.bib22)\), their differences living on 2–9 tasks, and the three loop operators that appeared to beat the comparator were rediscoveries of it\. A level\-2 verdict is also a function of the peer electorate: growing it threefold changes a verdict for all 35 operators, so their flags need a pinned electorate, though held\-outΔ\\Deltamoves up rather than down \([AppendixG](https://arxiv.org/html/2608.18744#A7)\)\. What admission discards is unmeasured \([AppendixC](https://arxiv.org/html/2608.18744#A3)\)\. Both benchmarks come from one distribution family, so an out\-of\-sample*task list*needs a third oracle port\. And the loop is not the only route to an operator of this quality: a judge ties its delta, so what it buys is a complementary error profile, not a higher ceiling\.

## 7Conclusion

An evaluation metric can be authored automatically, and the artefact is small enough to read: 55 lines of Python, no weights, closing15\.4%15\.4\\%of the distance to a perfect operator on 428 unseen tasks\. It asks with a counterexample rather than a prompt, and three rules behind it carry over to any pool of executable checks:

- •Admit on the decision, not on coverage\.That change alone moved admissions from none to one per seed\.
- •When nothing resolves a counterexample, widen the interface\.336 attempts at the narrow interface admit none; the wider one admits six\.
- •On a selection endpoint, weight the classes\.TP−\(nTP/nFP\)​FP\\mathrm\{TP\}\-\(n\_\{\\mathrm\{TP\}\}/n\_\{\\mathrm\{FP\}\}\)\\mathrm\{FP\}, read off the training labels, reaches the top decile of a space whose precision\-flavoured objective sits in the bottom2%2\\%\.

Where unit tests exist, run the unit tests\. What remains is the setting that motivated the method, where an unresolved counterexample is the only evidence that a metric’s vocabulary, and so the claim it can support, has run out\.

## References

- \[1\]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, et al\.Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732, 2021\.
- \[2\]Yoav Benjamini and Yosef Hochberg\.Controlling the false discovery rate: a practical and powerful approach to multiple testing\.Journal of the Royal Statistical Society: Series B, 57\(1\):289–300, 1995\.
- \[3\]Anne Chao\.Estimating the population size for capture\-recapture data with unequal catchability\.Biometrics, 43\(4\):783–791, 1987\.
- \[4\]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, et al\.Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374, 2021\.
- \[5\]Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou\.Teaching large language models to self\-debug\.InInternational Conference on Learning Representations \(ICLR\), 2024\.
- \[6\]Koen Claessen and John Hughes\.QuickCheck: A lightweight tool for random testing of Haskell programs\.InInternational Conference on Functional Programming \(ICFP\), pages 268–279, 2000\.
- \[7\]Edmund Clarke, Orna Grumberg, Somesh Jha, Yuan Lu, and Helmut Veith\.Counterexample\-guided abstraction refinement\.InComputer Aided Verification \(CAV\), pages 154–169, 2000\.
- \[8\]Ben Griffin, Aaron Ontoyin Yin, Diego Vidaurre, Ugur Koyluoglu, et al\.Random Rule Forest \(RRF\): Interpretable and manageable ensembles of LLM\-generated questions for predicting success from unstructured data\.arXiv preprint arXiv:2505\.24622, 2025\.
- \[9\]Shengran Hu, Cong Lu, and Jeff Clune\.Automated design of agentic systems\.arXiv preprint arXiv:2408\.08435, 2024\.
- \[10\]Chin\-Yew Lin\.ROUGE: A package for automatic evaluation of summaries\.InText Summarization Branches Out, pages 74–81, 2004\.
- \[11\]Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang\.Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation\.InAdvances in Neural Information Processing Systems \(NeurIPS\), 2023\.
- \[12\]Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu\.G\-Eval: NLG evaluation using GPT\-4 with better human alignment\.InConference on Empirical Methods in Natural Language Processing \(EMNLP\), 2023\.
- \[13\]Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, et al\.Self\-refine: Iterative refinement with self\-feedback\.InAdvances in Neural Information Processing Systems \(NeurIPS\), 2023\.
- \[14\]William M\. McKeeman\.Differential testing for software\.Digital Technical Journal, 10\(1\):100–107, 1998\.
- \[15\]Jean\-Baptiste Mouret and Jeff Clune\.Illuminating search spaces by mapping elites\.arXiv preprint arXiv:1504\.04909, 2015\.
- \[16\]Kishore Papineni, Salim Roukos, Todd Ward, and Wei\-Jing Zhu\.BLEU: a method for automatic evaluation of machine translation\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\), pages 311–318, 2002\.
- \[17\]Pouya Pezeshkpour and Estevam Hruschka\.AutoPyVerifier: Learning compact executable verifiers for large language model outputs\.arXiv preprint arXiv:2604\.22937, 2026\.
- \[18\]Bernardino Romera\-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, et al\.Mathematical discoveries from program search with large language models\.Nature, 625:468–475, 2024\.
- \[19\]William F\. Shen, Xinchi Qiu, Chenxi Whitehouse, Lisa Alazraki, et al\.Rethinking rubric generation for improving LLM judge and reward modeling for open\-ended tasks\.arXiv preprint arXiv:2602\.05125, 2026\.
- \[20\]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao\.Reflexion: Language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\), 2023\.
- \[21\]Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, et al\.Large language models are not fair evaluators\.arXiv preprint arXiv:2305\.17926, 2023\.
- \[22\]Frank Wilcoxon\.Individual comparisons by ranking methods\.Biometrics Bulletin, 1\(6\):80–83, 1945\.
- \[23\]Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q\. Weinberger, and Yoav Artzi\.BERTScore: Evaluating text generation with BERT\.InInternational Conference on Learning Representations \(ICLR\), 2020\.
- \[24\]Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, et al\.Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track, 2023\.

## Appendix ARequest scope, screens and nulls

The request handed to the author covers a whole fault kind rather than the two colliding samples, because a two\-sample request is satisfiable by a lookup table: an earlier round of this work measured exactly that, with operators firing on a handful of training faults and on none of a held\-out class\. The two kinds are faults that already fail the visible checks and faults that pass them and fail only the hidden suite, and the request covers whichever kind the colliding sample belongs to\.

The subtlest of the four screens of[Section3](https://arxiv.org/html/2608.18744#S3)is the prompt\-keyword one: an operator that branches on a keyword answers from the task’s identity rather than from the candidate in front of it, which is oracle reconstruction by dispatch table\.

Level 2’sctxexposes the candidate’s peers, a runner, the prompt\-visible asserts and a generator of inputs, and nothing that reads the hidden suite\. Inputs come from perturbing the arguments of the prompt\-visible asserts, which the solver also saw, with the number of unperturbed ones exposed so that an operator can tell the two apart\. An operator that exceeds the observation cap abstains rather than stalling\.

The shuffle null of[Section3](https://arxiv.org/html/2608.18744#S3)usesb=1000b\{=\}1000resamples throughout, seeded so that a rerun reproduces them exactly\.

## Appendix BThe transfer budget

Because oracle labels enter only on the training split, the cost of moving EvalCEGAR to an oracle\-free domain is the cost of that split’s labels, and it must be quoted in*decidable*examples: an example informs the loop only if it carries both a good and a bad candidate that survive the visible checks\. Of our 75 training tasks 10 are decidable, giving 80 \(correct, wrong\) pairs and four distinguishable levels of the training objective\. That is demonstrably enough to*admit*operators that transfer \(\+0\.0065\+0\.0065\) and demonstrably*not*enough to tune one scalar: a≥4\\geq 4\-of\-26 vote is worth\+0\.0087\+0\.0087held\-out, and neither trainingΔ\\Deltanor cross\-validation inside training can distinguish it from the union\.

Three measurements complete the budget\.*\(i\)*The crossover for tuning aggregation is 20–30 decidable tasks: below it the choice collapses to the union in 9 of 9 directions; above it, rotating a held\-back fold recovers\+0\.0077\+0\.0077\(p≤0\.0004p\\leq 0\.0004, identical atm=2,3,4m\{=\}2,3,4\), 57% of the oracle’s gain\.*\(ii\)*A better gate removes that requirement rather than paying it: 9 operators from three seeds of the target\-only gate \([AppendixC](https://arxiv.org/html/2608.18744#A3)\) score\+0\.0071\+0\.0071\(p=0\.0022p\{=\}0\.0022\) as a plain union with no threshold to set, so gate quality and aggregation are substitutes\.*\(iii\)*Composition changes the currency altogether, and this is the cheapest line item: error independence needs*labelled flags*, not decidable tasks, and the same training split holds 426 of them\. The≥2\\geq 2\-of\-kkrule matches*\(i\)*’s rotated selection with 5 operators instead of 26 and nothing to tune, at a measured price of∼\\sim60 labelled candidates \(at half that sample it degrades to\+0\.0065/\+0\.0033\+0\.0065/\{\+\}0\.0033\)\. So the budget line for a new domain is∼\\sim10 decidable examples to author and admit plus∼\\sim60 labelled candidates to decide which admitted operators vote, and the second number is the one a practitioner can actually produce, because it does not require a single example to hold both a correct and an incorrect candidate\.

## Appendix CTargeting versus accepting

An early ablation on a synthetic bank favoured the collision 4 admissions to 0, with 31 of 36 undirected candidates rejected as*untargeted*\. That did not replicate on the natural bank and we discard the claim it licensed\. A later single\-flag experiment settles the division of labour instead: keeping the collision as the*target*but dropping it as the*acceptance test*raises distinct\-metric yield from0\.380\.38to1\.671\.67per seed \(4\.4×4\.4\\times\) with peak quality unchanged \(\+0\.0063\+0\.0063vs\+0\.0065\+0\.0065\), and moves the binding rejection stage to the objective \(60\.3% of rejections, up from 28\.6%\)\. So the collision is doing two separable jobs, and only the first is worth its cost\. What the gate discards is unmeasured in the other direction too: one operator it turns down for flagging nothing in training is our most reliable one held out\.

## Appendix DThe mechanism ablations in full

Both arms run inside the loop on the loop’s own candidates, so neither is a re\-scoring of operators authored under the other\. Both read the same training\-split labels, and no gate call in either arm hit its abort guard\. The floor the recall\-gated arm applies asks an operator to fire on at least 30% of the 140 fault classes the pool still misses, at no more than a 2% false\-alarm rate on the samples that must pass\. The narrow\-only arm was also given double the level\-1 attempts per target, 48 per seed against the full loop’s 24, and pooling its 144 with every level\-1 attempt the eight full runs made gives the 336\.

## Appendix EThe gate ledger in full

Of the 336 level\-1 attempts of[Table3](https://arxiv.org/html/2608.18744#S5.T3), the 124 that survived every earlier screen are the ones the gate saw and[Figure3](https://arxiv.org/html/2608.18744#A5.F3)draws; a target is attempted at the narrow interface first, so most of an escalating run’s consultations are narrow ones and reading a run’s ledger as one interface would over\-attribute them\. Each of the six operators admitted across the eight runs has a recorded level\-1 rejection immediately before it\. The ledger doubles as the sensitivity analysis for the admission floor: at helped≥2\\geq 2the narrow interface would admit 55 of its 124 consultations, so the floor is what an interface has to clear, and what those admissions would transfer to is unmeasured\.

Figure 3:Every consultation of the admission gate, not just the ones that ended in an admission, by how many tasks the candidate helped on the training split the gate reads\. Admission needs three, and only the wider interface ever gets there, whether or not the narrow run could escalate\.
## Appendix FThe selection criterion: registration, robustness, and the class weight

All six criteria, their formulas and their predicted ranking were written down before the landscape was scored, so[Table4](https://arxiv.org/html/2608.18744#A6.T4)reports where each choice lands rather than which choice was searched for\.

Table 4:Six a\-priori criteria, and one added afterwards, against the enumerated landscape of[Figure2](https://arxiv.org/html/2608.18744#S5.F2)\. Cells run \(26\-operator pool, 35\-operator pool\) at each peer\-set setting\. C6 is the rule we adopt, and nothing is tuned:0\.18770\.1877is the training split’s class rationTP/nFPn\_\{\\mathrm\{TP\}\}/n\_\{\\mathrm\{FP\}\}\. C7 is the induction baseline’s own admission signal and the only row not registered in advance\.Both registered robustness checks behind[Section5\.3](https://arxiv.org/html/2608.18744#S5.SS3)are reported here in full\.*\(i\) Split\-half\.*Choosing the criterion on half A of the held\-out tasks and reporting on half B, with the split fixed in the registration, re\-selects C6 in 4 of 4 cells at the 85\.2–98\.2nd percentile of half B’s own landscape, againstJJ’s 9\.3–82\.4th\.*\(ii\) Per\-benchmark*, which is also what makes HumanEval\+ the criterion’s best pool\. The endpoint decomposes exactly by task, so the same held\-out tasks re\-partition by provenance at zero cost, recombining to the reported landscape maximum to 6 decimal places\. On HumanEval\+, which shares only the operator pool with the training split, C6 ranks at the 95\.9 / 97\.6 / 95\.9 / 97\.5th percentile, its best pool of the three, beatingJJin 4 of 4 cells and separating from it in a paired test in 3 of 4\. Reading the same 428 tasks three ways adds no independent evidence for the overall comparison, but it does rule out the gain being an artefact of the benchmark the labels came from\.

#### The sweep \([Figure4](https://arxiv.org/html/2608.18744#A6.F4)\)\.

λ=nTP/nFP\\lambda=n\_\{\\mathrm\{TP\}\}/n\_\{\\mathrm\{FP\}\}is read off the MBPP training split, so the question is whether it is a property of those labels or of selection endpoints generally\. Twelve values ofλ\\lambdawere scored in each of the four cells of[Table4](https://arxiv.org/html/2608.18744#A6.T4)\. HumanEval\+’s optimal plateau contains the training\-split ratio in 4 of 4 cells,λ=1\\lambda\{=\}1lies strictly outside it every time, andλ=0\\lambda\{=\}0, which ignores false alarms altogether, is never strictly better\.

Figure 4:Sweepingλ\\lambdainTP−λ​FP\\mathrm\{TP\}\-\\lambda\\,\\mathrm\{FP\}, one line per held\-out pool: HumanEval\+’s optimal plateau \(shaded\) containsλ=nTP/nFP=0\.1877\\lambda\{=\}n\_\{\\mathrm\{TP\}\}/n\_\{\\mathrm\{FP\}\}\{=\}0\.1877, read off the MBPP\+ training split, and excludesJJ’sλ=1\\lambda\{=\}1\.

## Appendix GPeer\-dependent verdicts

A level\-2 operator readsctx, which is built from every raw sample for the task, so its verdict is a function of that peer electorate and not of the candidate alone\. The two effects were measured apart\. Against a frozen electorate, 0 of 35 operators change a single verdict, held\-out or training, an exact identity we check rather than assume, and one every number here relies on\. Against an electorate grown from 3424 to 12744 samples, all 35 change at least one, 1996 flags added and 440 removed, one operator more than doubling its own flag count on rows whose code never changed\. So the wider interface buys discriminating power at the cost of referential stability, and a deployed level\-2 operator has to pin its electorate the way we pin ours\.

The direction of the change is what a conformity objection has to answer, and it is favourable\. Every frozen selection gains\+0\.0009\+0\.0009to\+0\.0036\+0\.0036under the grown electorate with no material reordering, so the operators are reading signal out of peer context rather than tracking consensus for its own sake\. On the same draw the five\-operator rule of[Section5\.3](https://arxiv.org/html/2608.18744#S5.SS3), selected on the training split and never re\-selected, scores\+0\.0220\+0\.0220\(z=8\.34z\{=\}8\.34,p=0\.0010p\{=\}0\.0010,\+79/−12\+79/\{\-\}12tasks\) on 9320 candidates from three generators it was never fitted to\. Raw deltas are not comparable across the two draws, because the deeper one is roomier \(headroom0\.09950\.0995against0\.04240\.0424\), which is why both are quoted as a fraction of their own headroom:22\.1%22\.1\\%against18\.9%18\.9\\%\. The operators are out of sample there; the task list is not\.

## Appendix HRedundancy and the diversity veto

[Figure5](https://arxiv.org/html/2608.18744#A8.F5)holds the redundancy evidence behind[Section6](https://arxiv.org/html/2608.18744#S6)\. The veto itself was run, not merely proposed: under the fold rotation of[AppendixB](https://arxiv.org/html/2608.18744#A2)it is at or below the plain union \(\+0\.0065/\+0\.0057/\+0\.0047\+0\.0065/\{\+\}0\.0057/\{\+\}0\.0047atm=2,3,4m\{=\}2,3,4, harming9/11/119/11/11tasks against the union’s 8\), and development folds disable it in 5 of 9 directions\. On the rediscovery\-heavy pool of[Section6](https://arxiv.org/html/2608.18744#S6)the frozen rule holds on the 5 novel operators \(\+0\.0040\+0\.0040\)\. It buys parsimony, 7 operators for 26 operators’ effect, rather than quality\. It vetoes*flag\-set*overlap at admission, which makes it a weaker instrument than the behavioural\-novelty gate of[Section6](https://arxiv.org/html/2608.18744#S6), and the two fail differently\.

Figure 5:\(a\)Exact rarefaction over all 63 seed subsets and 720 arrival orders, not a bootstrap: the 23 operators admitted across those six seeds fall into 16 distinct behavioural classes, and the curve does not saturate\.\(b\)The indicated remedy, run live: what became of each of the 11 candidates a behavioural\-novelty term at admission blocks\. The strip below reads redundancy at three layers, textual, behavioural and strategic, giving for each the distinct classes, the share of pairs at Jaccard≥0\.8\{\\geq\}0\.8and the largest class\.
## Appendix IThe judge and induction baselines

#### The judge, in four configurations\.

The judge sees the task statement, the prompt\-visible asserts and the candidate source, which is exactly the level\-1 interfaceop\(task, code\), and never the hidden suite\. Only the 1762 distinct \(task, candidate\) pairs that pass the visible checks are judged, because those are the only ones the endpoint scores\. The configurations cross two models, the one that authors operators and the cheaper Claude Haiku 4\.5, which is also one of the candidate generators of[Section3](https://arxiv.org/html/2608.18744#S3), so the cheap judge scores some of its own output; and two prompts: a one\-word verdict, and a reason\-then\-verdict form whose wording states the training split’s own class balance, so the judge is given no statistic the operators are denied\. Replies are cached by \(pool, task, source hash\) and a reply naming no verdict flags nothing, which is the conservative reading; rescoring the caches twice reproduces byte\-identical output\.

Table 5:The judge as a metric, on the endpoint and null of[Table1](https://arxiv.org/html/2608.18744#S5.T1)\. Precision, recall and accuracy read the judge as a*classifier*of the same pairs, which is a different question from the endpoint delta\. The last two rows are the same two references as[Table1](https://arxiv.org/html/2608.18744#S5.T1)\.In[Table5](https://arxiv.org/html/2608.18744#A9.T5), reasoning before answering raises the cheap model’s delta and lowers its flag count, and both author\-model configurations sit at roughly half that delta\. As a classifier the author model is the more precise \(0\.6770\.677against0\.4030\.403\) and the less useful, which is the recall dominance of[Section3](https://arxiv.org/html/2608.18744#S3)again: a precise judge that says nothing on most bad candidates cannot change what is left to draw from\. No reply in either reason\-then\-verdict configuration failed to name a verdict; the one\-word configurations lost 1 of 1762 and 5 of 1762, and an unnamed verdict flags nothing\.

#### Overlap and cost\.

[Table6](https://arxiv.org/html/2608.18744#A9.T6)scores the two flag sets and their Boolean combinations through the same code\. Read the overlap before the union delta: a union that beats both parts is only interesting when the parts disagree, and here they do\.

Table 6:The loop\-authored operator and the best judge configuration as flag sets, on the same endpoint\. Jaccard between the two is0\.1050\.105: 27 flags in common of 257\.Cost comes from the call ledgers rather than an estimate\. The run that wrote the operator made 47 calls, and every candidate it has scored since cost nothing; all eight runs together made 376\. The judge costs one call per candidate with nothing amortised, so a single pass over this endpoint is 1762 calls,37\.5×37\.5\\timesthe run that produced the operator or4\.7×4\.7\\timesall eight runs including the two that admitted nothing, and every future candidate costs that again\.

#### Verifier induction\.

Same author model, same system prompts, same four screens in the same order, same training split, endpoint and null\. Two things change deliberately: candidates are commissioned against a balanced random labelled sample rather than against a signature collision, and they are admitted for predicting the*label*, at least 3 true positives at precision at least0\.50\.5, which are the label\-space analogues of the deployed gate’s three helped tasks and helped\>\>hurt\.[Table7](https://arxiv.org/html/2608.18744#A9.T7)reports both interfaces under three ways of combining whatever was admitted\.

Table 7:Verifier induction at each interface\. Combiners are the plain union of admitted checks, a≥2\\geq 2\-of\-kkvote, and the single check with the best*training*F1F\_\{1\}; only the last is a rule a practitioner could apply without held\-out labels\.Level 1 fails the way our own level\-1 search fails: 48 calls, 2 admissions, one run admitting nothing, and no combination separating from its null\. Level 2 works\. The screens fired on this baseline as they do on our own author path, rejecting candidates for source\-pattern fitting and for prompt dispatch, so it was given no exemption our method does not have\. Its best*single*check reaches\+0\.0074\+0\.0074, above both the comparator and our operator, but that check is chosen by reading held\-out deltas and is not a legitimate headline\. The legitimate reading is the warning: on this endpoint the ceiling reachable by predictive\-fit admission at the wider interface is at least as high as the one we report\.

#### Label prediction as a composition criterion\.

The same criterion swap can be made with no model and no sampling at all, because[Section5\.3](https://arxiv.org/html/2608.18744#S5.SS3)’s enumeration puts every subset in memory, so a criterion is a total order over a finite set and its choice, delta and percentile are counts rather than estimates\. CarryingJJas the instrument check, its argmax reproduces the published one in all four cells\.

Table 8:Held\-outΔ\\Deltaand exact landscape percentile of four criteria across the four cells of[Table4](https://arxiv.org/html/2608.18744#A6.T4)\. C8 is precision under the same floor of 3 true positives, and it is degenerate with the floor in place: it picks the same two\-operator set flagging ten tasks in every cell\.[Table8](https://arxiv.org/html/2608.18744#A9.T8)adds two criteria to the six of[Table4](https://arxiv.org/html/2608.18744#A6.T4), and one of them wins againstJJ: C7 beats it in three cells, ties it in the fourth, and does not collapse when the pool grows from 26 operators to 35, whichJJdoes\. So the gap[Section5\.3](https://arxiv.org/html/2608.18744#S5.SS3)measures is partly recoverable by the criterion the related work already uses, and C6 recovers more of it in all four cells\. C7 selects a different subset fromJJeach time, at Jaccard0\.220\.22to0\.670\.67againstJJ’s choice, so this is not two names for one ranking\.