Meta-Reflective Evolution Effectively Improves Production AI Systems

# Meta-Reflective Evolution Effectively Improves Production AI Systems
Source: [https://arxiv.org/html/2610.12794](https://arxiv.org/html/2610.12794)
Chang Zeng††thanks:Work done during internship at Snowflake\.Affiliation:UMass AmherstAditi SomayajulaAffiliation:StanfordSahithi AnkireddyAffiliation:StanfordRohan KosalgeAffiliation:UC BerkeleyAvishree KhareAffiliation:UPennJohn MitchellAffiliation:SnowflakeAffiliation:StanfordAditya G\. ParameswaranAffiliation:SnowflakeAffiliation:UC BerkeleyRajhans SamdaniAffiliation:SnowflakeAnupam DattaAffiliation:Snowflake

###### Abstract

Large Language Model \(LLM\)\-guided evolutionary optimization has successfully improved prompts, programs, and even entire AI systems\. Most optimization methods select a candidate from the search frontier, then mutate it in one LLM pass based only on its failures\. This focus on the frontier can stall on production AI systems, where isolated failures reveal little about complex, hidden interactions\. We introduce CoCoEvolve \(CortexCodeEvolve\), an evolutionary optimizer built around what we call*meta\-reflection*, wherein an agentic mutator \(Cortex Code\) selectively retrieves evidence from the entire evolutionary history and probes the live system before verifying and proposing mutations\. We apply CoCoEvolve to optimize three classes of production AI systems: data agents, LLM\-powered data pipelines, and data engineering workflows\. On DABStep, CoCoEvolve tops the public leaderboard, reaching 92\.8% held\-out hard accuracy compared to 78\.4% for Meta\-Harness and 30\.4% for GEPA under matched conditions\. Across six LLM data pipeline workloads, it outperforms every evaluated baseline\. On data\-eng\-bench, CoCoEvolve lifts test\-set solve rate from 59\.1% to 77\.4%, past the best baseline of 73\.6%\. It also has 11pp higher accuracy than GEPA on HotpotQA while remaining competitive with specialized optimizers on established mathematical and systems benchmarks\. Matched ablations confirm that each component of CoCoEvolve contributes to these gains, demonstrating the value of meta\-reflection for optimizing production AI systems\.

## 1Introduction

Large language models \(LLMs\) are increasingly used as semantic mutation operators: given a candidate prompt or program and evaluation feedback for that candidate, an LLM proposes a revised candidate\. This approach, exemplified by evolutionary optimizers like AlphaEvolve and OpenEvolve, has led to breakthrough improvements in prompts, algorithms, and other text\-based artifacts because LLMs can make semantically meaningful changes that conventional random mutation cannot\([Novikov et al\., 2025](https://arxiv.org/html/2610.12794#bib.bib23);[Sharma, 2025](https://arxiv.org/html/2610.12794#bib.bib26)\)\. However, most LLM\-guided evolutionary optimizers select a single parent, use its failures to propose a new candidate in one LLM call, and only then evaluate the result\.

This parent\-local, one\-shot interface is effective when the failure makes both the defect and its repair apparent\. It falls short for production AI systems, where isolated failures can reveal little about interacting causes and relevant evidence may be distributed across the search history\. First, the selected parent’s failure may not reveal the root cause, and a locally beneficial edit may regress capabilities elsewhere in the search history\. Second, the mutator cannot interact with the target environment, test hypotheses, or revise its candidate in response to intermediate results\. Search can consequently trade one solved task for another, revisit exhausted strategies, or end up plateauing\([Zhou et al\., 2026](https://arxiv.org/html/2610.12794#bib.bib38)\)\.

We introduce CoCoEvolve \(CortexCodeEvolve\), an evolutionary optimizer built aroundmeta\-reflection: \(i\) selectively retrieving evidence from the accumulated search history beyond the selected parent’s failure and \(ii\) actively probing the target environment while constructing and validating a mutation\. To support meta\-reflection, CoCoEvolve uses Cortex Code as its coding\-agent mutator\([Snowflake Inc\., 2026b](https://arxiv.org/html/2610.12794#bib.bib29)\)\. The mutator can inspect the parent, run focused experiments, revise its implementation, and validate the result before committing a child\. It accesses evolutionary history through both a compact summary and structured queries, allowing it to retrieve prior programs that solved a specific task, inspect the mutation that restored a lost capability, or identify related tasks that have historically regressed together\.

Figure[1](https://arxiv.org/html/2610.12794#S1.F1)illustrates meta\-reflection for a DABStep data agent\([Egg et al\., 2025](https://arxiv.org/html/2610.12794#bib.bib14)\), where payment\-processing rules must be inferred across documentation and live data\. Here, a one\-shot, parent\-local mutation doesn’t make progress, whereas CoCoEvolve’s meta\-reflective mutation retrieves the missing evidence, validates the corresponding repair, and succeeds\. The same behavior repeats with*Lockpicking with Interference*, a synthetic stress test where every leading baseline plateaus below perfect accuracy within 50 iterations, while CoCoEvolve reaches 100% in 6 iterations \(Appendix[G\.1](https://arxiv.org/html/2610.12794#A7.SS1)\)\.

Figure 1:Left:given under\-determined feedback, the parent\-local mutator guesses a repair in one call and makes no progress\.Right:a meta\-reflective mutator queries the evolutionary history and reads the trace of a previously evaluated program to determine that*all matching fees stack*\. It also reads documentation to refine which fees match, and synthesizes the two observations\. It tests the resulting hypothesis before commit and raises training accuracy from28/30028/300to95/30095/300\.We evaluate CoCoEvolve primarily on production AI systems, where correctness depends on interactions with live systems rather than prompt text alone\. These include agent specifications coupled to tools and semantic data models, SQL pipelines that invoke LLMs to process textual data\([Liskowski et al\., 2025](https://arxiv.org/html/2610.12794#bib.bib19)\), and data\-engineering workflows\([dbt Labs, 2026](https://arxiv.org/html/2610.12794#bib.bib11)\)\. We additionally evaluate standard prompt, mathematical, and systems optimization benchmarks from prior work to test whether the method remains competitive when interaction with the environment is limited or unnecessary\.

CoCoEvolve optimizes an agent specification on DABStep\([Egg et al\., 2025](https://arxiv.org/html/2610.12794#bib.bib14)\), reaching 92\.8% held\-out hard accuracy, versus 78\.4% for an interactive Meta\-Harness baseline and 30\.4% for GEPA\([Lee et al\., 2026](https://arxiv.org/html/2610.12794#bib.bib18);[Agrawal et al\., 2026b](https://arxiv.org/html/2610.12794#bib.bib2)\); it also tops the validated public leaderboard\.111At submission, this score has not been externally validated; we have contacted the DABStep maintainers but have not received a response\.CoCoEvolve also improves over the strongest evaluated baseline on six LLM\-powered data processing pipeline workloads by up to 10 pp\. On held\-out HotpotQA, it gains 11 pp over GEPA and 43 pp over the seed\([Yang et al\., 2018](https://arxiv.org/html/2610.12794#bib.bib34);[Agrawal et al\., 2026b](https://arxiv.org/html/2610.12794#bib.bib2)\)\. On data engineering, it achieves the strongest overall performance among the evaluated evolutionary methods and leads on held\-out data\-eng\-bench\([Snowflake AI Research, 2026](https://arxiv.org/html/2610.12794#bib.bib27)\), solving 77\.4% of tasks\. Across mathematical and systems optimization objectives, it is competitive with leading evolutionary methods such as AdaEvolve\([Cemri et al\., 2025](https://arxiv.org/html/2610.12794#bib.bib4)\), with the largest improvements occurring on objectives that exhibit extended search plateaus\. Matched ablations confirm that each component of CoCoEvolve contributes to these gains\.

## 2CoCoEvolve

We introduce CoCoEvolve, an evolutionary optimizer built around*meta\-reflection*\(Section[2\.2](https://arxiv.org/html/2610.12794#S2.SS2)\)\. Figure[2](https://arxiv.org/html/2610.12794#S2.F2)summarizes its evolutionary loop and Algorithm[1](https://arxiv.org/html/2610.12794#alg1)in Appendix[C](https://arxiv.org/html/2610.12794#A3)specifies it in full\. Our implementation instantiates the coding\-agent mutator with Cortex Code\.

Figure 2:CoCoEvolve proposes a new candidate in each iteration by selecting a parent from the archive, curating evidence from the search history to include in the prompt, and equipping a coding\-agent mutator with on\-demand tools for querying prior lineages and the live target environment\. A coding agent investigates, edits, and self\-tests a child; the evaluator scores it, records the resulting observations in the history, and, under the default MAP\-Elites policy, either discards it or admits it to the archive if it improves its niche\. This loop combines compact prompt\-visible evidence with environment feedback while preserving diverse programs for future selection\.### 2\.1Problem Formulation

We treat the object of optimization as a*program*pp— code, a prompt, a configuration, an agent specification, or any other text\-representable artifact — drawn from a program space𝒫\\mathcal\{P\}\. Let𝒯\\mathcal\{T\}denote the task space and𝒟\\mathcal\{D\}a distribution over it\. An evaluator

f:𝒫×𝒯→ℝ×𝒪,f⁡\(p,τ\)=\(s⁡\(p,τ\),o⁡\(p,τ\)\)f:\\mathcal\{P\}\\times\\mathcal\{T\}\\to\\mathbb\{R\}\\times\\mathcal\{O\},\\qquad f\(p,\\tau\)=\\big\(s\(p,\\tau\),\\ o\(p,\\tau\)\\big\)returns both a scalar scores⁡\(p,τ\)∈ℝs\(p,\\tau\)\\in\\mathbb\{R\}and an*observation*o⁡\(p,τ\)∈𝒪o\(p,\\tau\)\\in\\mathcal\{O\}: the full record produced while computing that score, including tool calls, intermediate outputs, executed queries, and error messages\. The optimization goal is to findp∗=arg⁡maxp∈𝒫⁡Φ⁡\(p\)p^\{\\ast\}=\\arg\\max\_\{p\\in\\mathcal\{P\}\}\\Phi\(p\)withinTTiterations, whereΦ⁡\(p\)=𝔼τ∼𝒟​\[s⁡\(p,τ\)\]\\Phi\(p\)=\\mathbb\{E\}\_\{\\tau\\sim\\mathcal\{D\}\}\[s\(p,\\tau\)\]represents the aggregate fitness of a program\.

CoCoEvolve pursues this goal via an iterative evolutionary loop\. After iterationt−1t\-1, it maintains an archive𝒜t−1\\mathcal\{A\}\_\{t\-1\}of previously evaluated candidate programs\. At iterationtt, the harness selects a parent programpt′∈𝒜t−1p^\{\\prime\}\_\{t\}\\in\\mathcal\{A\}\_\{t\-1\}and produces a childptp\_\{t\}\. Most prior methods condition mutation only on the selected parentpt′p^\{\\prime\}\_\{t\}and its observations; CoCoEvolve instead uses thefull search historyaccumulated so far, denoted by

ℋt−1=\{\(pi,𝒯i,Si,Oi\)\}i<t,\\mathcal\{H\}\_\{t\-1\}=\\\{\(p\_\{i\},\\mathcal\{T\}\_\{i\},S\_\{i\},O\_\{i\}\)\\\}\_\{i<t\},Here,𝒯i⊆𝒯\\mathcal\{T\}\_\{i\}\\subseteq\\mathcal\{T\}is the subset selected to evaluatepip\_\{i\}, withSi:=\{s⁡\(pi,τ\)∣τ∈𝒯i\}S\_\{i\}:=\\\{s\(p\_\{i\},\\tau\)\\mid\\tau\\in\\mathcal\{T\}\_\{i\}\\\}andOi:=\{o⁡\(pi,τ\)∣τ∈𝒯i\}O\_\{i\}:=\\\{o\(p\_\{i\},\\tau\)\\mid\\tau\\in\\mathcal\{T\}\_\{i\}\\\}denoting the resulting sets of scores and observations\. This history supports the mutation policy developed next: context curation exposes selected evidence fromℋt−1\\mathcal\{H\}\_\{t\-1\}, environment augmentation provides tools for retrieving additional evidence on demand, and the coding\-agent mutator uses both to construct the child\.

### 2\.2Meta\-Reflection

We define*meta\-reflection*as reasoning beyond one parent’s observations by selectively retrieving historical evidence from the accumulated search history and actively acquiring feedback from the target environment\.

Formally, a meta\-reflective mutator receives two complementary inputs\. First, a compact context bundlectc\_\{t\}, curated from the search history, is injected directly into the mutator’s prompt\. Second, an environmentℰt\\mathcal\{E\}\_\{t\}provides tools that the mutator invokes on demand\. It augments a base environmentℰ0⊆ℰ\\mathcal\{E\}\_\{0\}\\subseteq\\mathcal\{E\}of execution, validation, and target\-system tools with selective access to the search history\.

ct=Curate⁡\(ℋt−1\)∈𝒞,ℰt=Augment⁡\(ℰ0,pt′,ℋt−1\)⊆ℰ\.c\_\{t\}=\\mathrm\{Curate\}\(\\mathcal\{H\}\_\{t\-1\}\)\\in\\mathcal\{C\},\\qquad\\mathcal\{E\}\_\{t\}=\\mathrm\{Augment\}\(\\mathcal\{E\}\_\{0\};p^\{\\prime\}\_\{t\},\\mathcal\{H\}\_\{t\-1\}\)\\subseteq\\mathcal\{E\}\.A coding\-agent mutator then performs meta\-reflection by reasoning overctc\_\{t\}and gathering additional evidence fromℰt\\mathcal\{E\}\_\{t\}as needed while constructing and validating the childptp\_\{t\}before committing it\.

pt∼MetaReflect⁡\(pt′,ct,ℰt\)\.p\_\{t\}\\sim\\mathrm\{MetaReflect\}\(p^\{\\prime\}\_\{t\},c\_\{t\};\\mathcal\{E\}\_\{t\}\)\.
Relation to prior reflective mutation\.This interface recovers prior reflective mutation as the special case in whichct=Ot′c\_\{t\}=O\_\{t\}^\{\\prime\},andℰt=ℰ0=∅\\mathcal\{E\}\_\{t\}=\\mathcal\{E\}\_\{0\}=\\emptyset: each revision conditions only on that trace orside information\([Agrawal et al\., 2026a](https://arxiv.org/html/2610.12794#bib.bib1)\)\. Additionally, these reflective methods implement mutation with a single model call over this preassembled evidence\.

In CoCoEvolve, these interface components take the following concrete forms\.

Ledger to gauge stagnation\.CoCoEvolve’sCurate\\mathrm\{Curate\}operator constructs a compact ledger of recent strategies and their outcomes \(Appendix[C\.1](https://arxiv.org/html/2610.12794#A3.SS1)\)\. The agent uses this ledger to assess stagnation and choose whether to continue the current strategy, adopt a newly proposed one, or revert to undirected mutation\. AdaEvolve makes a related search\-level decision through a fixed trigger, invoking tactic generation when improvement falls below a threshold\([Cemri et al\., 2025](https://arxiv.org/html/2610.12794#bib.bib4)\)\. CoCoEvolve instead lets the same class of model that performs mutation reason over concurrently tracked strategies and make this decision from the search trajectory, without a hand\-tuned stagnation trigger\.

Live environment interaction\.The base environmentℰ0\\mathcal\{E\}\_\{0\}provides domain\-specific tools for execution, testing, and target\-system queries\. For example, in mathematical and systems optimization,ℰ0\\mathcal\{E\}\_\{0\}exposes scratch runners that reveal syntax and execution errors\. In DABStep agent optimization, it provides tools for SQL execution, scratch\-agent deployment, target\-question tests, and deployment checks\. This feedback lets the agent diagnose and repair failures before committing a child to outer\-loop evaluation\. Appendix[C\.2](https://arxiv.org/html/2610.12794#A3.SS2)specifies the interfaces, and Appendix[D\.2](https://arxiv.org/html/2610.12794#A4.SS2)compares static, edit\-only agentic, and self\-validating mutation\.

Tools to query interlinked history\.Beyond these domain capabilities,Augment\\mathrm\{Augment\}formsℰt\\mathcal\{E\}\_\{t\}by adding on\-demand tools that selectively query the interlinked program archive as evidence is needed\. These tools expose per\-task lineage: which program first solved a task, whether a later mutation lost that capability, and which tasks tend to co\-regress\. The agent can therefore retrieve the exact diff that last restored a lost capability or vet a candidate child against known co\-regressions before committing\. Appendix[C\.2](https://arxiv.org/html/2610.12794#A3.SS2)and Table[6](https://arxiv.org/html/2610.12794#A3.T6)specify the complete query interface, while Appendix[D\.3](https://arxiv.org/html/2610.12794#A4.SS3)isolates the contributions of the injected ledger and history\-query tools\.

After validation, we return the generated childptp\_\{t\}to the outer search loop, whereffscores it,ℋt\\mathcal\{H\}\_\{t\}records it, and the search policy governs archive admission and subsequent parent selection\.

### 2\.3Search Policy

By default, CoCoEvolve uses standard MAP\-Elites parent selection\([Sharma, 2025](https://arxiv.org/html/2610.12794#bib.bib26);[Mouret & Clune, 2015](https://arxiv.org/html/2610.12794#bib.bib21)\)\. For objectives scored over a pool of questions or test cases, we introduce a novel budgetedℓp\\ell^\{p\}roster that prioritizes complementary coverage; Appendix[C\.5](https://arxiv.org/html/2610.12794#A3.SS5)defines it, and Appendix[D\.1](https://arxiv.org/html/2610.12794#A4.SS1)compares it with MAP\-Elites, best\-only selection, and pareto\-frontier sampling\.

Beyond parent selection, an optional similarity\-informed task\-selection policy uses proxy scores to reduce evaluation cost and focus mutation on high\-information tasks\.

Similarity\-guided sparse evaluation and observation curation\.Prior methods typically expose the mutator to observations from a random task minibatch\. When the task poolVVis large or individual evaluations are costly, however, exhaustively scoring every candidate is impractical\. On DABStep, for example, each agent query takes roughly 3–10 minutes, so scoring the full 300\-question search pool requires 15–50 agent\-hours before parallelism\. At the same time, tasks need not be independent: a repair that equips a candidate to solve one task may transfer to other tasks requiring the same capability\. This motivates sparse, targeted evaluation: score a bounded subset, infer likely performance on the remaining tasks, and direct the next mutation toward an unresolved target with broad expected transfer\.

CoCoEvolve realizes this idea with a semantic task graph\. Letκ:V×V→\[0,1\]\\kappa:V\\times V\\to\[0,1\]measure task similarity\. Using only evaluated tasks, it forms a Nadaraya–Watson kernel estimate of the candidate’s score onτ\\tau\([Nadaraya, 1964](https://arxiv.org/html/2610.12794#bib.bib22);[Watson, 1964](https://arxiv.org/html/2610.12794#bib.bib33)\):

s^​\(p,τ\)=∑τ′∈tested⁡\(p\)κ⁡\(τ,τ′\)​s​\(p,τ′\)∑τ′∈tested⁡\(p\)κ⁡\(τ,τ′\),\\widehat\{s\}\(p,\\tau\)=\\frac\{\\sum\_\{\\tau^\{\\prime\}\\in\\mathrm\{tested\}\(p\)\}\\kappa\(\\tau,\\tau^\{\\prime\}\)s\(p,\\tau^\{\\prime\}\)\}\{\\sum\_\{\\tau^\{\\prime\}\\in\\mathrm\{tested\}\(p\)\}\\kappa\(\\tau,\\tau^\{\\prime\}\)\},so evaluated neighbors provide a surrogate expected score for tasks not yet tested\. These surrogate scores also determine where mutation effort is likely to be most informative\. Each task not solved by the parent receives an information\-gain score:

gain⁡\(τ\)=\(1−s^​\(p,τ\)\)​δ​\(τ\)​∑τ′∈Vκ⁡\(τ,τ′\)\.\\mathrm\{gain\}\(\\tau\)=\\bigl\(1\-\\widehat\{s\}\(p,\\tau\)\\bigr\)\\delta\(\\tau\)\\\!\\sum\_\{\\tau^\{\\prime\}\\in V\}\\\!\\kappa\(\\tau,\\tau^\{\\prime\}\)\.Here,δ⁡\(τ\)\\delta\(\\tau\)downweights repeatedly unsuccessful targets; sampling bygain\\mathrm\{gain\}then focusesctc\_\{t\}on likely failures with broad semantic reach\. This policy is a task\-level counterpart to AdaEvolve’s island\-level Dynamic Exploration Intensity\([Cemri et al\., 2025](https://arxiv.org/html/2610.12794#bib.bib4)\): it directs scarce evaluation and mutation effort toward failures with the greatest estimated transfer\. Appendix[C\.3](https://arxiv.org/html/2610.12794#A3.SS3)details the policy, Table[7](https://arxiv.org/html/2610.12794#A3.T7)records where it is active, and matched DABStep ablations showcase its value \(Appendix[D\.4](https://arxiv.org/html/2610.12794#A4.SS4)\)\.

## 3Experiments

Where applicable, we follow a standard train/validation/test split\. CoCoEvolve has full access to training examples during optimization\. Validation scores may be used to guide search, but the content of validation instances is never exposed to the optimizer directly\. Final results are reported on the held\-out test split\. We evaluate CoCoEvolve on four task categories: agent optimization, AI Functions, data engineering, and mathematical/system/prompt optimization\. Parent\-selection variants of the archive rule are specified in Appendix[C](https://arxiv.org/html/2610.12794#A3)and compared in Appendix[D\.1](https://arxiv.org/html/2610.12794#A4.SS1); the remaining mechanism\-ablation protocol is Appendix[D\.2](https://arxiv.org/html/2610.12794#A4.SS2)–[D\.4](https://arxiv.org/html/2610.12794#A4.SS4)\. Evaluator details for the math and systems suite appear in Appendix[I](https://arxiv.org/html/2610.12794#A9)\.

### 3\.1Agent Optimization

Major enterprise platforms increasingly deploy fully managed*agentic frameworks*inside the data perimeter\([Snowflake Inc\., 2026a](https://arxiv.org/html/2610.12794#bib.bib28);[Databricks, 2024](https://arxiv.org/html/2610.12794#bib.bib10);[Amazon Web Services, 2023](https://arxiv.org/html/2610.12794#bib.bib3)\)\. These agents iteratively decompose requests and use intermediate results to orchestrate semantic data access, unstructured retrieval, sandboxed code execution, and custom APIs\. Although the platform hides infrastructure and state management, reliability depends on the instructions, tool interfaces, and data models that guide this reasoning, making each a target for optimization\.

Setup\.We formulate agent optimization as searching over*agent specifications*p∈𝒫p\\in\\mathcal\{P\}encompassing system prompts, tool schemas, and planning instructions to maximize end\-to\-end success across diverse user queries on tables and unstructured documents containing domain knowledge\. We evaluate on DABStep, a benchmark of 450 payment\-processing questions requiring multi\-step reasoning over more than nine join dimensions and wildcard joins\. The setting is challenging: the original authors report only 14\.55% accuracy for state\-of\-the\-art agents\. Starting from the same agent specification, we run CoCoEvolve, GEPA, and Meta\-Harness for 100 iterations and compare performance on both the held\-out set and the validated public leaderboard\. Because evaluations are costly, CoCoEvolve uses information\-gain guided observation curation \(Section[2\.3](https://arxiv.org/html/2610.12794#S2.SS3)\) to direct its budget toward likely failures with broad neighborhood reach\. Appendix[F\.1](https://arxiv.org/html/2610.12794#A6.SS1)details this instantiation\.

Results\.Table[1](https://arxiv.org/html/2610.12794#S3.T1)reports the train/test split for CoCoEvolve, Meta\-Harness, GEPA, and the two baselines \(Snowflake Cortex Code Agent, Stock Cortex Agent\)\. On the official 450\-question suite, CoCoEvolve reaches 95\.24% hard accuracy, topping the validated public leaderboard; Table[15](https://arxiv.org/html/2610.12794#A6.T15)in Appendix[F](https://arxiv.org/html/2610.12794#A6)gives the full comparison\. Appendix[F\.1](https://arxiv.org/html/2610.12794#A6.SS1)specifies the scoring protocol\.

Table 1:DABStep accuracy \(%\) on split \(train: 47 easy / 253 hard; test: 25 easy / 125 hard\)\.Analysis\.Under matched splits and budgets, CoCoEvolve outperforms GEPA and Meta\-Harness even when the latter is equipped with self\-verification and self\-testing tools that provide a comparable pre\-commit validation surface \(Appendix[F\.1](https://arxiv.org/html/2610.12794#A6.SS1)\); three mechanisms explain this gap\.

Live environment interaction grounds mutation in execution feedback\.DABStep intentionally tests domain rules whose downstream implications must be inferred or composed across documentation and data, a failure mode emphasized by the benchmark authors\([Egg et al\., 2025](https://arxiv.org/html/2610.12794#bib.bib14)\)\. The live environment turns this ambiguity from guesswork into testable hypotheses\. The Cortex Code mutator repeatedly formed fee interpretations, tested them through live SQL against a known merchant total, and revised its diagnosis\. This live feedback exposed two hidden logic errors and enabled a verified vectorized fee engine, producing the run’s largest single gain, from 28 to 95 correct answers \(Appendix[F\.4](https://arxiv.org/html/2610.12794#A6.SS4)\)\.

Curated environment tools compress otherwise lengthy investigations\.The base environment exposes scripts, managed skills, and developer tools that encode reusable investigation and validation workflows, allowing the mutator to invoke proven procedures instead of reconstructing them across mutation cycles\. During the fee\-engine revision, the mutator first checked its raw SQL, then used an existing testing skill to deploy the draft as a live scratch agent and exercise multiple question families as self\-validation before commit\. Each passed in a single tool call, whereas the prior specification failed after roughly 40 tool calls, allowing the mutator to invest its budget in structural reasoning rather than verification plumbing \(Appendix[C\.2](https://arxiv.org/html/2610.12794#A3.SS2)\)\.

*Structural changes are more robust than prompt changes\.*Reflective prompt evolution is effective\([Agrawal et al\., 2026b](https://arxiv.org/html/2610.12794#bib.bib2)\), but a single model call over preassembled evidence offers no path to explore and validate changes to the underlying tool architecture\. In a matched DABStep run, GEPA revised the instructions but left the default single\-row fee calculator tool unchanged\. Its evolved agent therefore aggregates 26 transaction fees through 26 tool calls and asks the LLM to sum the outputs in context, producing arithmetic drift such as 19\.51 instead of 19\.36\. Such remedies require live environment interaction: inspecting schemas, executing draft SQL, and validating outputs against known totals before commit\. By testing structural edits in situ, CoCoEvolve discovered two independently verified remedies: a vectorized fee\-engine stored procedure and a dynamic table that materializes per\-transaction fees for downstream SQL aggregation\. These solutions shift computation from fallible in\-context arithmetic into deterministic system components, illustrating how coding\-agent mutation can advance where prompt\-only search plateaus \(Appendix[F\.4](https://arxiv.org/html/2610.12794#A6.SS4)\)\.

### 3\.2AI Functions

Modern cloud data platforms expose*AI functions*, declarative SQL operators that embed LLM inference inside query engines for row\-wise tasks such as generation, classification, and extraction\([Liskowski et al\., 2025](https://arxiv.org/html/2610.12794#bib.bib19);[Databricks, 2023](https://arxiv.org/html/2610.12794#bib.bib9);[Microsoft, 2024](https://arxiv.org/html/2610.12794#bib.bib20)\)\. Industry workloads compose these operators with common table expressions \(CTEs\) and joins into multi\-stage pipelines, so reliability depends jointly on prompts, output schemas, and pipeline topology\. For instance, a pipeline may first use a generation function to identify personally identifiable information \(PII\) spans and define a JSON schema, then pass this plan to a downstream function that performs contextual redaction\. We therefore search over executable SQL programsp∈𝒫p\\in\\mathcal\{P\}, allowing CoCoEvolve to restructure both LLM prompts and operator composition for diverse row distributions, missing data, and edge cases\.

Setup\.Each task starts from a minimal SQL programp0p\_\{0\}containing one generative AI\-function call, which the mutator may rewrite into a multi\-stage, multi\-operator pipeline\.*Lockpicking with Interference*\(Appendix[G\.1](https://arxiv.org/html/2610.12794#A7.SS1)\) scores exact match on 50 labeled six\-digit strings governed by a hidden ruleset\. Beyond this synthetic diagnostic, we evaluateclassificationon Reddit comments \(go\_emotions\) and lengthy legal documents \(lex\_glue\)\([Demszky et al\., 2020](https://arxiv.org/html/2610.12794#bib.bib12);[Chalkidis et al\., 2022](https://arxiv.org/html/2610.12794#bib.bib5)\)\. We also studyfinancial numerical reasoning:finqarequires multi\-step arithmetic over tables and prose, whileconvfinqadistributes such reasoning across a dialogue\([Chen et al\., 2021](https://arxiv.org/html/2610.12794#bib.bib6);[Chen et al\., 2022](https://arxiv.org/html/2610.12794#bib.bib7)\)\. Finally,multilexnormrequires normalizing noisy, nonstandard social\-media tokens into canonical forms\([van der Goot et al\., 2021](https://arxiv.org/html/2610.12794#bib.bib31)\)\.

Because evaluation is inexpensive, this setting omits similarity\-informed targeting\. The mutator can instead execute candidate pipelines and repair syntax, schema, or engine\-specific failures before committing, which is especially valuable for brittle SQL programs\.Results\.Table[2](https://arxiv.org/html/2610.12794#S3.T2)shows that CoCoEvolve outperforms the strongest evaluated baseline on all six tasks across classification, numerical reasoning, text normalization, and rule discovery\. The largest gains are a 10 pp improvement that exactly recovers Lockpicking’s hidden rules and a 7\.7 pp gain onmultilexnorm, whose intermingled, context\-dependent normalization patterns echo the latent\-rule challenge isolated by Lockpicking\.

Table 2:AI\-function benchmark performance\.Analysis\.We next examine two mechanisms behind these gains\.

*On\-demand history queries help escape local minima\.*When a highly coupled objective plateaus, reasoning only over the current parent’s failures may not expose a route out\. AdaEvolve’s adaptive scheduler illustrates this limitation on Lockpicking\([Cemri et al\., 2025](https://arxiv.org/html/2610.12794#bib.bib4)\): it first reaches 90% at iteration 34; over the remaining 16 iterations, two stagnation\-triggered paradigm generations propose targeted worked examples and exhaustive condition matrices for the five residual failures, yet its best score never moves\. CoCoEvolve reaches exact recovery at iteration 6, before AdaEvolve’s first paradigm generation at iteration 10, by using on\-demand history tools to query evidence across lineages rather than repeatedly reframing the same residual errors \(Appendix[G\.1](https://arxiv.org/html/2610.12794#A7.SS1)\)\.

*Self\-validation prevents wasted mutation cycles\.*Without a cheap validator, mutation can waste substantial search budget on SQL programs that never execute\. Ongo\_emotions, GEPA repeatedly placedLATERAL FLATTENinside a UDF; because each compiler error collapsed to a zero score without diagnostic feedback, broken variants consumed up to 71% of the budget\. Alternatives such as OpenEvolve mitigate evaluation cost with cascading evaluators that stop after compilation failure, but each rejected candidate still consumes an evolutionary iteration\([Sharma, 2025](https://arxiv.org/html/2610.12794#bib.bib26)\)\. CoCoEvolve instead executes drafts on a cheap sample and repairs syntax failures within the same multi\-turn mutation before commit\. Self\-validation thus converts compiler errors from terminal scores into actionable context, reserving outer\-loop evaluation for executable programs\.

### 3\.3Data Engineering

Data engineering is the process of transforming raw warehouse data into trusted, queryable models\. Agents often fail to construct or repair transformations due to challenges in managing the relevant context, such as data warehouse details, implicit relationships, and domain\-specific business rules\. We optimize over context\-curation pipeline specificationsp∈𝒫p\\in\\mathcal\{P\}that govern the creation of a context\-grounded knowledge graph \(KG\) and a suite of executable adherence tests derived from it, both of which are supplied to a downstream agent prior to executing a transformation\. We evaluate on data\-eng\-bench\([Snowflake AI Research, 2026](https://arxiv.org/html/2610.12794#bib.bib27)\), a data\-engineering benchmark comprising 103 tasks over an enterprise warehouse with 579 tables, 19 business schemas, and more than 8,000 columns\.

Setup\.The seed pipelinep0p\_\{0\}is a single agent with a minimal prompt and the scaffolding for three skills—lean human priors for authoring KGs and the adherence tests\. The mutator may refine the prompt, introduce new skills, contribute to existing ones, or reorganize the pipeline into a multi\-agent protocol\.Φ\\Phiis evaluated through a two\-stage process: the mutator evolves the upstream pipelineppwhile the downstream agent remains fixed; a score is produced only after the agent consumes the generated KG, iterates over the corresponding adherence tests, and implements the transformation that is scored by the benchmark’s verifier\. We defer further details to Appendix[H](https://arxiv.org/html/2610.12794#A8)\.

Results\.As shown in Table[3](https://arxiv.org/html/2610.12794#S3.T3), CoCoEvolve raises the solve rate on the test split from 59\.1% at baseline to 77\.4%, surpassing GEPA at 73\.6% and Meta\-Harness and AdaEvolve at 71\.7%\. On the dev split, CoCoEvolve reaches 68\.0%, tying Meta\-Harness for the highest solve rate and exceeding the baseline, AdaEvolve, and GEPA by 14\.0, 8\.0, and 6\.0 pp, respectively\.

Table 3:Solve rate \(%\) on the dev and test splits ofdata\-eng\-bench\.Analysis\.Meta\-reflection preserves globally relevant skills\.During evolution, an earlier lineage introduced a skill to resolve a recurring dbt configuration error\. A later hillclimb on an unrelated target dropped the skill\. When the failure re\-appeared, CoCoEvolve’s mutator retrieved the archived mutation plan via the ledger that first named the fix and re\-implemented the skill, rather than re\-deriving it from scratch\. We observe that the mutator matches returning failures against past resolutions in the evolution ledger, linking current errors to the earlier mutations that fixed them\. Meta\-reflection thus allows for capabilities the search has already learned to remain visible to the mutator even when not directly present in the parent’s trace\.

### 3\.4Mathematical, Systems, and Prompt Optimization

These experiments ask whether CoCoEvolve can match specialized optimizers on established mathematical and systems benchmarks and whether its benefits persist on HotpotQA, AIME, and IFBench from GEPA’s benchmark suite when live interaction is unavailable\. Within each task, all arms share the task definition, initial artifact, backbone, evaluator, and data split where applicable, enabling matched end\-to\-end comparisons across executable programs and prompt\-only systems\.

Setup\.On math and systems tasks, each evaluator returns one scalar for the complete program\. CoCoEvolve uses a lightweight reflector to detect stagnation and decide whether to continue, request a new direction, or revert to undirected mutation\. When a new direction is needed, a distinct proposer queries techniques from any lineage while preserving the mutator’s turn for implementation\. Appendix[E](https://arxiv.org/html/2610.12794#A5)specifies the complete reflector\-proposer protocol and prompts\. AIME and IFBench use GPT\-5\.6\-luna because Sonnet\-5 saturates AIME at the seed, leaving no optimization signal\. The main table omits standard deviations for readability; bold and underlining mark the best and second\-best results within each backbone, with ties sharing marks\. Appendix[I](https://arxiv.org/html/2610.12794#A9)reports standard deviations, while Appendix[J\.1](https://arxiv.org/html/2610.12794#A10.SS1)details the prompt protocol\.

Results\.Table[4](https://arxiv.org/html/2610.12794#S3.T4)reports the mathematical and ADRS systems benchmarks\. CoCoEvolve is best or tied best in 25 of 44 Avg/Best cells, including 13 of 24 math and 12 of 20 ADRS cells, more than any other method\. On Sonnet\-5, the largest gains occur on plateau\-prone Signal Processing, TXN, and Heilbronn Convex; on Signal Processing, CoCoEvolve reaches0\.7660\.766versus AdaEvolve’s0\.7470\.747\. Gaps largely vanish on saturated objectives, while mixed GPT\-5\.6\-terra results show that the advantage depends on the backbone\. Appendix[I\.3](https://arxiv.org/html/2610.12794#A9.SS3)analyzes plateau escape, structural family switching, and saturation controls task by task\. Table[5](https://arxiv.org/html/2610.12794#S3.T5)reports prompt optimization, where CoCoEvolve leads GEPA and the seed on every HotpotQA and IFBench split, while AIME is inconsistent under repeated evaluation\.

Table 4:Mathematical and ADRS systems benchmarks\. Higher is better except for Cloudcast\. Human/SOTA is the maximum published human or automated result reported by AlphaEvolve and AdaEvolve\. Full\-precision means, standard deviations, and best values appear in Tables[18](https://arxiv.org/html/2610.12794#A9.T18)and[19](https://arxiv.org/html/2610.12794#A9.T19)\.Table 5:Prompt optimization on HotpotQA, AIME, and IFBench\. HotpotQA reports a single Sonnet\-5 run; AIME and IFBench report mean±\\pmstandard deviation over three evaluations of each final candidate usingopenai\-gpt\-5\.6\-luna; these repeats measure evaluation noise, not search variance\. Bold marks column\-best results that are robust to measurement noise;†the observed zero deviation is coincidental\.Analysis\.Across the mathematical and systems benchmarks, CoCoEvolve recovers the gains of AdaEvolve’s specialized three\-level adaptive controller through meta\-reflection over the search history, offering a more flexible route to stagnation\-aware search\.

The strategy ledger turns plateaus into new search directions\.On Signal Processing, the global best remained at 0\.7534 for 26 iterations even though the current parent and champion represented distinct Savitzky\-Golay and Butterworth filter families\. Reasoning over the strategy ledger, the lightweight reflector recognized a population\-wide plateau and requested a new proposal rather than another local mutation or crossover\. Using on\-demand history queries to identify what the search had not tried, the separate proposer found an orthogonal family: an L1 trend filter formulated as a linear program\. The mutator implemented this paradigm and raised the score to 0\.7745 in one iteration\. This trace illustrates how the strategy ledger and on\-demand history tools separate stagnation detection, paradigm selection, and implementation: the search history reveals not only which ideas succeeded, but which structural alternatives remain unexplored\.

On\-demand history queries improve prompt optimization even without live environment interaction\.On HotpotQA, CoCoEvolve reaches0\.7930\.793held\-out accuracy versus GEPA’s0\.6830\.683and the seed’s0\.3630\.363, an 11 pp margin in GEPA’s own setting\. IFBench preserves the ordering CoCoEvolve\>\>GEPA\>\>seed on every repeat and split, although the held\-out margin is narrower\. AIME provides no reliable separation: all arms lie in the same0\.830\.83–0\.880\.88test band, while1717–27%27\\%of individual answers change correctness across repeated evaluations of an unchanged prompt\. Together, these results isolate the value of on\-demand history access for text\-only artifacts, while AIME illustrates how stochastic evaluation can erase apparent search gains\.

## 4Related Work

Evolutionary Harnesses\.LLM\-guided evolutionary optimization shifts improvement from training\-time parameter updates toward inference\-time search, using LLMs as semantic mutation operators over prompts, algorithms, and programs\. Building on quality\-diversity search\([Mouret & Clune, 2015](https://arxiv.org/html/2610.12794#bib.bib21)\), systems such as FunSearch, AlphaEvolve, and OpenEvolve apply this paradigm to scientific and algorithmic discovery\([Romera\-Paredes et al\., 2024](https://arxiv.org/html/2610.12794#bib.bib25);[Novikov et al\., 2025](https://arxiv.org/html/2610.12794#bib.bib23);[Sharma, 2025](https://arxiv.org/html/2610.12794#bib.bib26)\)\. GEPA enriches mutation with language\-valued execution traces that expose reasoning, tool calls, outputs, and errors beyond scalar rewards\([Agrawal et al\., 2026b](https://arxiv.org/html/2610.12794#bib.bib2)\)\. Subsequent work broadens the optimized artifact space through unified declarative APIs\([Agrawal et al\., 2026a](https://arxiv.org/html/2610.12794#bib.bib1)\), allocates search adaptively\([Cemri et al\., 2025](https://arxiv.org/html/2610.12794#bib.bib4);[Lange et al\., 2025](https://arxiv.org/html/2610.12794#bib.bib17)\), and standardizes outer harnesses\([Ursekar et al\., 2026](https://arxiv.org/html/2610.12794#bib.bib30)\)\. For prompts and LM pipelines, EvoPrompt and Promptbreeder evolve prompts with LLM\-driven operators\([Guo et al\., 2025](https://arxiv.org/html/2610.12794#bib.bib16);[Fernando et al\., 2024](https://arxiv.org/html/2610.12794#bib.bib15)\), MIPROv2 jointly searches instructions and demonstrations for multi\-stage programs\([Opsahl\-Ong et al\., 2024](https://arxiv.org/html/2610.12794#bib.bib24)\), and TextGrad backpropagates natural\-language feedback through compound systems\([Yuksekgonul et al\., 2025](https://arxiv.org/html/2610.12794#bib.bib36)\)\. These methods generally retain a parent\-local, one\-shot mutation interface in which the harness preselects evidence and evaluates the proposed edit only after generation\. Promptbreeder also evolves its mutation prompts, whereas CoCoEvolve changes the mutation interface itself through meta\-reflection over the search history and target environment\. Meta\-Harness\([Lee et al\., 2026](https://arxiv.org/html/2610.12794#bib.bib18)\)similarly uses a coding\-agent proposer with filesystem access to prior traces; CoCoEvolve combines compact curated context with structured, on\-demand queries over evolutionary history and a live environment\.

Context Curation Approaches\.The efficacy of LLM agents relies heavily on rich, domain\-specific context, such as organizational schemas\([Döhmen et al\., 2026](https://arxiv.org/html/2610.12794#bib.bib13)\)or dynamically constructed failure taxonomies\([Cojocaru et al\., 2026](https://arxiv.org/html/2610.12794#bib.bib8)\)\. Recent frameworks automate context engineering by treating contexts as evolving playbooks\([Zhang et al\., 2026](https://arxiv.org/html/2610.12794#bib.bib37)\)or co\-evolving curation skills alongside artifacts\([Ye et al\., 2026](https://arxiv.org/html/2610.12794#bib.bib35)\)\. CoCoEvolve applies this principle to the optimization trajectory itself, distilling accumulated search history into a compact ledger\.

Limitations of Evolutionary Approaches\.AI\-driven evolutionary frameworks exhibit systematic vulnerabilities, including hidden regressions\([Zhou et al\., 2026](https://arxiv.org/html/2610.12794#bib.bib38)\), overfitting to the initial task set under continual learning\([Wang et al\., 2026](https://arxiv.org/html/2610.12794#bib.bib32)\), and context costs that grow with the accumulated archive\([Lee et al\., 2026](https://arxiv.org/html/2610.12794#bib.bib18)\)\. CoCoEvolve addresses these failure modes with bounded, queryable access to historical co\-regressions, helping the optimizer avoid cycling through brittle solutions that break previously solved cases\.

## 5Conclusion

We present CoCoEvolve, an evolutionary optimizer for production AI systems\. By both probing the environment and querying an evolutionary ledger and prior lineages, CoCoEvolve’s mutator extends parent\-local reflection into*meta\-reflection*over the search process, enabling it to diagnose failures, reconstruct hidden mechanisms, preserve complementary discoveries, and verify structural revisions before committing\. Across domains, our results suggest that the combination of curating selected historical evidence, providing on\-demand tools for probing the live environment and evolutionary history, and using a coding\-agent mutator to reason over both offers an effective path to improving the prompts, code, tools, data models, and pipelines of real\-world AI systems\.

#### Acknowledgments

We thank Danmei Xu, Garett Tok, Zihan Li, Varich Boonsanong, Kyle Schmaus, and Boxin Jiang for insightful discussions, feedback and suggesting relevant datasets\.

## References

- Agrawal et al\. \(2026a\)Lakshya A\. Agrawal, Donghyun Lee, Shangyin Tan, Wenjie Ma, Karim Elmaaroufi, Rohit Sandadi, Sanjit A\. Seshia, Koushik Sen, Dan Klein, Ion Stoica, Joseph E\. Gonzalez, Omar Khattab, Alexandros G\. Dimakis, and Matei Zaharia\.optimize\_anything: A universal API for optimizing any text parameter\.In*ACM Conference on AI and Agentic Systems \(CAIS ’26\)*, 2026a\.URL[https://arxiv\.org/abs/2605\.19633](https://arxiv.org/abs/2605.19633)\.
- Agrawal et al\. \(2026b\)Lakshya A\. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl\-Ong, Arnav Singhvi, Herumb Shandilya, Michael J\. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alex Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab\.GEPA: Reflective prompt evolution can outperform reinforcement learning\.In*The Fourteenth International Conference on Learning Representations*, 2026b\.URL[https://openreview\.net/forum?id=RQm2KQTM5r](https://openreview.net/forum?id=RQm2KQTM5r)\.Oral\.
- Amazon Web Services \(2023\)Amazon Web Services\.Agents for Amazon Bedrock\.[https://aws\.amazon\.com/bedrock/agents/](https://aws.amazon.com/bedrock/agents/), 2023\.Accessed: 2026\-08\-09\.
- Cemri et al\. \(2025\)Mert Cemri, Shubham Agrawal, Akshat Gupta, Shu Liu, Audrey Cheng, Qiuyang Mang, Ashwin Naren, Lutfi Eren Erdogan, Koushik Sen, Matei Zaharia, Alex Dimakis, and Ion Stoica\.AdaEvolve: Adaptive LLM driven zeroth\-order optimization\.*arXiv preprint arXiv:2602\.20133*, 2025\.URL[https://arxiv\.org/abs/2602\.20133](https://arxiv.org/abs/2602.20133)\.
- Chalkidis et al\. \(2022\)Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras\.LexGLUE: A benchmark dataset for legal language understanding in English\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(ACL\)*, pp\. 4310–4330, 2022\.URL[https://aclanthology\.org/2022\.acl\-long\.297/](https://aclanthology.org/2022.acl-long.297/)\.
- Chen et al\. \(2021\)Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting\-Hao Huang, Bryan Routledge, and William Yang Wang\.FinQA: A dataset of numerical reasoning over financial data\.In*Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pp\. 3697–3711, 2021\.URL[https://aclanthology\.org/2021\.emnlp\-main\.300/](https://aclanthology.org/2021.emnlp-main.300/)\.
- Chen et al\. \(2022\)Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang\.ConvFinQA: Exploring the chain of numerical reasoning in conversational finance question answering\.In*Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pp\. 6279–6292, 2022\.URL[https://aclanthology\.org/2022\.emnlp\-main\.421/](https://aclanthology.org/2022.emnlp-main.421/)\.
- Cojocaru et al\. \(2026\)Andrei Cojocaru, Mert Cemri, Melissa Pan, Shu Liu, Shubham Agarwal, Alex Krentsel, Jay Tang, Kannan Ramchandran, Joseph Gonzalez, Matei Zaharia, Alex Dimakis, and Ion Stoica\.Fantastic adaptive taxonomies and how to use them\.In*FAGEN Workshop at ICML 2026*, 2026\.URL[https://openreview\.net/pdf?id=krDmXYWzmJ](https://openreview.net/pdf?id=krDmXYWzmJ)\.Best Paper Award\.
- Databricks \(2023\)Databricks\.Introducing AI functions: Integrating large language models with Databricks SQL\.[https://www\.databricks\.com/blog/2023/04/18/introducing\-ai\-functions\-integrating\-large\-language\-models\-databricks\-sql\.html](https://www.databricks.com/blog/2023/04/18/introducing-ai-functions-integrating-large-language-models-databricks-sql.html), 2023\.Accessed: 2026\-08\-09\.
- Databricks \(2024\)Databricks\.Mosaic AI agent framework\.[https://www\.databricks\.com/product/machine\-learning/mosaic\-ai\-agent\-framework](https://www.databricks.com/product/machine-learning/mosaic-ai-agent-framework), 2024\.Accessed: 2026\-08\-09\.
- dbt Labs \(2026\)dbt Labs\.What is dbt?dbt Developer Hub, 2026\.URL[https://docs\.getdbt\.com/docs/introduction](https://docs.getdbt.com/docs/introduction)\.
- Demszky et al\. \(2020\)Dorottya Demszky, Dana Movshovitz\-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi\.GoEmotions: A dataset of fine\-grained emotions\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics \(ACL\)*, pp\. 4040–4054, 2020\.URL[https://aclanthology\.org/2020\.acl\-main\.372/](https://aclanthology.org/2020.acl-main.372/)\.
- Döhmen et al\. \(2026\)Till Döhmen, Jacob Matson, and Jordan Tigani\.Towards a context layer for self\-improving data agents\.In*SAO Workshop at ACM Conference on AI and Agentic Systems \(CAIS ’26\)*, 2026\.URL[https://bauplanlabs\.github\.io/SAO\-workshop/papers/73\.pdf](https://bauplanlabs.github.io/SAO-workshop/papers/73.pdf)\.
- Egg et al\. \(2025\)Alex Egg, Martin Iglesias Goyanes, Friso Kingma, Andreu Mora, Leandro von Werra, and Thomas Wolf\.DABstep: Data agent benchmark for multi\-step reasoning\.*arXiv preprint arXiv:2506\.23719*, 2025\.URL[https://arxiv\.org/abs/2506\.23719](https://arxiv.org/abs/2506.23719)\.
- Fernando et al\. \(2024\)Chrisantha Fernando, Dylan Sunil Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel\.Promptbreeder: Self\-referential self\-improvement via prompt evolution\.In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp \(eds\.\),*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pp\. 13481–13544, 21–27 Jul 2024\.URL[https://proceedings\.mlr\.press/v235/fernando24a\.html](https://proceedings.mlr.press/v235/fernando24a.html)\.
- Guo et al\. \(2025\)Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang\.Evoprompt: Connecting llms with evolutionary algorithms yields powerful prompt optimizers\.In*Proceedings of the 31st International Conference on Computational Linguistics*, 2025\.URL[https://aclanthology\.org/2025\.coling\-main\.345/](https://aclanthology.org/2025.coling-main.345/)\.
- Lange et al\. \(2025\)Robert Tjarko Lange, Yuki Imajuku, and Edoardo Cetin\.ShinkaEvolve: Towards open\-ended and sample\-efficient program evolution\.*arXiv preprint arXiv:2509\.19349*, 2025\.URL[https://arxiv\.org/abs/2509\.19349](https://arxiv.org/abs/2509.19349)\.
- Lee et al\. \(2026\)Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn\.Meta\-Harness: End\-to\-end optimization of model harnesses\.In*Conference on Language Modeling \(COLM\)*, 2026\.URL[https://arxiv\.org/abs/2603\.28052](https://arxiv.org/abs/2603.28052)\.
- Liskowski et al\. \(2025\)Paweł Liskowski, Benjamin Han, Paritosh Aggarwal, Bowei Chen, Boxin Jiang, Nitish Jindal, Zihan Li, Aaron Lin, Kyle Schmaus, Jay Tayade, Weicheng Zhao, Anupam Datta, Nathan Wiegand, and Dimitris Tsirogiannis\.Cortex AISQL: A production SQL engine for unstructured data, 2025\.URL[https://arxiv\.org/abs/2511\.07663](https://arxiv.org/abs/2511.07663)\.
- Microsoft \(2024\)Microsoft\.Invoke AI capabilities using Azure SQL database and Azure OpenAI\.[https://learn\.microsoft\.com/en\-us/azure/azure\-sql/database/invoke\-ai\-with\-azure\-openai](https://learn.microsoft.com/en-us/azure/azure-sql/database/invoke-ai-with-azure-openai), 2024\.Accessed: 2026\-08\-09\.
- Mouret & Clune \(2015\)Jean\-Baptiste Mouret and Jeff Clune\.Illuminating search spaces by mapping elites\.*arXiv preprint arXiv:1504\.04909*, 2015\.URL[https://arxiv\.org/abs/1504\.04909](https://arxiv.org/abs/1504.04909)\.
- Nadaraya \(1964\)E\. A\. Nadaraya\.On estimating regression\.*Theory of Probability and Its Applications*, 9\(1\):141–142, 1964\.doi:10\.1137/1109020\.URL[https://doi\.org/10\.1137/1109020](https://doi.org/10.1137/1109020)\.
- Novikov et al\. \(2025\)Alexander Novikov, Ngân Vu, Marvin Eisenberger, Emilien Dupont, Po\-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J\. R\. Ruiz, Abbas Mehrabian, M\. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog\.AlphaEvolve: A coding agent for scientific and algorithmic discovery\.White paper, Google DeepMind, 2025\.URL[https://storage\.googleapis\.com/deepmind\-media/DeepMind\.com/Blog/alphaevolve\-a\-gemini\-powered\-coding\-agent\-for\-designing\-advanced\-algorithms/AlphaEvolve\.pdf](https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/AlphaEvolve.pdf)\.
- Opsahl\-Ong et al\. \(2024\)Krista Opsahl\-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab\.Optimizing instructions and demonstrations for multi\-stage language model programs\.In Yaser Al\-Onaizan, Mohit Bansal, and Yun\-Nung Chen \(eds\.\),*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp\. 9340–9366, Miami, Florida, USA, November 2024\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.emnlp\-main\.525\.URL[https://aclanthology\.org/2024\.emnlp\-main\.525/](https://aclanthology.org/2024.emnlp-main.525/)\.
- Romera\-Paredes et al\. \(2024\)Bernardino Romera\-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M\. Pawan Kumar, Emilien Dupont, Francisco J\. R\. Ruiz, Jordan S\. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi\.Mathematical discoveries from program search with large language models\.*Nature*, 625:468–475, 2024\.doi:10\.1038/s41586\-023\-06924\-6\.
- Sharma \(2025\)Asankhaya Sharma\.OpenEvolve: Open\-source implementation of AlphaEvolve\.GitHub, 2025\.URL[https://github\.com/codelion/openevolve](https://github.com/codelion/openevolve)\.
- Snowflake AI Research \(2026\)Snowflake AI Research\.Data\-eng\-bench: A data engineering benchmark for AI agents\.Snowflake Engineering Blog, 2026\.URL[https://www\.snowflake\.com/en/blog/engineering/data\-eng\-bench\-data\-engineering\-agent\-benchmark/](https://www.snowflake.com/en/blog/engineering/data-eng-bench-data-engineering-agent-benchmark/)\.
- Snowflake Inc\. \(2026a\)Snowflake Inc\.Cortex agents\.Snowflake Documentation, 2026a\.URL[https://docs\.snowflake\.com/en/user\-guide/snowflake\-cortex/cortex\-agents](https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-agents)\.
- Snowflake Inc\. \(2026b\)Snowflake Inc\.Snowflake CoCo: Snowflake\-native AI coding agent for data\.Snowflake Product Page, 2026b\.URL[https://www\.snowflake\.com/en/product/snowflake\-coco/](https://www.snowflake.com/en/product/snowflake-coco/)\.
- Ursekar et al\. \(2026\)Varun Ursekar, Apaar Shanker, Veronica Chatrath, Yuan Xue, and Samuel Marc Denton\.VeRO: A harness for agents to optimize agents\.In*ICML 2026*, 2026\.URL[https://arxiv\.org/abs/2602\.22480](https://arxiv.org/abs/2602.22480)\.
- van der Goot et al\. \(2021\)Rob van der Goot, Alan Ramponi, Arkaitz Zubiaga, Barbara Plank, Benjamin Muller, Iñaki San Vicente Roncal, Nikola Ljubešić, Özlem Çetinoğlu, Rahmad Mahendra, Talha Çolakoğlu, Timothy Baldwin, Tommaso Caselli, and Wladimir Sidorenko\.MultiLexNorm: A shared task on multilingual lexical normalization\.In*Proceedings of the Seventh Workshop on Noisy User\-generated Text \(W\-NUT\)*, 2021\.URL[https://aclanthology\.org/2021\.wnut\-1\.55/](https://aclanthology.org/2021.wnut-1.55/)\.
- Wang et al\. \(2026\)Wenxiao Wang, Priyatham Kattakinda, and Soheil Feizi\.Do agent optimizers compound? a continual\-learning evaluation on Terminal\-Bench 2\.0\.*arXiv preprint arXiv:2607\.14004*, 2026\.URL[https://arxiv\.org/abs/2607\.14004](https://arxiv.org/abs/2607.14004)\.
- Watson \(1964\)Geoffrey S\. Watson\.Smooth regression analysis\.*Sankhyā: The Indian Journal of Statistics, Series A*, 26\(4\):359–372, 1964\.URL[https://www\.jstor\.org/stable/25049340](https://www.jstor.org/stable/25049340)\.
- Yang et al\. \(2018\)Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W\. Cohen, Ruslan Salakhutdinov, and Christopher D\. Manning\.HotpotQA: A dataset for diverse, explainable multi\-hop question answering\.In*Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pp\. 2369–2380, 2018\.URL[https://aclanthology\.org/D18\-1259/](https://aclanthology.org/D18-1259/)\.
- Ye et al\. \(2026\)Haoran Ye, Xuning He, Vincent Arak, Haonan Dong, and Guojie Song\.Meta context engineering via agentic skill evolution\.*arXiv preprint arXiv:2601\.21557*, 2026\.URL[https://arxiv\.org/abs/2601\.21557](https://arxiv.org/abs/2601.21557)\.
- Yuksekgonul et al\. \(2025\)Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou\.Optimizing generative AI by backpropagating language model feedback\.*Nature*, 639:609–616, 2025\.doi:10\.1038/s41586\-025\-08661\-4\.
- Zhang et al\. \(2026\)Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Zou, and Kunle Olukotun\.ACE: Agentic context engineering: Evolving contexts for self\-improving language models\.*arXiv preprint arXiv:2510\.04618*, 2026\.URL[https://arxiv\.org/abs/2510\.04618](https://arxiv.org/abs/2510.04618)\.
- Zhou et al\. \(2026\)Yajie Zhou, Ao Li, Ashwin Silla, Zaoxing Liu, and Vyas Sekar\.AIChilles: Automatically uncovering hidden weaknesses in AI\-evolved systems\.*arXiv preprint arXiv:2606\.15834*, 2026\.URL[https://arxiv\.org/abs/2606\.15834](https://arxiv.org/abs/2606.15834)\.

## Appendix ATable of Contents

This appendix is organized as follows\. Appendix[C](https://arxiv.org/html/2610.12794#A3)specifies the evolutionary loop \(Appendix[C\.1](https://arxiv.org/html/2610.12794#A3.SS1)\), the history\-query and workspace tools \(Appendix[C\.2](https://arxiv.org/html/2610.12794#A3.SS2)\), the similarity\-graph InfoGain branch used by agent evolution \(Appendix[C\.3](https://arxiv.org/html/2610.12794#A3.SS3)\), the experimental hyperparameters \(Appendix[C\.4](https://arxiv.org/html/2610.12794#A3.SS4)\), and the budgetedℓp\\ell^\{p\}roster \(Appendix[C\.5](https://arxiv.org/html/2610.12794#A3.SS5)\)\. Appendix[D](https://arxiv.org/html/2610.12794#A4)reports parent selection, mutation and self\-validation, history access, and similarity\-manifold targeting ablations\. The history\-access study includes Signal Processing and DABStep\. Each ablation subsection reports its trial count and score provenance\. Appendix[E](https://arxiv.org/html/2610.12794#A5)records the mutator, reflector, and proposer prompts\. Appendix[F](https://arxiv.org/html/2610.12794#A6)gives the DABStep protocol, analyzes Meta\-Harness’s post\-improvement churn, and presents an evolved agent instruction\. Appendix[G](https://arxiv.org/html/2610.12794#A7)presents the Lockpicking synthetic diagnostic \(Appendix[G\.1](https://arxiv.org/html/2610.12794#A7.SS1)\) and describes the AI\-function experiments and their evolved artifacts\. Appendix[H](https://arxiv.org/html/2610.12794#A8)describes the data\-engineering pipeline, its evaluation, and an evolved artifact\. Appendix[I](https://arxiv.org/html/2610.12794#A9)records math and ADRS evaluator settings and the detailed result tables \(Appendix[I\.2](https://arxiv.org/html/2610.12794#A9.SS2)\)\. Appendix[I\.3](https://arxiv.org/html/2610.12794#A9.SS3)gives the per\-task analysis: which methods lead each cell, then Heilbronn Convex, Circle Packing, Cloudcast, and TXN traces \(Figures[4](https://arxiv.org/html/2610.12794#A9.F4)–[7](https://arxiv.org/html/2610.12794#A9.F7)\)\. Appendix[J](https://arxiv.org/html/2610.12794#A10)gives the methodology for all three prompt\-optimization tasks \(Appendix[J\.1](https://arxiv.org/html/2610.12794#A10.SS1)\) and a sim\-graph\-targeted, cascaded\-evaluation efficiency variant of HotpotQA \(Appendix[J\.2](https://arxiv.org/html/2610.12794#A10.SS2)\)\. Appendix[K](https://arxiv.org/html/2610.12794#A11), placed last, includes the complete base and final programs for Heilbronn Convex, Cloudcast, and TXN\.

## Appendix BUse of Large Language Models \(LLMs\) and other AI Tools

The authors used large language models \(LLMs\) to assist in literature discovery and polishing the manuscript’s prose\. The authors reviewed and verified all suggested references, claims, and revisions, and assume full responsibility for the final content\. Coding assistants, including Snowflake Cortex Code, Anthropic Claude, and Cursor, helped implement the authors’ ideas\. The methodology, scientific contributions, and core results are the authors’ original work\.

## Appendix CAlgorithm and Method

This appendix instantiates the operators of Section[2](https://arxiv.org/html/2610.12794#S2)\. Section[C\.1](https://arxiv.org/html/2610.12794#A3.SS1)gives the evolutionary loop; Section[C\.2](https://arxiv.org/html/2610.12794#A3.SS2)lists the history tools added throughAugment\\mathrm\{Augment\}and the domain tools supplied byℰ0\\mathcal\{E\}\_\{0\}; Section[C\.3](https://arxiv.org/html/2610.12794#A3.SS3)records the kernel, decay, and softmax used for curated observations; Section[C\.4](https://arxiv.org/html/2610.12794#A3.SS4)lists the hyperparameters used in the experiments of Section[3](https://arxiv.org/html/2610.12794#S3); and Section[C\.5](https://arxiv.org/html/2610.12794#A3.SS5)specifies the budgetedℓp\\ell^\{p\}roster used as a population objective and as a parent\-selection policy\.

### C\.1Evolutionary loop

Algorithm 1CoCoEvolve evolutionary loop\.1:seed

p0p\_\{0\}, budget

TT, evaluator

ff, parent policy

π\\pi, base environment

ℰ0⊆ℰ\\mathcal\{E\}\_\{0\}\\subseteq\\mathcal\{E\}
2:

𝒜0←\{p0\}\\mathcal\{A\}\_\{0\}\\leftarrow\\\{p\_\{0\}\\\},

ℋ0←∅\\mathcal\{H\}\_\{0\}\\leftarrow\\emptyset
3:for

t=1,…,Tt=1,\\ldots,Tdo

4:

pt′∼π⁡\(𝒜t−1\)p^\{\\prime\}\_\{t\}\\sim\\pi\(\\mathcal\{A\}\_\{t\-1\}\)⊳\\trianglerightParent selection

5:

ct←Curate⁡\(ℋt−1\)c\_\{t\}\\leftarrow\\mathrm\{Curate\}\(\\mathcal\{H\}\_\{t\-1\}\)⊳\\trianglerightContext curation

6:

ℰt←Augment⁡\(ℰ0,pt′,ℋt−1\)\\mathcal\{E\}\_\{t\}\\leftarrow\\mathrm\{Augment\}\(\\mathcal\{E\}\_\{0\};p^\{\\prime\}\_\{t\},\\mathcal\{H\}\_\{t\-1\}\)⊳\\trianglerightEnvironment augmentation

7:

pt∼MetaReflect⁡\(pt′,ct,ℰt\)p\_\{t\}\\sim\\mathrm\{MetaReflect\}\(p^\{\\prime\}\_\{t\},c\_\{t\};\\mathcal\{E\}\_\{t\}\)⊳\\trianglerightAgentic investigation

8:select

𝒯t⊆𝒯\\mathcal\{T\}\_\{t\}\\subseteq\\mathcal\{T\}; evaluate

ptp\_\{t\}with

ffto obtain

StS\_\{t\}and

OtO\_\{t\}
9:

ℋt←ℋt−1∪\{\(pt,𝒯t,St,Ot\)\}\\mathcal\{H\}\_\{t\}\\leftarrow\\mathcal\{H\}\_\{t\-1\}\\cup\\\{\(p\_\{t\},\\mathcal\{T\}\_\{t\},S\_\{t\},O\_\{t\}\)\\\}
10:update

𝒜t−1\\mathcal\{A\}\_\{t\-1\}with

ptp\_\{t\}to form

𝒜t\\mathcal\{A\}\_\{t\}
11:endfor

12:return

arg⁡maxp∈𝒜T​Φ​\(p\)\\arg\\max\_\{p\\in\\mathcal\{A\}\_\{T\}\}\\Phi\(p\)

The loop selects parentpt′p^\{\\prime\}\_\{t\}from𝒜t−1\\mathcal\{A\}\_\{t\-1\}usingπ\\pi; the parent policy uses either MAP\-Elites or theℓp\\ell^\{p\}roster of Section[C\.5](https://arxiv.org/html/2610.12794#A3.SS5)\. Context curation preparesctc\_\{t\}, while environment augmentation adds on\-demand history\-query tools toℰ0\\mathcal\{E\}\_\{0\}to formℰt\\mathcal\{E\}\_\{t\}\. Depending on the task,Curate\\mathrm\{Curate\}reflects on the strategy ledger and selects a target instance using the similarity graph of Section[C\.3](https://arxiv.org/html/2610.12794#A3.SS3)\. Because operational requirements vary by domain, these preparation mechanisms are layered according to the search objective\. We detail the exact workflow applied for each experiment in its task subsection\.

In general, the injected bundlectc\_\{t\}contains the approach ledger \(recent strategies and their score deltas\), the active paradigm text if one is in force, and island elites\. When an instance pool is defined,ctc\_\{t\}also includes the curated target observation and any donor program that has already solved it\. The augmented environmentℰt\\mathcal\{E\}\_\{t\}combines the domain tools inℰ0\\mathcal\{E\}\_\{0\}with the opt\-in history tools of Section[C\.2](https://arxiv.org/html/2610.12794#A3.SS2)\. The Cortex Code coding\-agent mutator then performsMetaReflect\\mathrm\{MetaReflect\}by investigating, implementing, and validating the child, after which admission is typically a MAP\-Elites update on a complexity–diversity grid\.

### C\.2On\-demand history and environment tools

Table[6](https://arxiv.org/html/2610.12794#A3.T6)lists the on\-demand tools provisioned inℰt\\mathcal\{E\}\_\{t\}\. The mutator can call these tools from its task workspace while editing and validating a candidate\. The ledger is injected every iteration; tool results enter the context only when requested, so the full archive need not appear in the prompt\. For archive access, the mutator callspython3 evo\.pyto query prior programs, lineages, scores, and traces\. For local execution, the constrainedevo\_sweep\.pyrunner executes the candidate without granting unrestricted command execution\. The math, ADRS, Signal Processing, and DABStep experiments all expose lineage and trajectory queries\. Per\-task queries are enabled for DABStep and Signal Processing, whose evaluators expose instance\-level or per\-signal outcomes, and disabled for the remaining math and ADRS tasks, which return only a scalar score\. DABStep additionally overlays warehouse tools that issue SQL, deploy a scratch agent, and pre\-screen a draft against a target question before committing the child\.

ChannelToolReturnsLineageget\_programcode body of a programget\_parent/get\_lineageparent id; children with score deltasget\_best\_programstop\-NNprograms by primary scoreget\_diverse\_programsbest program per occupied MAP\-Elites cellget\_architecture\_historyper\-cell landing historyTrajectoryget\_recent\_scoresrecent primary\-score traceget\_thinking\_tracetool/thinking events of iterationIIPer\-taskget\_question\_solverstop programs on one instance, with parentget\_question\_memorychild<<parent regressions on one instanceget\_suggested\_questionsinstances ranked worst\-first for a programget\_hard\_questionsinstances by population\-best score, ascendingWarehouse \(DABStep\)queryread\-only SQL against live tablesexecute\_sqlallowed write SQL against the warehousetest\_agent/test\_draftscore a scratch agent on questionQQcheck\_deploycompile\-check oneCREATEDDLanalyst\_querytext\-to\-SQL against the semantic view

Table 6:On\-demand tools available inℰt\\mathcal\{E\}\_\{t\}\. Lineage, trajectory, and per\-task tools are added throughAugment\\mathrm\{Augment\}to query the evolutionary archive; the DABStep base environmentℰ0\\mathcal\{E\}\_\{0\}additionally provides live\-warehouse validation tools\.
### C\.3Curated observations: kernel, decay, and softmax

Section[2\.3](https://arxiv.org/html/2610.12794#S2.SS3)writes a generic similarity kernelκ\\kappa\. This section applies to the DABStep agent\-evolution setting, which evaluates a program on many instances and maintains a similarity graph over that pool\. Math, ADRS, and Signal Processing have no such pool and skip this channel, even though Signal Processing still exposes per\-signal query tools \(Table[7](https://arxiv.org/html/2610.12794#A3.T7)\)\. We precompute an embedding\-similarity matrixκ\\kappaonVV, with values in\[0,1\]\[0,1\]\. DABStep specializes the estimator in Section[2\.3](https://arxiv.org/html/2610.12794#S2.SS3)by replacingκ⁡\(τ,τ′\)\\kappa\(\\tau,\\tau^\{\\prime\}\)withκ​\(τ,τ′\)4\\kappa\(\\tau,\\tau^\{\\prime\}\)^\{4\}in both the numerator and denominator, giving more weight to nearby tested instances\. After each mutation, a local success or regression update refines the estimate\. Estimates for untested coordinates are capped at0\.70\.7, and coordinates with observed failures are floored at0\.10\.1\.

The next targetτ∗\\tau^\{\\ast\}is drawn by softmax overgain\\mathrm\{gain\}with temperature0\.10\.1\. Candidates exclude instances the parent has already solved \(score≥0\.9\\geq 0\.9\)\. The decay in Section[2\.3](https://arxiv.org/html/2610.12794#S2.SS3)is instantiated as

δ⁡\(τ\)=\{1if any program in𝒜thas solvedτ,\(1/2\)c⁡\(τ\)otherwise,\\delta\(\\tau\)=\\begin\{cases\}1&\\text\{if any program in $\\mathcal\{A\}\_\{t\}$ has solved $\\tau$\},\\\\ \(1/2\)^\{c\(\\tau\)\}&\\text\{otherwise\},\\end\{cases\}wherec⁡\(τ\)c\(\\tau\)is the number of timesτ\\tauhas been selected as the mutation target without success\. A solve anywhere in the archive therefore resets the penalty, so a newly acquired donor can be reused; repeated failures on an unsolved instance geometrically suppress it\.

### C\.4Hyperparameters

Table[7](https://arxiv.org/html/2610.12794#A3.T7)records the settings used in Section[3](https://arxiv.org/html/2610.12794#S3)\. The default math, ADRS, and DABStep runs sample parents from MAP\-Elites islands using exploration, elite selection, and fitness\-weighted draws\. Math and ADRS use exploration weight0\.20\.2and elite\-selection weight0\.650\.65\. The AI\-function objective and the parent\-selection ablation instead use theℓp\\ell^\{p\}roster of Section[C\.5](https://arxiv.org/html/2610.12794#A3.SS5)\. InfoGain targeting \(Section[C\.3](https://arxiv.org/html/2610.12794#A3.SS3)\) is on for DABStep only\.

Table 7:Domain hyperparameters and tool availability\. All listed domains edit candidates in a task workspace and expose on\-demand history queries; DABStep additionally enables InfoGain targeting and warehouse tools\.∗Cascade evaluation is enabled for Circle Packing \(Square\) with thresholds\[0\.5,0\.8\]\[0\.5,0\.8\]and disabled for the remaining math and ADRS tasks\.Signal Processing exposes per\-task query tools because its evaluator returns per\-signal scores, although it has noVVand does not use InfoGain\. AI functions use theℓp\\ell^\{p\}roster and full validation evaluation rather than InfoGain targeting; they are omitted from the table because their remaining settings follow Section[3\.2](https://arxiv.org/html/2610.12794#S3.SS2)\.

### C\.5Budgeted Populationℓp\\ell^\{p\}Objective and Parent Selection

Let𝒜t\\mathcal\{A\}\_\{t\}denote the archive after iterationtt\. Suppose each programp∈𝒜tp\\in\\mathcal\{A\}\_\{t\}is evaluated alongNNdimensions, with per\-dimension scoresi​\(p\)s\_\{i\}\(p\)\. A dimension may represent a task instance, objective, or behavioral criterion\. We associate each program with the set

ℐ⁡\(p\)=\{i∈\[N\]:si​\(p\)≥θi\}\\mathcal\{I\}\(p\)=\\\{i\\in\[N\]:s\_\{i\}\(p\)\\geq\\theta\_\{i\}\\\}of dimensions on which it is considered successful\. The thresholdsθi\\theta\_\{i\}may be absolute or determined relative to the current population\. For binary per\-dimension scores,ℐ⁡\(p\)\\mathcal\{I\}\(p\)is exactly the set of dimensions solved by the program\. For continuous scores, we use either a calibrated absolute threshold or a per\-dimension population quantile, allowing each dimension to be compared against its own achievable score range\.

##### Budgeted diversity roster\.

Althoughℋt\\mathcal\{H\}\_\{t\}retains the complete evaluation history, only a budgeted active rosterℛt⊆𝒜t\\mathcal\{R\}\_\{t\}\\subseteq\\mathcal\{A\}\_\{t\},\|ℛt\|≤K\|\\mathcal\{R\}\_\{t\}\|\\leq K, is used to represent the population and select the next parent\. Starting withℛt\(0\)=∅\\mathcal\{R\}^\{\(0\)\}\_\{t\}=\\emptysetandC\(0\)=∅C^\{\(0\)\}=\\emptyset, we greedily choose

pj=arg⁡maxp∈𝒜t∖ℛt\(j−1\)​\|ℐ⁡\(p\)∖C\(j−1\)\|,p\_\{j\}=\\arg\\max\_\{p\\in\\mathcal\{A\}\_\{t\}\\setminus\\mathcal\{R\}^\{\(j\-1\)\}\_\{t\}\}\|\\mathcal\{I\}\(p\)\\setminus C^\{\(j\-1\)\}\|,define its marginal contribution as

mj=\|ℐ⁡\(pj\)∖C\(j−1\)\|,m\_\{j\}=\|\\mathcal\{I\}\(p\_\{j\}\)\\setminus C^\{\(j\-1\)\}\|,and update

ℛt\(j\)=ℛt\(j−1\)∪\{pj\},C\(j\)=C\(j−1\)∪ℐ⁡\(pj\)\.\\mathcal\{R\}^\{\(j\)\}\_\{t\}=\\mathcal\{R\}^\{\(j\-1\)\}\_\{t\}\\cup\\\{p\_\{j\}\\\},\\qquad C^\{\(j\)\}=C^\{\(j\-1\)\}\\cup\\mathcal\{I\}\(p\_\{j\}\)\.Construction stops afterKKprograms or when no remaining program contributes new coverage\. This is the standard greedy maximum\-coverage construction: it retains programs because of capabilities not already represented by higher\-ranked roster members, rather than solely because of aggregate fitness\.

##### Populationℓp\\ell^\{p\}score\.

Given the ordered roster marginals𝐦=\(m1,…,m\|ℛt\|\)\\mathbf\{m\}=\(m\_\{1\},\\ldots,m\_\{\|\\mathcal\{R\}\_\{t\}\|\}\), we define

Lp​\(ℛt\)=‖𝐦‖pN=1N​\(∑jmjp\)1/p\.L\_\{p\}\(\\mathcal\{R\}\_\{t\}\)=\\frac\{\\\|\\mathbf\{m\}\\\|\_\{p\}\}\{N\}=\\frac\{1\}\{N\}\\left\(\\sum\_\{j\}m\_\{j\}^\{p\}\\right\)^\{1/p\}\.Equivalently, the evaluator may report itspp\-th power

Jp​\(ℛt\)=Lp​\(ℛt\)p=∑jmjpNp,J\_\{p\}\(\\mathcal\{R\}\_\{t\}\)=L\_\{p\}\(\\mathcal\{R\}\_\{t\}\)^\{p\}=\\frac\{\\sum\_\{j\}m\_\{j\}^\{p\}\}\{N^\{p\}\},which induces the same ordering for fixedpp\. Atp=1p=1, the score is the fraction of dimensions covered by the roster\. Forp\>1p\>1, it also rewards consolidation: for the same total coverage, the score is higher when fewer programs carry that coverage\. Greedy roster construction retains complementary capabilities, while the objective rewards programs that combine them\.

##### ℓp\\ell^\{p\}as an optimization objective\.

For tasks in which population quality is itself the target, a newly evaluated childptp\_\{t\}is assessed by inserting it into the candidate pool, rebuilding the roster, and measuring

Δ​Jp=Jp​\(ℛt∣𝒜t∪\{pt\}\)−Jp​\(ℛt∣𝒜t\)\.\\Delta J\_\{p\}=J\_\{p\}\(\\mathcal\{R\}\_\{t\}\\mid\\mathcal\{A\}\_\{t\}\\cup\\\{p\_\{t\}\\\}\)\-J\_\{p\}\(\\mathcal\{R\}\_\{t\}\\mid\\mathcal\{A\}\_\{t\}\)\.The evolutionary loop then directly optimizesJpJ\_\{p\}\. This is the formulation used in our AI\-function setting withp=2p=2, whereJ2=∑jmj2/N2J\_\{2\}=\\sum\_\{j\}m\_\{j\}^\{2\}/N^\{2\}rewards both broad population coverage and the consolidation of specialist capabilities\.

##### ℓp\\ell^\{p\}as a parent\-selection policy\.

When the task retains its original scalar objective, the same roster can instead be used only to choose the next parent\. We assign each roster member probability

Pr⁡\(pj∣ℛt\)=mjp∑ℓmℓp\.\\Pr\(p\_\{j\}\\mid\\mathcal\{R\}\_\{t\}\)=\\frac\{m\_\{j\}^\{p\}\}\{\\sum\_\{\\ell\}m\_\{\\ell\}^\{p\}\}\.This samples programs according to their contribution to the populationℓp\\ell^\{p\}score without replacing the task’s evaluator\. Atp=0p=0, the policy is uniform over roster members; atp=1p=1, it is proportional to exclusive marginal coverage; our defaultp=2p=2emphasizes large complementary contributions quadratically; and asp→∞p\\rightarrow\\infty, it concentrates on a maximum\-marginal roster member\. An agentic variant exposes the same roster, marginals, and population score to an LLM and allows it to choose the parent directly, using the mechanical distribution as a fallback\. Table[9](https://arxiv.org/html/2610.12794#A4.T9)reports a HotpotQA sweep ofp∈\{0\.5,1,1\.5,2\}p\\in\\\{0\.5,1,1\.5,2\\\}for this parent\-selection policy\. Thelparm usesp=2p=2\.

#### C\.5\.1Rare\-capability carrier selection

Marginal coverage alone can under\-allocate mutations to valuable specialists\. Suppose most programs solve the same large set of easy dimensions, while different programs solve only one or two distinct hard dimensions\. The greedy roster retains these specialists, but forp\>1p\>1its mechanical distribution can place nearly all probability on the first member, whose marginal includes the common easy dimensions\. To remove this dependence on raw instance frequency, let

ρi=1\|𝒜t\|∑p∈𝒜t𝕀\[i∈ℐ\(p\)\],hi=\(1−ρi\)γ,\\rho\_\{i\}=\\frac\{1\}\{\|\\mathcal\{A\}\_\{t\}\|\}\\sum\_\{p\\in\\mathcal\{A\}\_\{t\}\}\\mathbb\{I\}\[i\\in\\mathcal\{I\}\(p\)\],\\qquad h\_\{i\}=\(1\-\\rho\_\{i\}\)^\{\\gamma\},whereρi\\rho\_\{i\}is the population solve rate andγ\>0\\gamma\>0controls how strongly rare dimensions are emphasized\. We define the hard\-capability mass carried by a program as

H\(p\)=∑i=1Nhi𝕀\[i∈ℐ\(p\)\]\.H\(p\)=\\sum\_\{i=1\}^\{N\}h\_\{i\}\\,\\mathbb\{I\}\[i\\in\\mathcal\{I\}\(p\)\]\.We incorporate rare\-capability mass into parent selection by sampling according to

qrare​\(p\)∝exp⁡\(β⁡\[λ​Φ~​\(p\)\+\(1−λ\)​H​\(p\)p\]\),q\_\{\\mathrm\{rare\}\}\(p\)\\propto\\exp\\\!\\left\(\\beta\\left\[\\lambda\\,\\widetilde\{\\Phi\}\(p\)\+\(1\-\\lambda\)H\(p\)^\{p\}\\right\]\\right\),whereΦ~\\widetilde\{\\Phi\}is normalized aggregate fitness\. The fitness term favors the strongest carrier of a given rare capability, rather than weak programs in general, whileH​\(p\)pH\(p\)^\{p\}gives increasing preference to lineages that accumulate multiple hard capabilities\. If all specialists carry one equally rare capability and have equal fitness, they remain indistinguishable and are sampled symmetrically; once one lineage combines multiple rare capabilities, its selection probability increases\. Because no selector can exploit a dimension that no current program solves, this policy can be mixed with the baseline selector,

q⁡\(p\)=\(1−ϵ\)​qrare​\(p\)\+ϵ​qbase​\(p\)\.q\(p\)=\(1\-\\epsilon\)q\_\{\\mathrm\{rare\}\}\(p\)\+\\epsilon q\_\{\\mathrm\{base\}\}\(p\)\.The rare\-capability term is part of the parent\-selection mechanism\. It changes which existing program is selected for mutation without adding a donor or changing the evaluation pool\. All candidates are evaluated on the same instances, so their capabilities remain directly comparable\.

## Appendix DAblation Details

This section reports four ablations against the mechanisms of Appendix[C](https://arxiv.org/html/2610.12794#A3)and Section[3](https://arxiv.org/html/2610.12794#S3)\. Parent selection \(Appendix[D\.1](https://arxiv.org/html/2610.12794#A4.SS1)\) compares four parent\-selection setups and examines theℓp\\ell^\{p\}sampling exponent\. Three claim\-driven ablations each isolate one further axis of the evolutionary loop: mutation execution and self\-validation \(Appendix[D\.2](https://arxiv.org/html/2610.12794#A4.SS2)\), the history\-access interface \(Appendix[D\.3](https://arxiv.org/html/2610.12794#A4.SS3)\), and similarity\-manifold targeting \(Appendix[D\.4](https://arxiv.org/html/2610.12794#A4.SS4)\)\.

##### Common controls\.

Each claim\-driven ablation below estimates the effect of one named treatment while holding the rest of the loop fixed; a comparison that changes two axes at once is not reported as a causal ablation\. Unless stated as the named treatment, every arm of a given ablation shares the same backbone model, temperature, and model\-facing mutation budget; the same seed program and train/test split; the same parent\-selection policy and population configuration; the same editable artifact surface; the same number of outer\-loop iterations; and the same evaluator thresholds and per\-child evaluation allowance\. Persistent per\-program and archive\-level solved\-question state is retained across arms, so a no\-graph arm does not remember only the current minibatch\. The tables below report the final metrics available for each ablation\. Each subsection states the number of trials\. These controls are already in force below unless a subsection states an exception\.

### D\.1Parent selection

We test whether useful parent selection requires both preserving complementary capabilities and directing mutations toward programs that can consolidate them\. The four\-way comparison examines which programs remain eligible as parents; the HotpotQA exponent sweep probes how mutation effort is allocated among them\.

##### Setup\.

We implement four parent\-selection policies in the same OpenEvolve codebase\. Within each task, all arms use the same seed, split, backbone, mutation process, island configuration, and 100\-iteration budget\. Only the parent\-selection policy changes\.select\-bestalways chooses the current best program\.map\-elitesuses OpenEvolve’s exploration/exploitation/fitness draw from the MAP\-Elites grid\.pareto\-frontiersamples the per\-instance Pareto frontier by win count, as in GEPA\.lpbuilds a greedyKK\-roster from per\-instance solved sets and samples memberjjwith probability proportional tomjpm\_\{j\}^\{p\}, wheremjm\_\{j\}is its marginal coverage \(Appendix[C\.5](https://arxiv.org/html/2610.12794#A3.SS5)\); herep=2p=2\. The frontier and roster draw from all islands\. Table[8](https://arxiv.org/html/2610.12794#A4.T8)compares the complete setups on each task\.

To probe whether stronger preference for large marginal contributions helps the roster consolidate specialists, we compare HotpotQA configurations withp∈\{0\.5,1,1\.5,2\}p\\in\\\{0\.5,1,1\.5,2\\\}while keeping the roster construction fixed\. All four useopenai\-gpt\-5\.6\-luna, a 3\-example mutation minibatch from train\-150, val\-300 exact match for search and export, and one 100\-iteration search per setting\. Thep=2p=2run reuses the HotpotQAlpresult above;ppis the only treatment varied in this comparison\.

##### Results\.

Table[8](https://arxiv.org/html/2610.12794#A4.T8)shows the results for the four parent\-selection setups and Table[9](https://arxiv.org/html/2610.12794#A4.T9)shows the results for the HotpotQAℓp\\ell^\{p\}sweep\.

Table 8:Parent\-selection setups on HotpotQA and IFBench\. One 100\-iteration search per arm; entries are single held\-out test evaluations\. HotpotQA uses exact match; IFBench uses its per\-constraint score\. EachΔ\\Deltais the score difference from OpenEvolve’s defaultmap\-elitesarm on the same task\. Thelprow usesp=2p=2\. Bold marks the highest observed score in each task\.Table 9:HotpotQA validation and held\-out test exact match acrossℓp\\ell^\{p\}roster exponents\.Δ\\Deltais the test\-EM difference fromp=1p=1; bold marks the highest observed score\.
##### Analysis\.

These selectors use different evidence to identify useful parents\.select\-bestspends every mutation on the current incumbent, leaving complementary capabilities in other programs unused\.map\-elitespreserves programs in distinct grid cells, but those cells need not correspond to distinct solved instances\.pareto\-frontieruses per\-instance wins, yet win\-count weighting does not discount wins already represented by another parent\. The greedyℓp\\ell^\{p\}roster instead retains a program only while it adds uncovered instances, then favors roster members with larger exclusive contributions\. Theℓp\\ell^\{p\}roster has the highest observed held\-out score on both tasks in Table[8](https://arxiv.org/html/2610.12794#A4.T8), consistent with allocating mutations by complementary coverage rather than grid diversity alone\.

The exponent sweep probes how to allocate mutations after complementary parents have entered the roster\. Flatter weighting spends more turns on small\-marginal specialists; quadratic weighting concentrates ordinary mutations on parents that already contribute many distinct solves, giving later edits more opportunities to consolidate those capabilities\. Thep=2p=2row leads on both validation and held\-out test, consistent with a balance between consolidation pressure and specialist access\. The dip atp=1\.5p=1\.5shows that the effect is not simply monotonic inpp: changing the weighting changes which lineages receive repeated edits, and those search paths can produce different final programs\.

### D\.2Mutation execution and self\-validation

This ablation separates two benefits of an interactive mutator: editing the current candidate with access to its environment, and testing that candidate before committing it\. The first can support broad instruction\-level fixes; the second is especially useful for changes to executable code or dependent objects\.

##### Setup\.

We run this ablation on Heilbronn Convex \(N=13N\{=\}13\) and DABStep\. Within each task, the three arms share the parent policy, population and island settings, feature grid, history\-access interface, iteration budget, targeting configuration, and reflector/proposer configuration\. They differ in the mutator’s action space\. Heilbronn Convex has no instance similarity graph; DABStep keeps its similarity graph enabled across all three arms\. The reflector/proposer remains on in every arm, matching the published configurations\. A1 \(*static mutator*\) is one LLM completion over the parent, ledger, and active paradigm; it cannot inspect the workspace or execute anything\. A2 \(*agentic, no candidate testing*\) can edit the candidate and query history throughevo\.py, but cannot execute the edited candidate before committing it\. A3 \(*full mutator*, published\) adds pre\-commit execution: a local Python sweep on Heilbronn Convex, or deployment and end\-to\-end testing of the edited agent on DABStep\. On DABStep, A1 can only diagnose from injected artifacts, A2 can additionally query the database but cannot deploy the edited agent, and A3 has the complete query, deployment, and testing interface\.

##### Results\.

We report three Sonnet\-5 trials per Heilbronn Convex arm\. One trial stopped after 83 iterations; the others ran for 100\. Its score is included in the reported summary, so that arm’s mean is not based on three equal\-budget trials\. One 100\-iteration trial is reported per arm on DABStep\. All DABStep Train/Test values are fresh full\-pool re\-evaluations of each arm’s champion rather than in\-run estimates based on limited evaluations; A1’s initial estimate used a 25\-question bootstrap sample\. A1 never advanced past the first evaluation gate, so its champion is the unmodified seed also reported in Table[1](https://arxiv.org/html/2610.12794#S3.T1)and as A1 in Table[13](https://arxiv.org/html/2610.12794#A4.T13)\. A3 on both tasks reuses the published champion rather than a new run; the DABStep champion is also the C3 arm in Table[13](https://arxiv.org/html/2610.12794#A4.T13)\. EachΔ\\Deltais the raw\-count difference from A1\.

Table 10:Mutation\-execution ablation on Heilbronn Convex and DABStep\. Values summarize three trials per arm for Heilbronn Convex and one full\-pool champion evaluation per arm for DABStep;Δ\\Deltais relative to A1\.
##### Analysis\.

Performance improves monotonically from the static mutator \(A1\), to agentic editing without pre\-commit candidate testing \(A2\), to the full mutator \(A3\) on both tasks\. The mean Heilbronn Convex score rises from0\.02600\.0260to0\.02700\.0270to0\.030660\.03066, while DABStep test accuracy rises from47/15047/150to98/15098/150to137/150137/150\. The two transitions add different capabilities: A1 to A2 adds direct editing and on\-demand inspection, while A2 to A3 adds execution of the edited candidate before it is committed\.

Heilbronn Convex shows that the A1\-to\-A2 improvement is not explained by broader exploration: both arms attempt a similar range of strategies\. The difference is how those ideas are incorporated\. A1 frequently replaces large portions of the solver; in one matched trial, roughly one quarter of its scored edits regress and another third have no effect\. A2 generally preserves the existing solver and adds components around it, reducing regressions to about one in eight, although it learns whether an edit worked only after evolutionary evaluation\. A3 can test proposed changes through scratch execution and regresses on roughly4%4\\%of scored edits across the trials\. Agentic editing therefore helps preserve prior functionality, while self\-validation turns more of the budget into measured refinement\.

DABStep reveals why the relative gains differ across tasks\. A1 sometimes identifies real instruction problems from injected context, but cannot inspect the database or run the revised agent\. A2 can query the database, allowing it to correct fee\-rule enumeration; the same accepted edit also tightens the response contract for exact\-match answers\. Because both changes affect many questions, they raise train/test accuracy from33\.7%/31\.3%33\.7\\%/31\.3\\%to67\.3%/65\.3%67\.3\\%/65\.3\\%\. This closes roughly55%55\\%of the A1\-to\-A3 gap, compared with about22%22\\%on Heilbronn Convex\. A3 additionally deploys and tests the edited agent, allowing it to validate a structural replacement for the fee\-computation procedure and reach95\.3%/91\.3%95\.3\\%/91\.3\\%\. Thus, database\-grounded editing can capture broad instruction\-level improvements, whereas executable\-code and dependent\-object rewrites benefit most from end\-to\-end validation\.

### D\.3History\-access interface

We test two ways to give the mutator access to earlier search: a short ledger injected into each turn and structured tools for retrieving prior programs and traces\. The four arms compare each mechanism alone and both together on Signal Processing and DABStep\.

##### Setup \(Signal Processing\)\.

The selected parent’s code and evaluation are available in every arm; this ablation changes access to other programs’ history\. B1 \(*parent\-only*\) receives no additional history\. B2 \(*ledger\-only*\) receives recent approaches, score changes, stagnation information, and the active paradigm in its prompt, but cannot fetch prior programs or traces\. B3 \(*tools\-only*\) can request earlier programs, lineage, scores, and thinking traces throughevo\.py, but receives no ledger\. B4 \(*full*\) has both mechanisms\. All four arms use Sonnet\-5 for 100 iterations with the same population, islands, evaluator, and model\-facing mutation budget\. The evolution graph and reflector/proposer are off throughout\. B4 therefore uses the published Signal Processing history\-access interface without the reflector/proposer\.

##### Results \(Signal Processing\)\.

Table[11](https://arxiv.org/html/2610.12794#A4.T11)reports the bestoverall\_scorereached in one trial per arm\.

Table 11:History\-access interface ablation, Signal Processing\. One 100\-iteration Sonnet\-5 trial per arm, reflector/proposer off throughout\. Bestoverall\_scorereached within the 100\-iteration budget \(iterations 73, 97, 99, and 75 respectively\)\.
##### Analysis \(Signal Processing\)\.

Scores rise from0\.66140\.6614for B1 to0\.70740\.7074with the ledger alone \(B2\) and0\.74870\.7487with tools alone \(B3\)\. B4, with both mechanisms, reaches0\.76810\.7681\. Thus each mechanism helps relative to parent\-only in these runs, and the combination scores highest\. The B4 score is near the published Sonnet\-5 mean of0\.76560\.7656\(best0\.78410\.7841\), although the reflector/proposer is disabled in this ablation\.

The four final Signal Processing programs use different filtering designs\. B1 combines a quadratic ridge\-trend filter with a noise\-adaptive moving average\. B2’s champion describes an adaptive Kalman filter\. B3 adds Rauch–Tung–Striebel backward smoothing to a Kalman filter\. B4 combines Kalman and Savitzky–Golay smoothing with a median prefilter and slope\-hysteresis logic\.

The inspected winning turns illustrate how the interfaces are used\. B1, with no access to other attempts, proposes a larger\-window Savitzky–Golay filter and Holt smoothing, tests both, rejects them, and keeps a small parameter change\. B3 first queries the best programs, recent scores, and lineage, then verifies a refinement of its Kalman/smoothing filter with local score sweeps\. B4 recognizes a plateau of more than thirty iterations, queries high\-scoring and diverse programs, and uses a local parameter sweep to find an improvement at a narrow threshold boundary\. The B4 turn shows the ledger’s broad progress summary and targeted history lookup being used in the same edit\.

##### Setup \(DABStep\)\.

DABStep uses the same four ledger/history\-tool settings as Signal Processing\. Its evolution graph and reflector/proposer remain enabled in every arm\. The graph selects target questions on the controller side, while this ablation changes the history visible to the mutator; turning the graph off would change a second part of the loop\. The reflector/proposer also stays fixed across arms\. For B1 and B2, the coreevo\.pyhistory commands and the task\-specificget\_performance,get\_suggested\_questions, andget\_artifactshistory commands are removed\. B3 and B4 retain them\. All four arms keep the six commands used to inspect and test the current candidate:query,test\_agent,analyst\_query,check\_deploy,execute\_sql, andtest\_draft\.

##### Results \(DABStep\)\.

We report one primary 100\-iteration Sonnet\-5 trial per arm\. Each champion is freshly scored on all 300 training and 150 held\-out test questions\. B4 reuses the published flagship champion\. A second B4 run is discussed below\.

Table 12:History\-access interface ablation on DABStep\. One primary 100\-iteration Sonnet\-5 trial per arm; each champion is freshly scored on the full 300\-question training and 150\-question test pools\. Rows follow B1–B4 arm order\.
##### Analysis \(DABStep\)\.

The four primary runs show a different pattern from Signal Processing\. Ledger\-only B2 scores below parent\-only B1 on both train \(210210versus234234\) and test \(9999versus116116\)\. Tools\-only B3 is only slightly above B1 \(242242train and119119test\)\. Full B4 reaches286286on train and137137on test, the highest result in this comparison\. On DABStep, the large observed gain appears when both history\-access mechanisms are enabled; neither mechanism alone shows a comparable gain\. This is the central result of the DABStep four\-arm table\.

##### A useful fix lost at the evaluation gate\.

The B1 and B2 traces show that discovering a useful fix does not guarantee it survives search\. Both arms repeatedly identify a seed instruction that lists only five merchants from an older dataset version\. For an unlisted merchant, the seed can answer “Not Applicable” without calling a tool, even though the current data contains thirty valid merchants\. B1 repairs the instruction, and its assigned target question also improves, so the edit enters the surviving lineage\. B2 makes and tests the same kind of repair in its first iteration, but its assigned target is unrelated to the merchant bug; the evaluation gate rejects the edit, and later candidates still inherit the stale instruction\.

##### The ledger helps when the mutator can inspect prior programs\.

The ledger gives the mutator a live\-updated record of earlier approaches and projected scores\. Its score column is revised as the similarity mechanism estimates full\-pool performance, so a ledger value can differ from the program’s immediate measured result\. In one traced case, a program scored0\.17670\.1767on its immediate evaluation but later appeared in the ledger at0\.40110\.4011\. B1 sees no such history table\. B2 receives the changing projections on every turn but cannot retrieve the earlier programs and traces needed to judge what a high\-scoring entry actually does\. The ledger\-only arm is therefore exposed to uncertain historical scores without a way to check the underlying work, which helps explain its weaker result\. B3 and B4 can inspect that work through their history tools\.

##### How B3 and B4 use history\.

B3 trails B1 through much of the search and gains most in its final thirty iterations\. Before two late improvements, the mutator queries prior programs, lineage, scores, and thinking traces\. It then builds a counterfactual simulator for questions about alternate fee schemes and lookup cubes for card\-scheme and ACI questions, testing both before acceptance\. B1 also attempts substantial late changes, including a case bank for difficult questions; the distinction in these traces is that B3 consults the population’s history before deciding where to work, not that B1 is incapable of complex edits\.

B4 reaches B1’s final score by roughly iteration4040and improves further\. Around iteration3636, it queries earlier high\-scoring programs and suggested questions, then builds and checks a vectorized aggregation engine for group\-by questions\. The checks expose an unsupported array/object parameter type, which the mutator works around before the edit is kept\. Around iteration8484, it adds a computation mode for ACI\-selection questions and checks it against known cases\. These episodes show the full arm using its history tools while carrying out substantial, verified edits\.

##### Variation within the full configuration\.

A second B4 run uses the same configuration and model as the flagship but scores248/300248/300on train and114/150114/150on test, rather than286/300286/300and137/150137/150\. It ends near B3’s primary242/300242/300and119/150119/150result despite having both history\-access mechanisms available\. The two B4 runs have similar Bash\-call volume per iteration \(32\.332\.3versus33\.733\.7\), but allocate those calls differently\. In the flagship,20922092of32293229Bash calls invokeevo\.pyand974974invokeevo\_sweep\.py, about95%95\\%combined\. In the second run,16711671of33673367invokeevo\.pyand127127invokeevo\_sweep\.py, about53%53\\%combined\. The remaining calls, many of which inspect the edited blueprint directly, are far more common in the second run \(about47%47\\%versus5%5\\%\)\. The tool\-use pattern and outcome vary together in these two runs; tool use was observed rather than assigned as another ablation treatment\.

### D\.4Similarity\-manifold targeting

We test whether the true question\-similarity graph improves DABStep target selection and expansion\. The full semantic configuration \(C3\) scores well above the uniform and shuffled controls in these runs\.

##### Setup\.

All three arms use DABStep’s 300\-question training pool and keep the mutator, history\-access interface, parent policy, first\-evaluation gate, regression\-sample size, expansion\-question count, scout count, and acceleration budget fixed\. The evolution graph remains enabled in every arm\. The arms differ in how they select a target and expand evaluation to previously untested questions, as described below\.

C1 \(*uniform matched control*\) samples the target uniformly from the same unsolved candidate set, keeps the regression step identical, and evaluates the same number of random previously untested questions, including the same scout count; persistent solved\-state, donor registry, and target\-attempt counts are retained\. C2 \(*shuffled similarity graph*\) runs C3’s identical information\-gain formula, but reads the similarity term off a symmetric index permutation of the real matrix rather than the real values — same value distribution, wrong question pairing, isolating whether the specific semantic geometry matters rather than merely having*some*structured \(non\-uniform\) sampling bias\. C3 \(*full semantic manifold*\) selects the target by information gain over questions the parent has not solved, runs the usual regression sample from the parent’s cumulative solved set, and expands the search to semantically similar previously untested questions; archive\-wide solvers may be retrieved as donors\. All three arms share the same regression\-risk sampler: whatever target a given arm’s rule selects, that target’s most at\-risk previously\-solved questions are drawn from the true, unpermuted similarity graph\. Only target selection and untested\-question expansion differ across arms\.

C3’s champion is the flagship run already reported in Table[1](https://arxiv.org/html/2610.12794#S3.T1); we report a fresh full\-pool re\-evaluation of that same champion here, rather than the in\-run bookkeeping trajectory used elsewhere, to match C1/C2’s methodology exactly\.

##### Results\.

One 100\-iteration Sonnet\-5 trial per fresh arm \(C1, C2\); C3 and A1 reuse the flagship and seed champions already reported elsewhere, each freshly re\-evaluated against the same full pools as C1/C2\.

Table 13:Similarity\-manifold targeting on DABStep\. C1 and C2 each use one 100\-iteration Sonnet\-5 search; C3 reuses the published flagship and A1 the seed\. All four champions are freshly scored on the full 300\-question training and 150\-question test pools\.Δ\\Deltais relative to A1\. The flagship’s in\-run trace reached293293on train, while its independent full\-pool re\-score here is286286\.
##### Analysis\.

C3 reaches286/300286/300on train and137/150137/150on test, well above C1 \(95/30095/300,42/15042/150\) and C2 \(113/300113/300,51/15051/150\)\. C1 scores below the seed \(101/300101/300,47/15047/150\), while C2 is only slightly above it\. A repeat held\-out evaluation of C2’s champion scored49/15049/150rather than51/15051/150, underscoring the small size of that margin\.

The per\-question comparison in Table[14](https://arxiv.org/html/2610.12794#A4.T14)shows both more gains and fewer regressions for C3: it gains189189questions and regresses44, versus2929gained/3535regressed for C1 and2828gained/1616regressed for C2\. Most gains in every arm transfer beyond its directly targeted questions \(C3:143/189143/189; C1:25/2925/29; C2:22/2822/28\)\. All arms use the true graph for regression\-risk sampling, so aligning target selection and expansion with that graph may help protect previously solved neighbors\. These runs evaluate those two semantic operations together; they do not separate their contributions\.

Table 14:Per\-question diff against the A1 seed baseline on the training pool \(300 questions\), by arm\. Net is Gained minus Regressed\. Gain via target/transfer partitions each arm’s Gained count by whether that question was ever one of the arm’s own Phase\-0 targets across its 100 iterations\.

## Appendix EPrompts Used

CoCoEvolve uses Cortex Code as its mutator, alongside a reflector and proposer\. The mutator edits the candidate in its workspace\. The reflector chooseskeep,new, ornonefrom the approach ledger;newinvokes the proposer to name a strategy\. The excerpts below show the instructions relevant to these roles\.

The math, ADRS, and Signal Processing mutators share the same generic interface: an injected ledger, on\-demandevo\.pyhistory queries, and a task workspace for editing and execution\. DABStep replaces the generic system message with a Cortex\-agent builder prompt that adds warehouse tools\. The data engineering prompt defines the scaffolding of the context\-curation pipeline that CoCoEvolve improves\.

RicherhistoricalevidenceisavailableondemandthroughaCLI:

python3evo\.pyget\_best\_programs\-n5

python3evo\.pyget\_lineage–id<pid\>

python3evo\.pyget\_program–id<pid\>

python3evo\.pyget\_recent\_scores\-n20

Thecurrentprogramisalreadywrittentoafileinthisdirectory\.

Editthatfiledirectly\.Donotpastecodeintothereply\.

Beforefinishing,writethedeclaredapproachtoAPPROACH\.txt:

thesinglewordhillclimbornew\.

YouareanexpertSnowflakeCortexAgentbuilder\.Youevolveagent

blueprintstoimproveaccuracyonpaymentprocessingquestions\.

Diagnosispriority:testbeforeyoudiagnose\.Runtheexistingtool

thefailingquestionwoulduse\(query,test\_agent,oranalyst\_query\)\.

Ifitreturnstherightresult,thefixisrouting;ifnot,thefix

isthetoolitself\.

Liveinvestigationtools\(python3evo\.py<subcommand\>\):

query–read\-onlySQLagainstlivetables

test\_agent–runascratchcloneofthecurrentagent

check\_deploy–compile\-checkacandidateDDL

execute\_sql–CREATE/CALLagainstthewarehouse

test\_draft–deployyoureditedprogram\.jsononquestionQ

Youevolvethe\*\*KGconstructionpipeline\*\*fordata\-eng\-benchtasks\.

\#\#WhataKnowledgeGraphis\(andwhyitmatters\)

Eachdata\-eng\-benchtaskisasmallanalyticsengineeringchallenge:givena

databaseandan‘instruction\.md‘,anagentmustwritedbtmodelsthatpass

pytest\.Beforethatcodingstepruns,\*\*step1\*\*producesa\*\*KnowledgeGraph\*\*

—astructuredmarkdownfile\(‘KG\.md‘,≤300lines\)thatactsasabriefing

documentforthecoder\.AKGthatonlyrestates‘instruction\.md‘is

useless\.AgoodKGiswarehouse\-groundedandcaptures:

\-Whatthetaskasksfor\(outputs,metrics,grain\)

\-Whichschemas/tables/columnsexistandhowtheyjoin

\-Modellayering\(staging→intermediate→mart\),macros,andbusinessrules

\-Edgecases,validationchecks,andpitfallstoavoid

Aseparate\*\*Harborcodingagent\*\*\(fixed,notunderyourcontrol\)reads

‘KG\.md‘andwritesdbtSQL\.Beforescoring,itmayiterateagainst

\*\*KG\-derivedself\-tests\*\*\(‘test\_kg\.py‘compiledfromthatKGafterstep1\)\.

Thoseself\-testsareNOTthegrader—thehiddenverifierisunchanged\.

\*\*Your\*\*jobistodesign\*\*one\*\*KGfactoryreusedoneverydata\-eng\-bench

task—thepipelinethat\*creates\*thatKG\(andthelean

‘kg\-adherence\-test\-author‘protocolthatshapeshowtestsarederived\)viaskillsandagentorchestrationdescriptions\.

Betterarchitecture→richerKGs\+granularself\-tests→higherpytestpassrate\.

\#\#Thepipelineexplained

Yourpipeline\*\*exploresthetaskenvironmentand

produces‘KG\.md‘,whichishandedofftoadifferent,fixedagentthatsolves

thetask\.\*\*Thatistheonlyfixedcontract\.Everythingelse—howmany

agentsdotheexploring/writing,and\*\*whatreusableskillstheKGfactory

follows—isyourstodesign\.\*\*Harborstep2receives\*\*only\*\*‘KG\.md‘,not

catalogorruntimeskillfiles\.Theskillcatalog\*\*SHIPSwiththree

section\-skills\*\*\(‘kg\-agent\-usage\-guide‘,‘kg\-metrics\-and\-business\-rules‘,

‘kg\-structure‘\)\.Afourthcatalogkey,\*\*kg\-adherence\-test\-author\*\*,is

compile\-timeonly\(leanprotocolforderivinggranulartest\_kg\.pyfromeach

KG—notaKG\.mdsection\)\.Yourjob

isto\*\*EDITorGROWtheexistingpipeline\*\*:authorNEWskillsandDIVIDETHE

LABORacrossagentsorREFINE/IMPROVEtheexistingskills\.Refine

kg\-adherence\-test\-authorwhentracesshowtestsaretoogenericorHarbor

ignorestheself\-testloop\.Dropanyskillthatstopsearningitsplace\.

\#\#Twoaxesofarchitectureimprovement

\*\*Axis1—Agenttopology\*\*\(whorunswhere,doingwhat\)

\-Edit‘agents\[\]‘:anorderedlistofsequentialagentsessions\.

\-TheseedisONE‘architect‘thatexploresandwrites‘KG\.md‘inthesame

session\(queryresultsstayincontext\)\.ThisisaSTARTINGpoint\.

Donotstripthewarehouselook—afactorythatwritesfromthe

instructionaloneproducesauselessbriefing\.

\-Add,split,merge,orreorderagentsfreely\.Splitinto

explorer→architectonlyif‘explore\_coverage‘showsthelookwasskipped

andaseparatelookstepwouldhelp—ahandofffilelosesrawtooloutput\.

\-Writeeachagent’s‘prompt‘\(exploretheDB,synthesizefindings,verify

facts,renderthefinalKG,etc\.\)\.

\-Attachprovenskillstospecificpipelinestepsvia‘agents\[\]\.skills‘\.

SkillsguideKGexplorationandauthoringinsidethefactory;Harborstep2

receives\*\*only\*\*thegeneratedKG\(notcatalogorruntimeskillfiles\)\.

Agentsshareoneworkspace\.Preferlook\+writeinthesamesessionso

‘sql\_execute‘resultsstayincontext—ahandofffilelosesrawtooloutput\.

\*\*Axis2—Skills\*\*\(whatreusableprotocolstheteamneeds\)

\-Add,edit,orremovekeysin‘skills\{\}‘:‘\{description\}‘\.Attacheachskilltoapipelinestepvia‘agents\[\]\.skills‘\.

\-Theassignedagentwritesthefullskillthiseval→

‘runtime\_skills/<name\>\.md‘\.Youneverpastemarkdownintothespec\.

\#\#Evidence\(howtoimproveeitheraxis\)

Donotguesstopologyorskills\.JustifyanAxis1orAxis2editwitha

recurringgapintheparenteval:Harbortraces

\(‘agent\_trace\.txt‘\),verifierstdout\(‘verifier\_output\_raw\.txt‘\),

and/or‘explore\_coverage‘\.

\#\#Fullsystem\(threeplatformsteps\)

Step1—KGpipeline\(youdesign\)→‘KG\.md‘

Step1b—Platformcompiler\(fixed\)→‘test\_kg\.py‘fromKG\+‘kg\-adherence\-test\-author‘

Step2—Harboragent\(fixed\)→dbtmodels\+self\-testiteration→pytestscore

\#\#Yourartifact:‘\_PIPELINE\_SPEC‘

Editbetween‘\#EVOLVE\-BLOCK\-START‘and‘\#EVOLVE\-BLOCK\-END‘:

agents—orderedsteps:\{name,prompt,skills\[\]\}

skills—catalogof\{description\}perkey\(justificationoptional;assignedagentwrites\.md\)

\#\#Constraints

\-ValidPythondict;generictemplatesonly\(notask\-specifictable/columnnames\)\.

\-Agentpromptskeep‘\{task\_name\}‘,‘\{db\_name\}‘,‘\{timeout\_sec\}‘placeholders\.

\-FinalKG≤300lines;nosolutionSQLintheKG\.

\-Neverputmarkdownskillbodiesin‘skills\{\}‘—metadataonly\.

Youareastrategy\-poolcontrollerforanevolutionary

code\-optimizationrun\.Oneworkerslothasjustfreedupandyou

decideONEthingforit:feeditintoanexistingactivestrategy,

spawnabrand\-newstrategy\(aproposeragentwilldesignit\),or

runplainhillclimbingwithnonamedstrategy\.YoudoNOTdesign

thenewstrategyyourselfandyoudoNOTeditcode\.

Youareanexpertalgorithmresearcherandstrategicadvisor\.

SuggestbreakthroughalgorithmicIDEASinaJSONobjectwithan

’ideas’array\.Eachideadescribesahigh\-levelstrategy\(notcode\)\.

Focusonideasthatarefundamentallydifferentfromwhathasbeen

tried\.YouMUSTalwaysreturnatleastoneidea\.

## Appendix FAgent Optimization

### F\.1Detailed Methodology

DABStep optimization uses the InfoGain branch of Algorithm[1](https://arxiv.org/html/2610.12794#alg1)over a 300\-question training pool, with a disjoint held\-out test split reported in Table[1](https://arxiv.org/html/2610.12794#S3.T1)\. The run behind Table[15](https://arxiv.org/html/2610.12794#A6.T15)uses a 100\-iteration budget, one island, population 40, cascade threshold\[0\.005\]\[0\.005\], a task workspace,evo\.pylineage tools, and the warehouse overlay of Appendix[C\.2](https://arxiv.org/html/2610.12794#A3.SS2)\. The seedp0p\_\{0\}is a Cortex Agent blueprint with instructions,payment\_analyticstext\-to\-SQL,docs\_search, and a single\-transactionfee\_calculator\. The mutator may rewrite the instructions, tool schemas, stored procedures, and dynamic tables\. GEPA receives the same specification but optimizes only its instruction text\. A post\-hoc full\-pool evaluation of its champion scores142/300142/300on train and59/15059/150on test \(Table[1](https://arxiv.org/html/2610.12794#S3.T1)\)\.

Meta\-Harness\([Lee et al\., 2026](https://arxiv.org/html/2610.12794#bib.bib18)\)receives the same seed and 100\-iteration budget through a faithful port of the Terminal\-Bench\-2 loop\. The proposer prompt contains only a file pointer and receives no curated history or pre\-injected per\-question signal, matching the original design\. We make two changes to the port\. Cortex replaces the Anthropic API as the proposer backend, and the pooled\-parallel DABStep evaluator replaces Harbor sandboxes\. Each child is a Cortex Agent JSON specification scored on all 300 training questions\.

The main\-paper Meta\-Harness row uses the self\-test variant\. Its proposer receivestest\_draft,test\_agent, andcheck\_deploy, which provide the same self\-validation surface as CoCoEvolve’s A3 arm \(Appendix[D\.2](https://arxiv.org/html/2610.12794#A4.SS2)\)\. A held\-out re\-evaluation of the champion produces80\.0%80\.0\\%easy accuracy \(20/2520/25\),78\.4%78\.4\\%hard accuracy \(98/12598/125\), and78\.7%78\.7\\%total accuracy \(118/150118/150\), with no evaluator errors\. These are the values in Table[1](https://arxiv.org/html/2610.12794#S3.T1)\. Figure[3](https://arxiv.org/html/2610.12794#A6.F3)plots best\-so\-far train accuracy against iteration for all three methods\.

The remaining two rows are no\-search baselines, each scored once on all 450 questions with the same evaluator and scorer as the search methods\. The Snowflake Cortex Code Agent row scores the unmodified seed; the Stock Cortex Agent row scores a general\-purpose coding agent without the seed’s specification\.

Snowflake Cortex Code Agentredeploys the unmodified seedp0p\_\{0\}\. Its instructions,payment\_analyticstext\-to\-SQL tool,docs\_search, andfee\_calculatorremain unchanged\. This row measures what the seed already solves before optimization\.

Stock Cortex Agentremoves the Cortex Agent product entirely\. A single Cortex Code CLI call usingclaude\-sonnet\-5answers each question in a fresh session without memory or an evolutionary loop\. The model receives the DABStep documentation and read\-only SQL access to the live tables\. It does not receive a semantic view, text\-to\-SQL tool, fee\-calculation procedure, or the seed’s domain\-specific instructions\. Together, these two rows bound performance without search: the first retains the seed specification, while the second retains only a general\-purpose coding agent with raw data access\.

### F\.2Combined\-suite leaderboard

Table[15](https://arxiv.org/html/2610.12794#A6.T15)places our combined\-pool evaluations beside public DABStep leaderboard entries\([Egg et al\., 2025](https://arxiv.org/html/2610.12794#bib.bib14)\)and the context\-engineered configuration of[Döhmen et al\. \(2026\)](https://arxiv.org/html/2610.12794#bib.bib13)\. Our CoCoEvolve, Meta\-Harness\([Lee et al\., 2026](https://arxiv.org/html/2610.12794#bib.bib18)\), GEPA\([Agrawal et al\., 2026b](https://arxiv.org/html/2610.12794#bib.bib2)\), and no\-search rows are evaluated on the 300 training and 150 test questions together; the Meta\-Harness and GEPA rows use the champions from Table[1](https://arxiv.org/html/2610.12794#S3.T1)\. Public leaderboard submissions report combined\-suite accuracy without a separate train/test breakdown\. The table is ranked by hard\-question accuracy\.

Table 15:Accuracy on the combined 450\-question DABStep suite, ranked by hard\-split accuracy\.Figure 3:DABStep best\-so\-far train accuracy \(/300/300\) versus iteration\. CoCoEvolve and Meta\-Harness \(self\-test\) are running\-best curves from their evolution logs\. GEPA has no per\-iteration full\-pool trace because its search evaluates small subsamples, sometimes only one question per step, so its final post\-hoc score appears as a dashed reference line\. The second dashed line shows CambioML energent\.ai DS Agent, an external combined\-pool leaderboard submission without a matched budget or train\-300 trajectory\. Its easy and hard accuracies are94\.44%94\.44\\%and57\.67%57\.67\\%, respectively \(Table[15](https://arxiv.org/html/2610.12794#A6.T15)\)\. Appendix[F\.3](https://arxiv.org/html/2610.12794#A6.SS3)analyzes Meta\-Harness’s post\-iteration\-53 stall\.
### F\.3Why self\-test tools did not prevent Meta\-Harness’s churn

Figure[3](https://arxiv.org/html/2610.12794#A6.F3)shows Meta\-Harness’s self\-test variant landing its one real gain at iteration5353and then failing to improve for the remaining4747iterations despite holding the sametest\_draft/test\_agent/check\_deployself\-validation surface throughout\. We inspected the archived per\-iteration tool\-call transcripts to understand the stall\. They show two recurring problems: repeated searches for nonexistent question tables and semantic\-view edits that pass the preliminary check but fail at deployment\. These traces explain failures in this run; they do not establish that either problem would persist under every use of the self\-test tools\.

##### The query tool works; what recurs is a dead end nobody flagged, relearned every time\.

Meta\-Harness can query the real warehouse schema successfully\. At iteration6161, its proposer runsDESCRIBE TABLE DABSTEP\_PAYMENTSand receives a valid column list\. In the same iteration, it readsDABSTEP\_FEESandDABSTEP\_TXN\_FEE\_FACTcorrectly\.

The recurring failure is different\. On each occurrence, the proposer spends nine calls looking forDABSTEP\_QUESTIONS,DABSTEP\_EVAL\_QUESTIONS, andDABSTEP\_EVAL\. These tables do not exist because the question and answer bank is stored only infrontier\_val\.jsonandevolution\_summary\.jsonl\. The prompt tells the proposer to read these files for per\-question pass rates, but it never states that they are the only source of question\-level information\.

Both harnesses start from the sameinitial\_program\.json,agent\_spec, andreferenced\_tables\. The difference is therefore not the tool or seed\. It is what each prompt surfaces on every turn and whether a discovered fact persists\.

##### The self\-test tool cannot see the specific bug class that recurs\.

The DDL failure is structural\. A Cortex Analyst semantic view contains two representations that must remain synchronized: an outer SQLtables\(\)/relationships\(\)clause and an embedded Cortex Analyst extension JSON object\. The JSON separately lists tables, facts, and dimensions\.

At iteration6161, the proposer edits both representations with ad hoc Python string operations\. It inserts a relationship into the outer clause and separately patches the JSON table list instead of generating both from one source of truth\. The two representations can therefore drift apart\. In this case, a table appears intables\(\)but not in the JSON\.

The resulting text is syntactically well formed, socheck\_deploycannot reject it in advance\. That tool checks whether the DDL compiles but does not perform a data\-flow check \(mh\_tools\.py\)\. It can catch syntax failures, as it did for an unescaped quote in an earlier iteration, but it cannot validate cross\-references between the two representations\. Only the live Snowflake compiler exposes that error, and it returns only a line and position rather than a diff\.

The observed failures point to distinct remedies\. A prompt that identifies where question\-level data lives and carries learned dead ends forward could reduce repeated fruitless queries\. Structural validation of the semantic view’s two representations could catch the DDL mismatch before deployment\. In this run, self\-test access alone did not supply either safeguard\.

### F\.4Examples of Best Programs

DABStep’s artifact is a Cortex Agent spec\([Snowflake Inc\., 2026a](https://arxiv.org/html/2610.12794#bib.bib28)\)\. The champion reached an in\-run train estimate of293/300293/300\(combined\_score0\.9770\.977\)\. A fresh full\-pool re\-evaluation of the same artifact scored286/300286/300, as reported in Table[13](https://arxiv.org/html/2610.12794#A4.T13)\. While the mutator discovered a suite of optimizations across the specification \(including refined verified queries, precise orchestration instructions for disambiguation, and injected domain context\), we highlight the architectural shift we believe contributed most to its performance: demoting the text\-to\-SQL surface in favor of a deterministic compute kernel\. By routing fee, ranking, and glossary questions around the LLM entirely, it avoids freehand SQL errors\. The champion agent achieved this by rewriting the custom tool from a 5KB SQL procedure to a 53KB Python engine with 14 modes, backed by NumPy, SciPy, and NetworkX\. The agent can therefore answer complex questions with a single tool call instead of summing many single\-row fees in context, directly addressing the failure mode discussed in Section[3](https://arxiv.org/html/2610.12794#S3)\.

The deployed agent reads the following abridged specification, which heavily guides it to trust its new modes over its text\-to\-SQL capabilities\. Key routing instructions and tool descriptions added by the mutator are highlighted in blue:

\{

”agent\_spec”:\{

”instructions”:\{

”response”:”Provideprecise,data\-drivenanswers\.Alwaysshowtheexactnumbersfromqueryresults\.<abridged…\>”,

”orchestration”:”YouareaPaymentOperationsAnalystagentspecializedintheDabsteppaymentprocessingdataset\.<abridged…\>

@@\#\#ToolSelectionRules

\#\#\#Newfee\_calculatormodesforresidualquestionfamilies

RoutetheseDIRECTLYtofee\_calculatorinsteadoffreehandSQLviapayment\_analytics:

\-”For\[accounttypeXand/orMCCdescription:Y…\]whatwouldbetheaveragefee…”\-\>mode=’avg\_fee\_criteria’

\-”WhichfeeID\(s\)applyto<criteria\>”withnomerchantscope\-\>mode=’fee\_ids\_criteria’

\-”Howmanyunique/distinct<column\>arethere”\-\>mode=’distinct\_count’

<abridged…\>

\#\#FeeRuleEngine\(fee\_calculatortool\)–generalizedmodes

Thefee\_calculatortoolisageneralizedvectorizedfee\-matchingengineoveramerchant’srealtransactions\(itsumsALLmatchingfeerulespertransaction,thenaggregates\)\.Routethesequestioncategoriestoitdirectlyinsteadofhand\-writingSQL:<abridged…\>

Trustthistool’snumbersovermanuallyderivedSQLforthesecategories–italreadyhandlesmonthlyfraud/volumebracketcomputation,capture\_delaybucketing,andthecorrectintracountrydefinition\.<abridged…\>@@”

\},

”tools”:\[

\{

”tool\_spec”:\{

”type”:”cortex\_analyst\_text\_to\_sql”,

”name”:”payment\_analytics”,

”description”:”Usethistooltoquerypaymenttransactiondata,merchantinformation,feerules,andacquirerdetails\.<abridged…\>”

\}

\},

\{

”tool\_spec”:\{

”type”:”cortex\_search”,

”name”:”docs\_search”,

”description”:”Usethistooltosearchpaymentprocessingdomaindocumentationincludingthemerchantguide,feeoptimizationstrategies,fraudpreventionbestpractices\.<abridged…\>”

\}

\},

\{

”tool\_spec”:\{

”type”:”generic”,

”name”:”fee\_calculator”,

”description”:”@@Generalizedvectorizedfee\-ruleengineforamerchant’stransactions\.Usethistool\(NOTfreehandSQL\)forANYquestionabout:\(1\)whichcardschemeamerchantshouldsteertraffictoforminimum/maximumfees–mode=’steer’;\(2\)totalfeesamerchantpaid/shouldpayoverayear/month/day–mode=’total’;<abridged…\>@@”,

”input\_schema”:\{

”type”:”object”,

”properties”:\{

”p\_mode”:\{

”description”:”Oneof:’steer’\(best/worstcardschemeformin/maxfees\),’steer\_aci’\(best/worstACItomoveamerchant’sfraudulenttransactionstowardsformin/maxfees…<abridged…\>”,

”type”:”string”

\}

<abridged…\>

\}

\}

\}

\}

\]

\}

\}

A warehouse stored procedure backs thefee\_calculatortool\. While we omit the full 53KB Python body, the evolution trajectory reveals several striking mechanistic and semantic discoveries that the agent authored and verified via workspace execution:

- •Vectorized tensor contraction\.The initial tool used SQL to evaluate 9\-field conditional fee rules\. The champion replaces this with a sparse boolean tensor contraction in Python\. It precomputes a boolean “allowed” matrix for each field where wildcards become an all\-True row, combines them via bitwise&, and computes the final fees using sparse matrix\-vector products \(scipy\.sparse\.csr\_matrix\)\.
- •Semantic bug fixes\.The mutator fixed two silent bugs in the human\-authored seed\. First, it corrected a whitespace bug in a string label \(’\> 5’to’\>5’\) that had silently disabled 92 fee rules for 6 merchants\. Second, it redefined “intracountry” transactions fromIP\_COUNTRY = ISSUING\_COUNTRYtoISSUING\_COUNTRY = ACQUIRER\_COUNTRY, a definitional shift that alters the evaluation of 70% of the 138,236 rows in the dataset\.
- •Emergent domain semantics\.The agent discovered through trial and error that fees*stack*\. Its code comments explicitly warn its future self: “because fees STACK… Do NOT take the MAX of a single matching rule… that produces a different \(and wrong\) ranking\.”
- •Graph\-based join resolution\.Rather than letting the orchestrator guess join keys for text\-to\-SQL, the tool hardcodes the schema’s foreign\-key graph and usesnetworkx\.shortest\_pathto emit deterministicJOIN \.\.\. ONfragments, returning an explicit refusal if no path exists\.
- •Regex over semantic search\.For exact\-definition questions, the agent deliberately replaces vector search with a regex anddifflibparser over the documentation, reasoning that embeddings might return a topically\-related chunk that omits the exact definition\.

Beyond the champion tool shown above, other evolutionary lineages explored alternative structural solutions\. The most notable alternatives materialized dynamic tables to pre\-compute frequently\-joined columns and eliminate flaky on\-the\-fly joins and produced detailed sets of verified queries encoding domain\-specific reasoning paths\. While these alternative approaches were interesting, they ultimately lost to the champion’s centralized Python compute kernel\. Across all lineages, the agent also continually refined its orchestration guidelines, encoding lessons from observed failures into tight situational guardrails\.

## Appendix GAI Functions

### G\.1Synthetic Diagnostic: Lockpicking with Interference

##### Construction and protocol\.

This diagnostic isolates recovery of interacting rules from the semantic uncertainty of the natural\-data tasks\. Each example is a six\-digit stringP1​…​P6P\_\{1\}\\ldots P\_\{6\}with an integer label in\{0,…,6\}\\\{0,\\ldots,6\\\}\. A position first passes a position\-specific candidate\-set check; twelve ordered trigger rules \(24 trigger–target edges\) can then invalidate otherwise\-valid positions\. The label is the number of positions that remain valid\. For example,

P1∈\{1,4,7\}\.P\_\{1\}\\in\\\{1,4,7\\\}\.Two rules that can subsequently invalidateP1P\_\{1\}are

P2=8⟹P1,P3​invalid,P3=3⟹P1,P2​invalid\.P\_\{2\}=8\\Longrightarrow P\_\{1\},P\_\{3\}\\ \\mathrm\{invalid\},\\qquad P\_\{3\}=3\\Longrightarrow P\_\{1\},P\_\{2\}\\ \\mathrm\{invalid\}\.The seed provides all six candidate sets but only one of the twelve interference rules\. Every method receives the same 50 labeled strings and 50 mutation iterations\. There is no held\-out partition: train and test are the same fixed set, because the measured quantity is exact reconstruction of the latent rule system rather than distributional generalization\.

##### Why aggregate accuracy is underdetermined\.

The evaluator returns only the final count, not which positions were invalidated\. In the example above, if both triggers fire, omitting either edge intoP1P\_\{1\}need not change that row’s label\. A trigger can likewise be masked when its target was already rejected by the candidate\-set check or an earlier rule\. Consequently, many structurally different rule tables agree on most of the 50 examples\. This property makes the diagnostic useful: a locally successful patch can raise accuracy without identifying the rule responsible for the improvement\.

Table 16:Accuracy on*Lockpicking with Interference*after 50 iterations\.
##### Recovered search behavior\.

CoCoEvolve reaches50/5050/50after six mutations by contrasting multiple failures and compiling all twelve recovered rules into deterministic SQL\. OpenEvolve ends at27/5027/50with unresolved contradictions in its own worked examples\. AdaEvolve reaches45/5045/50at iteration 34 and then plateaus: only two of its twelve learned trigger rows have the correct target pair, five reuse the same\[P3,P5\]shortcut, and none can invalidateP1P\_\{1\}\. The gap is therefore structural rather than evaluation noise: the baselines fit surrogate rules that score well on the fixed set, whereas CoCoEvolve recovers the latent rule system\.

### G\.2Detailed Methodology

##### Program representation\.

Each candidate is an executable Snowflake SQL expression that forms the body of a scalar AI function\. The seed for every task contains oneAI\_COMPLETEcall, a task instruction, a row\-dependent user message, and a structured\-output schema\. Evolution may rewrite the instruction and schema descriptions, introduce CTEs or deterministic SQL post\-processing, and compose multiple model calls, but it must preserve the function signature, output field and type, and final accessor\. Thus the search space includes both prompt optimization and pipeline optimization while every candidate remains directly executable by the target query engine\. The seed is deliberately minimal: for example, thego\_emotionsseed asks for every emotion that “clearly applies,” and themultilexnormseed gives only four normalization examples\.

##### Tasks and metrics\.

The five live\-data tasks use the following row\-level objectives:go\_emotionsuses example\-level multi\-label F1;finqaandconvfinqause execution accuracy after canonicalizing the returned numeric answer;lex\_glueuses exact match over thirteen Supreme Court issue areas; andmultilexnormuses exact string match, including case, punctuation, and spacing\. The syntheticlockpicking\_6diagnostic uses exact match on the integer score\. A malformed SQL body, invalid structured output, or warehouse execution failure receives score zero together with the compilation or execution diagnostic\.

##### Data partitions\.

For each live\-data task, the search pool is partitioned once with seed 42\. Two thirds form a validation split used only for fitness and acceptance, while the remaining third supplies row contents and scorer feedback to the mutation agent\. The disjoint test table is never exposed during evolution; after the 50\-iteration run, the exported validation champion is re\-executed on that test table to obtain Table[2](https://arxiv.org/html/2610.12794#S3.T2)\. This separation is important because the mutator can inspect concrete training failures\.lockpicking\_6is intentionally different: all examples are visible and scored because its purpose is to test whether search can reconstruct a hidden rule system rather than generalize to an unseen natural\-data split\.

##### Mutation protocol\.

The AI\-function runs instantiate the general loop in Appendix[C\.1](https://arxiv.org/html/2610.12794#A3.SS1), the history interface in Appendix[C\.2](https://arxiv.org/html/2610.12794#A3.SS2), and theℓp\\ell^\{p\}roster in Appendix[C\.5](https://arxiv.org/html/2610.12794#A3.SS5)\. Their task\-specific specialization is a fresh\-session curator–mutator split: the curator inspects the roster, ledger, and training\-only failure history to select a parent, an optional merge partner, and up to ten representative rows; the mutator receives only that bounded packet and may verify its SQL on those rows up to three times\. The subsequent full\-validation evaluation is the sole admission gate\. Runs use 50 mutation attempts, one island, population 40, archive capacity 20, and eight parallel workers; similarity\-informed targeting is disabled because every child is evaluated on the complete validation split\.

##### Population objective\.

AI functions use the continuousp=2p=2instantiation of Appendix[C\.5](https://arxiv.org/html/2610.12794#A3.SS5), with a roster budget of ten\. Marginals are sums of row\-level score improvements rather than binary solved\-row counts; all other roster construction, consolidation, and parent\-selection details follow that section\.

##### Matched baselines and score provenance\.

OpenEvolve and AdaEvolve start from the same task\-specific SQL body and receive 50 mutation attempts\. Their validation champions are scored post hoc on the same held\-out test partition\. GEPA likewise optimizes the SQL body against the validation partition; Table[2](https://arxiv.org/html/2610.12794#S3.T2)uses the best completed budget for each task\. For the five live\-data tasks, the paper values are held\-out re\-evaluations rather than the optimizer’s internal archive fitness\. For lockpicking, all methods are scored on the same fixed visible set, consistent with the diagnostic protocol above\.

### G\.3Examples of Best Programs

The evolved programs remain ordinary SQL bodies, but they are substantially more operational than their seeds\. Across tasks, evolution turns a generic instruction into an explicit decision procedure: ordered rules for classification, reference and sign conventions for numerical reasoning, or token\-preservation contracts for normalization\. We give representative examples below, reproducing the two lockpicking champions in full and abridging the longer live\-task programs\. For the live tasks, the base boxes show the task\-bearing seed instruction; we omit repeated label enumerations, theAI\_COMPLETEwrapper, and the unchanged structured\-output schema\.

##### lockpicking\_6: exact rules versus a surrogate ruleset\.

The two champion artifacts make the structural difference directly inspectable\. The CoCoEvolve listing is the complete deterministic scoring expression produced by its winning child\. In the deployed AI\-function body, the resulting integer is passed toAI\_COMPLETEonly for schema\-conforming serialization\.

Scorea6\-digitprogramstring\.

Iftheinputisnotexactlysixdigits,return0\.

Validdigits:

P1:\{1,4,7\}P2:\{2,6,8\}P3:\{0,3,9\}

P4:\{1,4,7\}P5:\{0,5,8\}P6:\{2,5,9\}

Applyinterferencerulesinorder:

1\.IFP1=4\-\>invalidateP2,P6

Returnthenumberofpositionsstillvalidafterbothchecks\.

CoCoEvolve champion: deterministic scoring expression

The AdaEvolve listing is its complete45/5045/50champion prompt, including the ten worked examples accumulated during search\. We normalize typographic dashes and operators to ASCII for typesetting; the wording and rules are otherwise unchanged\.

AdaEvolve champion: evolved executor prompt

##### go\_emotions: conservative labeling and ordered rules\.

Identifyeveryemotionexpressedinthetext\.Acommentmayexpress

morethanoneemotion\.Returnanarraydrawnfromthedeclared

28\-labelvocabulary\.Includeeverycategorythatclearlyapplies;

use\[”neutral”\]onlywhennootheremotionapplies\.

The seed’s instruction to include every plausible emotion caused systematic over\-labeling\. At iteration 7, the curator queried the seed’s full failure set, inspected representative rows, and then used question memory to contextualize partial scores\. For “Happy Cake day\!” the gold label wasjoybut the parent received0\.50\.5; for “This is my favorite wild memory” it received0\.40\.4\. These scores exposed a common mechanism: the parent was returning the correct emotion together with one or more unnecessary labels\. The curator prescribed a conservative single\-label default\. The mutator’s first draft passed eight of ten selected rows; after adding rules for sarcastic positive language and social check\-ins, the same live verifier reported ten of ten\. The committed child increased the search objective from0\.00220\.0022to0\.17410\.1741in one iteration\.

The final champion generalizes this idea into a strict\-priority rule system:

ClassifyemotionsusingtheGoEmotionsannotationscheme\.

ALLOWEDLABELS—useonlythe28labelsinthedeclaredvocabulary\.

Applythefollowingrulesinstrictpriorityorder\.Recordwhichrule

matched,thenoutputonlythatrule’sprescribedemotionlist\.

RULEA—GRATITUDEEXPRESSION:

Ifthetextstartsorendswith”thanks”or”thankyou”,

output\[”gratitude”\]only\.

RULEB—EXPLICITFIRST\-PERSONEMOTION:

Checkexplicitworry,fear,love,sadness,joy,curiosity,pride,

andreliefpatternsinorder;stopatthefirstmatch\.

RULEC—INTERPERSONALSTATEQUESTION:

Foradirectquestionaboutanotherperson’swellbeing,

output\[”caring”,”curiosity”\]\.

…

DEFAULT:

Returnthesinglemostprominentemotionunlesstwoemotionsare

independentlyandexplicitlyexpressed\.

The champion was discovered at iteration 47 and obtains the held\-out score0\.4440\.444reported in Table[2](https://arxiv.org/html/2610.12794#S3.T2)\. Although the artifact is prompt\-heavy, its structure is algorithmic: ordered guards, explicit exceptions, a closed vocabulary, and a forced structured output replace the seed’s unconstrained semantic judgment\.

##### finqaandconvfinqa: executable arithmetic contracts\.

FINQA:

Answeranumericalquestionoverafinancialreport\(text\+table\)\.

Workouttheanswerandreturnthefinalnumericvalue\(oryes/no\)\.

Usenounits,thousandsseparators,orpercentsign\.

CONVFINQA:

Answerthecurrentturnofamulti\-turnnumericalconversationover

afinancialreport\.Usedialoguehistorytoresolvereferencessuch

as”thedifference”or”this”\.Returnthefinalplainnumericvalue

\(oryes/no\),withnounits,commas,orpercentsign\.

The financial\-QA seeds merely ask the model to work out a numeric answer\. Their champions instead define an operational contract for reference resolution, formula selection, sign direction, units, and answer serialization\. Thefinqaprogram distinguishes percent change, signed absolute change, relative increase, same\-period differences, ratios, averages, CAGR, and yes/no questions\. The conversational champion adds rules for resolving “that,” “the difference,” and “the last year,” and for reusing prior answers without silently substituting a nearby year\.

Workthrougheachquestionstepbystep:

1\.Identifytherelevantnumbersfromthetextortable\.

2\.Matchthequestion’sexactwordingtothecorrectformula\.

3\.Computecarefullyandshowintermediatesteps\.

FORMULASELECTION:

”percentchange”\-\>\(new\-old\)/old;preservethesign

”changeinX”\-\>new\-oldinthetable’sstatedunit

”increase/decrease”\-\>\(new\-old\)/oldasasignedratio

”howmuchbigger”\-\>X\-Y;donotdivide

”average”\-\>sum/count

”ratioofXtoY”\-\>X/Y

”CAGR”\-\>\(end/start\)^\(1/years\)\-1,thenx100

Returnplainvaluesonly:nocurrencysigns,commas,units,orpercent

signs\.Donotconvertawayfromthetable’sstatedunit\.Signmatters\.

STEP1—RESOLVEREFERENCES:

Resolvepronounsandshorthandfromthedialoguehistory\.Ifthecurrent

questionrequestsavaluealreadystatedforthesamequantityandyear,

reusethatexactvalue\.

STEP2—IDENTIFYOPERATIONANDSIGNS:

”declinefromAtoB”\-\>A\-B

”changefromAtoB”\-\>B\-A

”combinedtotal”\-\>sumthereferencedvalues

”asapercentageof”\-\>component/whole

STEP3—EXTRACTVALUES:

Listeveryrequiredvalueanditssource\.Matchtherequestedyearexactly\.

STEP4—COMPUTE:

Showthearithmeticexplicitly,thenemitaplainnumericanswer

withnounit,comma,orpercentsign\.

These contracts address errors that are difficult to repair with isolated answer demonstrations: the same numbers can require a ratio, signed difference, or magnitude depending on a short phrase\. The resulting held\-out scores are0\.8130\.813forfinqaand0\.9530\.953forconvfinqa\.

##### multilexnorm: conservative token transformation\.

Normalizenoisysocial\-mediatext\.Rewriteeachnon\-standardtoken

toitscanonicalform\(e\.g\.,u\-\>you,tmrw\-\>tomorrow,

gonna\-\>goingto,dont\-\>don’t\)\.Leavestandardwords,@mentions,

\#hashtags,andURLsunchanged\.Returnthefullnormalizedtext\.

Exact\-match normalization is unusually sensitive to helpful but unauthorized rewrites\. The evolved champion therefore begins with preservation rules, then applies a bounded lexicon of high\-confidence transformations while preserving capitalization, spacing, punctuation, mentions, hashtags, URLs, proper nouns, and uncertain or non\-English tokens\.

Rewriteeachnon\-standardEnglishtokentoitscanonicalform,

leavingeverythingelseexactlyaswritten\.

ALWAYSLEAVEUNCHANGED:

\-@mentions,\#hashtags,URLs,propernouns,andnon\-Englishwords

\-already\-standardinformalformssuchaswanna,kinda,lol,andbtw

\-uncertaintokens;whenindoubt,leavethetokenunchanged

\-emotionalsoundeffectswhoserepeatedlettersencodeemphasis

CAPITALIZATION:

Preservetheexactcapitalizationpatternoftheoriginaltoken\.

SPACINGANDPUNCTUATION:

Preserveallwhitespaceandpunctuationspacingexactly\.

HIGH\-CONFIDENCETRANSFORMS:

dont\-\>don’t;u\-\>you;ur\-\>your;pls/plz\-\>please;

tryn/tryna\-\>tryingto;thx/thnx\-\>thanks;nvm\-\>nevermind\.

The final program also records exceptions for contractions, elongated forms, and truncated\-ingwords\. This “transform only when confident” policy is the opposite of the seed’s broad invitation to normalize and achieves0\.5870\.587on the held\-out set, a7\.77\.7pp gain over GEPA\.

##### lex\_glue: hierarchy of legal exceptions\.

ClassifyaUSSupremeCourtopinionbyitsprimarySCDBissuearea\.

Chooseexactlyoneof:

CriminalProcedure,CivilRights,FirstAmendment,DueProcess,

Privacy,Attorneys,Unions,EconomicActivity,JudicialPower,

Federalism,InterstateRelations,FederalTaxation,Miscellaneous\.

Usethelabelglossary:rightsoftheaccused\-\>CriminalProcedure;

discriminationandvoting\-\>CivilRights;speech,press,andreligion

\-\>FirstAmendment;abortionandpersonalautonomy\-\>Privacy;

feesandprofessionaldiscipline\-\>Attorneys;laborrelations\-\>

Unions;antitrust,contracts,torts,andbusinessregulation\-\>

EconomicActivity;jurisdiction,standing,mootness,andremedies\-\>

JudicialPower\.Choosetheissueareathatdominatestheholding\.

The legal classifier evolves from a short glossary into a hierarchy of issue\-area distinctions\. Its champion begins with a priority scan for attorney\-fee holdings, then separates the constitutional substance of a case from its procedural posture\. For example, it distinguishes a speaker denied access to a government forum \(First Amendment\) from a captive listener resisting unwanted content \(Privacy\); federal\-court jurisdiction over an agency order \(Judicial Power\) from the substantive scope of an agency’s economic authority \(Economic Activity\); and federal labor preemption \(Federalism\) from the underlying labor right \(Unions\)\. The complete prompt is long, but these repeated “default–exception–counterexample” clauses are precisely the structure that the archive accumulated from observed confusions\. The champion scores0\.7370\.737on held\-out opinions\.

ChooseexactlyoneSCDBissueareafromthedeclared13\-labelvocabulary\.

FIRST—ATTORNEYFEESPRIORITYSCAN:

Iftheprimaryholdingconcernsattorney\-feeentitlement,computation,

recovery,ordenial,classifyasAttorneysimmediately;theunderlying

substantivedisputedoesnotchangetheclassification\.

SPEAKER/LISTENERDISTINCTION:

AspeakerdeniedaccesstoagovernmentforumisFirstAmendment\.

AcaptivelistenerresistingcompelledcontentisPrivacy\.

AGENCYCASES:

WhetherafederalcourtmayreviewanagencyorderisJudicialPower\.

Thesubstantivescopeoftheagency’sregulatoryauthorityis

EconomicActivity\(orUnionsforasubstantivelaborright\)\.

LABORPREEMPTION:

FederallaborlawdisplacingstateorlocalregulationisFederalism;

theunderlyingorganizingorbargainingrightisUnions\.

Together, these artifacts show the same pattern at different scales: evolution replaces broad task descriptions with explicit precedence, exception, and serialization rules\.

## Appendix HData Engineering

### H\.1Detailed Methodology

Data engineering optimization applies Algorithm[1](https://arxiv.org/html/2610.12794#alg1)to data\-eng\-bench\. We define 50 dev tasks and 53 held\-out test tasks, randomly sampled from the full set of 103 tasks\.

##### Knowledge graphs\.

A primary output of an evolved pipeline is a*knowledge graph*\(KG\), produced before the downstream agent attempts solving a task\. The KG is a context\-grounded markdown briefing,KG\.md, limited to 300 lines\. Evolved KGs describe schemas, tables, columns, dbt models, outputs, and metrics\. They also record joins, cardinalities,source\(\)/ref\(\)lineage, model dependencies, derivations, model grain, business rules, boundary behavior, null handling, fanout risks, validation checks, and task\-specific pitfalls\.

The KG is not an RDF store and contains no solution SQL\. The authoring pipeline verifies physical table locations, column types, categorical values, null encodings, and row counts usingSHOW,DESCRIBE,INFORMATION\_SCHEMA, andsql\_execute\. This avoids relying on a potentially stale\_sources\.ymlfile\.

##### The evolved program\.

The evolved artifact is the Python dictionary\_PIPELINE\_SPECininitial\_program\.py\. It jointly controls the KG authoring protocols, agent orchestration, and adherence test authoring protocol\. Executing this specification produces the task\-specificKG\.mdandtest\_kg\.pysupplied to the downstream coding agent\. The mutator can improve this pipeline through the following three components\.

1. 1\.KG authoring protocols\.Theskills\{\}catalog describes reusable procedures for constructing different parts of the KG, including its usage guide, metrics and business rules, and warehouse and model structure\. Catalog entries contain metadata rather than complete skill bodies\. During execution, assigned agents expand them intoruntime\_skills/<name\>\.md\. These skills guide KG production and may be shared among agents in the authoring pipeline\.
2. 2\.Agent orchestration\.Theagents\[\]list defines an ordered sequence of agent sessions that produce the KG\. Each session has an editable prompt and a set of assigned skills\. The mutator may rewrite prompts, add or remove agents, split or merge roles, reorder the pipeline, and redistribute skills\. The agents share a sanitized workspace\. This permits either a single agent or a pipeline with separate exploration, verification, and synthesis roles\.
3. 3\.Adherence test protocol\.A second form of context for the solver agent is executable tests derived from the KG\. It is a protocol used during compilation rather than a section ofKG\.md\. A fixed test authoring model consumes the KG together with this evolved protocol and producestest\_kg\.py\. Evolution can therefore improve both the declarative guidance given to the solver and the way that guidance is translated into executable feedback\.

##### Inference and evaluation\.

At inference time, the best evolved program is fixed and its executor materializes the evolved protocols and agent configuration\. All evaluated agent roles use the same out\-of\-the\-boxclaude\-sonnet\-5backbone and harness\. These roles include the ordered agent sessions that construct the KG and the downstream agent that consumes it before executing a task\. OnlyKG\.mdis registered as a skill for the Harbor agent\. The internalruntime\_skills/\*\.mdfiles are not forwarded\.

The sole model exception is the fixed adherence test author, which usesclaude\-opus\-4\-8with temperature0\.20\.2and a completion limit of 50,000 tokens\. It translatesKG\.mdand the evolved adherence test authoring protocol intotest\_kg\.py\. We hard pin this model becauseclaude\-sonnet\-5frequently produced truncated or syntactically invalid long test suites, particularly unterminated strings\. The generated suite is supplied to the Harbor agent as self\-correction feedback\. It is not the hidden grader, which remains fixed throughout evaluation\.

We note that adherence tests were generated for these evaluation runs but were not used for all runs\. We therefore treat adherence testing as optional additional context rather than a uniform component of the reported results\.

##### Search methods\.

The reported CoCoEvolve result uses the best program available after 200 iterations\. We configure one island, an active population of 15 candidates, an archive capacity of 8, 8 parallel evaluations, and the P0/P1/P2 evolution graph over the dev pool\. We compare against faithful ports of GEPA, Meta\-Harness, and AdaEvolve using the sameclaude\-sonnet\-5backbone, downstream agent, equivalent iteration budget, and seed program\_PIPELINE\_SPEC\.

### H\.2Examples of Best Programs

#### H\.2\.1Evolved Programs

The evolved specificationp∗p^\{\\ast\}retains the initial three seed skills and authors an additional four\. Two are pre\-exploration environment checks:kg\-environment\-preflightconfirms a usable, authorized warehouse, andkg\-dbt\-profiles\-default\-pathpins the~/\.dbt/default so fixes are persisted rather than session\-local\. The other two are a column\-verification protocol requiringDESCRIBEevidence for every named column, and a worked\-example protocol for weighted or ratio\-based metrics\. Thekg\-adherence\-test\-authordescription also tightens, requiring each stated business rule to be checked by live\-SQL recomputation against source and banning negative\-only assertions\. In this example, the initialarchitectagent is the sole agent; no other agents were spawned\.

\_PIPELINE\_SPEC=\{

”skills”:\{

”kg\-agent\-usage\-guide”:\{

”description”:\(

”AuthortheAgentUsageGuide:whatthedownstreamsolvermust”

”produce\(objectnames,grain\)andhowtoknowitisdone\.”

”Genericguidanceonly—notask\-specifictableorcolumnnames\.”

\),

\},

”kg\-metrics\-and\-business\-rules”:\{

”description”:\(

”AuthorMetrics&BusinessRules:recordnumericthresholds,”

”windows,andformulasfromtheinstruction\.Donotinventvalues\.”

\),

\},

”kg\-structure”:\{

”description”:\(

”AuthorStructure:confirmedsources\(livewarehouselocation\),”

”requiredmodels/outputs\(name,grain,columns,dependencies\),”

”andjoins\(keys,cardinality\)\.Verifiedwarehousefactsonly—”

”donotrestatetheinstruction\.Genericguidanceonly—no”

”task\-specifictableorcolumnnames\.”

\),

\},

\#Compile\-timeonly\(platformemitstest\_kg\.pyafterKG\.md—notaKG\.mdsection\)\.

”kg\-adherence\-test\-author”:\{

”description”:\(

”Oneprecise\[KG\-ADHERENCE\]testpertestableKGfact;failure”

”messagesquotetheKG\.Bangenericsmoketests\.”

\),

\},

\},

\#Seedtopology:onearchitect\(look\+writeinthesamesessionso

\#queryresultsstayincontext\)\.Splitonlyifexplore\_coverageshows

\#thelookwasskippedandahandoffwouldhelp\.

”agents”:\[

\{

”name”:”architect”,

”prompt”:\(

”YouarchitecttheKnowledgeGraphfortask\{task\_name\}\(database”

”\{db\_name\}\)\.Budget:\{timeout\_sec:\.0f\}s\.\\n\\n”

”Theinstructionisabrief,notthewarehouse\.Donotparaphrase”

”itintoKG\.md—thatbriefingisuselesstothesolver\.Looking”

”isnotoptional\.Use‘sql\_execute‘inTHISsessionbeforeyou”

”write;youarenotdoneuntileverytableyouwillnamehasbeen”

”DESCRIBE\(orSHOWCOLUMNS\)\.DonotstopatSHOWDATABASES\.”

”Confirmeachsource’sactualdatabase\.schema\.tableviaSHOW/”

”INFORMATION\_SCHEMA\.Donotstopbecausetheinstructionlooks”

”complete\.Forcategorical/boolean/dirty\-typedcolumnsthat”

”feedafilter,join,ormetric,GROUPBY/COUNTstoredtokens,”

”nullencoding,androwcounts\.Nameonlywhatthosequeries”

”confirmed\.Verified”

”factsonly;nosolutionSQL\.Thenwrite‘KG\.md‘\(<=300lines,”

”firstline‘\#KnowledgeGraph:\{task\_name\}‘\)\.\\n\\n”

”StructureKG\.mdbyfollowingEVERYskilllistedunder\\”Skillsto”

”author\\”below\.Authoreachas‘runtime\_skills/<name\>\.md‘before”

”youfinish\.”

\),

”skills”:\[

”kg\-agent\-usage\-guide”,

”kg\-metrics\-and\-business\-rules”,

”kg\-structure”,

\],

\},

\],

\}

\_PIPELINE\_SPEC=\{

”skills”:\{

”kg\-agent\-usage\-guide”:\{

”description”:\(

”AuthortheAgentUsageGuide:whatthedownstreamsolvermust”

”produce\(objectnames,grain\)andhowtoknowitisdone\.”

”Genericguidanceonly—notask\-specifictableorcolumnnames\.”

\),

\},

”kg\-metrics\-and\-business\-rules”:\{

”description”:\(

”AuthorMetrics&BusinessRules:recordnumericthresholds,”

”windows,andformulasfromtheinstruction\.Donotinventvalues\.”

\),

\},

”kg\-structure”:\{

”description”:\(

”AuthorStructure:confirmedsources\(livewarehouselocation\),”

”requiredmodels/outputs\(name,grain,columns,dependencies\),”

”andjoins\(keys,cardinality\)\.Verifiedwarehousefactsonly—”

”donotrestatetheinstruction\.Genericguidanceonly—no”

”task\-specifictableorcolumnnames\.”

\),

\},

\#Compile\-timeonly\(platformemitstest\_kg\.pyafterKG\.md—notaKG\.mdsection\)\.

”kg\-adherence\-test\-author”:\{

”description”:\(

”Oneprecise\[KG\-ADHERENCE\]testpertestableKGfact;failure”

”messagesquotetheKG\.Bangenericsmoketests\.Foranystated”

”filter/dedup/exclusionbusinessrule—especiallyaKGvalue”

”explicitlylabeledevidenceratherthanaprescribedtarget—”

”theassertionmustbealiveSQLcomparison:recomputethe”

”correctaggregatefromsourceinsidethetestbyembeddingthe”

”exactstatedpredicate/dedupkey,thenassertthemodeloutput”

”matchesthatlive\-recomputedvaluewithinatightnumeric”

”tolerance\.Donotassertonly’output\!=oneknown\-wrongvalue’”

”forabusinessrule—anegative\-onlycheckcanpasswhilethe”

”ruleisstillviolatedadifferentway;itmustbepairedwith”

”apositive,rule\-derivedcomparison\.”

\),

\},

”kg\-column\-verification\-protocol”:\{

”description”:\(

”BeforenaminganyspecificcolumninStructureorMetrics&”

”BusinessRules,thatcolumnmusthaveanassociated”

”DESCRIBE/SHOWCOLUMNS\(orinformation\_schema\.columns\)result”

”fromTHISsession\.Ifacolumnwasnotschema\-probed,omitit”

”ormarkitexplicitlyunverifiedratherthanstatingitas”

”fact\.Genericguidanceonly—notask\-specifictableor”

”columnnames\.”

\),

\},

”kg\-environment\-preflight”:\{

”description”:\(

”Beforeotherexploration,verifytheactualdatabasesession”

”contextthedownstreamsolverwillinherit:confirmwhich”

”warehouseisactive/authorizedfortheworkingrole\(asession”

”maystartwithnoactivewarehouse,oradefaultwarehousename”

”thatdoesnotexistorisnotauthorized\)\.Recordthe”

”confirmed\-workingwarehouseandtheexactpersistentfix\(e\.g\.”

”profile/project\-levelconfigoranexplicitUSEWAREHOUSEinthe”

”model\-buildpath\)asanAgentUsageGuidefact—nota”

”session\-localenv\-fileeditalaterstepcouldsilentlyrevert\.”

”Genericguidanceonly;notask\-specifictableorcolumnnames,”

”andnoassumptionaboutwhichwarehousenameisbrokenvs”

”workinginagivenaccount\.”

\),

\},

”kg\-dbt\-profiles\-default\-path”:\{

”description”:\(

”Authoraprofiles\-directoryfactfortheAgentUsageGuidewhen”

”thetaskrequirescreatinganewdbtproject\(ratherthan”

”editingapre\-existingone\):dbt’sCLIdefaultsto”

”‘~/\.dbt/profiles\.yml‘whenever‘–profiles\-dir‘/”

”‘DBT\_PROFILES\_DIR‘isnotexplicitlysetonagiveninvocation\.”

”Ifthesolveronlywrites‘profiles\.yml‘toacustompathand”

”reliesonexporting‘DBT\_PROFILES\_DIR‘foritsownmanual‘dbt”

”run‘calls,anyOTHERprocessthatlatercalls‘dbtrun‘/‘dbt”

”build‘inafreshshellwithoutthatenvvarwillfailatthe”

”defaultlocationinstead\.Stateexplicitly:alsoplace\(or”

”symlink\)acopyof‘profiles\.yml‘attheOS\-defaultdbt”

”profilesdirectory,notonlythecustomone\.Verifybychecking”

”whether‘~/\.dbt/‘alreadyexistsandwhatitcontainsbefore”

”assumingacustompathissufficient\.Genericdbt\-CLIbehavior”

”—notask\-specifictableorcolumnnames\.”

\),

\},

”kg\-composite\-formula\-worked\-example”:\{

”description”:\(

”ForanymetricinMetrics&BusinessRulesthatinvolvesa”

”weightedcompositescore,apercentile/rankdirection”

”\(ascendingvs\.inverted\),oraratio\-to\-median/cap\-and\-invert”

”normalization,donotdescribetheformulainprosealone\.”

”Pick1\-2realrowsfromthelivewarehouse,computeevery”

”intermediatefactorbyhandfollowingtheexactstatedformula”

”\(includingcap/invert/boundarybehaviorandNULL\-fallback”

”substitution\),andrecordthefinalnumericresultasaworked”

”exampleintheKG\.Iftheinstruction’swordingisgenuinely”

”ambiguousevenafterthisexercise,statetheambiguity”

”explicitlyasanopenquestionratherthansilentlypickingone”

”interpretationandpresentingitasfact\.Neverinvent”

”thresholdsnotpresentintheinstruction\.Genericguidance”

”only—notask\-specifictable/columnnamesorcopiedassertion”

”values\.”

\),

\},

\},

\#Seedtopology:onearchitect\(look\+writeinthesamesessionso

\#queryresultsstayincontext\)\.Splitonlyifexplore\_coverageshows

\#thelookwasskippedandahandoffwouldhelp\.

”agents”:\[

\{

”name”:”architect”,

”prompt”:\(

”YouarchitecttheKnowledgeGraphfortask\{task\_name\}\(database”

”\{db\_name\}\)\.Budget:\{timeout\_sec:\.0f\}s\.\\n\\n”

”Theinstructionisabrief,notthewarehouse\.Donotparaphrase”

”itintoKG\.md—thatbriefingisuselesstothesolver\.Looking”

”isnotoptional\.Use‘sql\_execute‘inTHISsessionbeforeyou”

”write;youarenotdoneuntileverytableyouwillnamehasbeen”

”DESCRIBE\(orSHOWCOLUMNS\)\.DonotstopatSHOWDATABASES\.”

”Confirmeachsource’sactualdatabase\.schema\.tableviaSHOW/”

”INFORMATION\_SCHEMA\.Donotstopbecausetheinstructionlooks”

”complete\.Forcategorical/boolean/dirty\-typedcolumnsthat”

”feedafilter,join,ormetric,GROUPBY/COUNTstoredtokens,”

”nullencoding,androwcounts\.Nameonlywhatthosequeries”

”confirmed\.Verified”

”factsonly;nosolutionSQL\.Thenwrite‘KG\.md‘\(≤300lines,”

”firstline‘\#KnowledgeGraph:\{task\_name\}‘\)\.\\n\\n”

”StructureKG\.mdbyfollowingEVERYskilllistedunder\\”Skillsto”

”author\\”below\.Authoreachas‘runtime\_skills/<name\>\.md‘before”

”youfinish\.\\n\\n”

”Beforeanyschemaexploration,runtheenvironment\-preflight”

”check:confirmthesessionactuallyhasausable,authorized”

”warehouse\(donotassumethedefaultenv/sessionwarehouse”

”works—itmaynotexistormaybeunauthorized\)\.Alsoconfirm”

”thedbtprofiles\-directoryresolutionfactfrom”

”kg\-dbt\-profiles\-default\-pathwhenthetaskbuildsanewdbt”

”project\.Foldboththeconfirmed\-workingwarehouse/persistent”

”fixandtheprofiles\-directoryfactintotheAgentUsageGuide”

”sectionofKG\.md;donotaddnewtop\-levelsectionsforthem”

”anddonotexceedthe300\-linebudget\.\\n\\n”

”Foranyweightedcompositescoreorpercentile/ratio\-based”

”metricinMetrics&BusinessRules,follow”

”kg\-composite\-formula\-worked\-example:computeatleastone”

”workednumericexamplefromliverowsbeforewritingthe”

”formuladown,ratherthandescribingitinprosealone\.”

\),

”skills”:\[

”kg\-agent\-usage\-guide”,

”kg\-metrics\-and\-business\-rules”,

”kg\-structure”,

”kg\-environment\-preflight”,

”kg\-column\-verification\-protocol”,

”kg\-dbt\-profiles\-default\-path”,

”kg\-composite\-formula\-worked\-example”,

\],

\},

\],

\}

#### H\.2\.2Final Pipeline Specification Artifacts

Below isKG\.mdfor taskdbt\-fix\-marketing\-attribution, produced byp∗p^\{\\ast\}\. It records verified tables, columns, data values, and environment settings for the downstream solver agent\. It also flags differences between the task instruction and the live warehouse instead of making unsupported assumptions\.

\#KnowledgeGraph:dbt\-fix\-marketing\-attribution

\#\#Structure

\*\*Confirmedsources\*\*\(database‘DBT\_BENCH\_RETAIL‘,schema‘MAIN‘;allconfirmedvia‘SHOWTABLES‘/‘SHOWVIEWS‘/‘INFORMATION\_SCHEMA\.TABLES‘\+‘DESCRIBEVIEW‘\):

\-‘DBT\_BENCH\_RETAIL\.MAIN\.STG\_GA\_\_SESSIONS‘\(VIEW,5053rows,5053distinct‘SESSION\_ID‘,0NULL‘SESSION\_ID‘/‘SESSION\_START‘\)

Columns:‘SESSION\_ID‘,‘VISITOR\_ID‘,‘CUSTOMER\_ID‘,‘CHANNEL\_ID‘,‘SESSION\_START‘\(VARCHAR—timestamp\-likestring,NOTnativedate/timestamp\),‘SESSION\_END‘\(VARCHAR\),‘DURATION\_SECONDS‘\(NUMBER\),‘PAGE\_VIEWS‘\(NUMBER\),‘LANDING\_PAGE‘,‘EXIT\_PAGE‘,‘REFERRER‘,‘UTM\_SOURCE‘,‘UTM\_MEDIUM‘,‘UTM\_CAMPAIGN‘,‘DEVICE\_TYPE‘,‘BROWSER‘,‘OS‘,‘IP\_ADDRESS‘,‘COUNTRY‘,‘IS\_CONVERTED‘\(VARCHAR\),‘ORDER\_ID‘\(VARCHAR,alwaysNULLinthisenv—0/5053populated\),‘CREATED\_AT‘,plusmetadatacols‘\_LOADED\_AT‘,‘\_SOURCE\_SYSTEM‘,‘\_BATCH\_ID‘,‘\_ROW\_NUMBER‘,‘\_ROW\_HASH‘\.

\-‘DBT\_BENCH\_RETAIL\.MAIN\.STG\_GA\_\_EVENTS‘\(VIEW,15047rows\)

Columns:‘EVENT\_ID‘,‘SESSION\_ID‘\(1131rowsNULL\),‘EVENT\_TYPE‘,‘EVENT\_NAME‘,‘EVENT\_TIMESTAMP‘\(VARCHAR;0literalNULLsbut3801/15047rowsfail‘TRY\_CAST\(…ASTIMESTAMP\)‘—dirty/malformedtimestampstrings\),‘PAGE\_URL‘,‘ELEMENT\_ID‘,‘ELEMENT\_CLASS‘,‘PRODUCT\_ID‘,‘EVENT\_VALUE‘\(VARCHARstoringnumeric\-lookingdecimalstrings,e\.g\.‘86\.52592478731349‘\),‘EVENT\_DATA‘,‘CREATED\_AT‘,plussamemetadatacolsassessions\.

\-‘DBT\_BENCH\_RETAIL\.MAIN\.INT\_SESSIONS\_EVENTS\_JOINED‘\(VIEW,LEFTJOINofthetwoabove;15775rowsvs5053distinct‘SESSION\_ID‘→confirmedfanout~3\.1xrowspersession\)

Columns\(15total,confirmedvia‘DESCRIBEVIEW‘\):‘SESSION\_ID‘,‘VISITOR\_ID‘,‘CUSTOMER\_ID‘,‘CHANNEL\_ID‘,‘SESSION\_START‘,‘SESSION\_END‘,‘DURATION\_SECONDS‘,‘PAGE\_VIEWS‘,‘T2\_EVENT\_ID‘,‘T2\_EVENT\_TYPE‘,‘T2\_EVENT\_NAME‘,‘T2\_EVENT\_TIMESTAMP‘,‘T2\_PAGE\_URL‘,‘T2\_ELEMENT\_ID‘,‘T2\_ELEMENT\_CLASS‘\.

\*\*‘T2\_EVENT\_VALUE‘DOESNOTEXISTinthisviewinthisenvironment\*\*—confirmedabsentfromthe15\-column‘DESCRIBEVIEW‘output\.Thismatchestheinstruction’swarned”mayomit‘t2\_event\_value‘entirely”branch,andinthislivewarehousethatbranchistheactualcase:revenuecannotbesourcedfrom‘int\_sessions\_events\_joined‘here\.

Rejected/decoycandidatesseeninschemasearch\(donotuse—wronggrainorduplicativeoftheabove\):‘STG\_RAW\_GA\_\_SESSIONS\*‘,‘STG\_SESSIONS\*‘,‘RAW\_GA\.SESSIONS\*‘,‘STG\_RAW\_GA\_\_EVENTS\*‘,‘FCT\_MULTI\_TOUCH\_ATTRIBUTION‘,‘FCT\_RECONSTRUCTED\_SESSIONS‘,‘RPT\_SESSION\_HOURLY\_DISTRIBUTION‘—theseareseparatestaging\-history/rebuildvariantsandpre\-existingmartsunrelatedtotherequired‘int\_sessions\_events\_joined‘/‘stg\_ga\_\_sessions‘/‘stg\_ga\_\_events‘lineagenamedbytheinstruction\.

\*\*Requiredoutput\*\*\(perinstruction,notre\-verifiedinwarehousesinceitdoesnotyetexist\):dbtmodel‘marts/marketing/rpt\_attribution\_fixed\.sql‘,materializingtoschema‘main‘\(Snowflakebackend\)or‘analytics‘\(DuckDBbackend\)\.Grain:onerowper\(‘attribution\_date‘,‘channel‘,‘medium‘,‘campaign‘\)\.Dependson‘stg\_ga\_\_sessions‘,‘stg\_ga\_\_events‘,‘int\_sessions\_events\_joined‘\(allconfirmedabove\)\.

\*\*Joins:\*\*

\-‘stg\_ga\_\_sessions\.SESSION\_ID‘=‘int\_sessions\_events\_joined\.SESSION\_ID‘—thisisthejoinalreadybakedintothe‘int\_sessions\_events\_joined‘view\(LEFTJOINsessions→events\);cardinality1:many,fanoutconfirmed5053sessions→15775joinedrows\.

\-‘stg\_ga\_\_events\.SESSION\_ID‘=‘stg\_ga\_\_sessions\.SESSION\_ID‘—1131/15047eventrowshaveNULL‘SESSION\_ID‘\(orphanevents,cannotjoin\)\.

\#\#Metrics&BusinessRules

See‘runtime\_skills/kg\-metrics\-and\-business\-rules\.md‘forthegenericcaptureprotocol\.Task\-specificverifiedfacts:

\*\*Windows:\*\*90\-daytrailingwindowanchoredto‘MAX\(CAST\(session\_startASDATE\)\)‘computedfrom‘stg\_ga\_\_sessions‘itself\(perinstruction\)—NOTanchoredtocurrentwall\-clockdate\.Confirmed‘SESSION\_START‘valuesin‘stg\_ga\_\_sessions‘rangefrom‘1765\-02\-04‘to‘2026\-01\-03‘;46/5053rowsarepre\-2020outlier/garbagedates,5007/5053fallinanormalrecentrange\.TheMAXusedforthe90\-dayanchorwillbepulledfromthissamedistribution\(maxobserved=‘2026\-01\-0319:00:08‘\)\.

\*\*Formula:\*\*‘conversion\_rate=conversions/sessions‘,divide\-by\-zeromustbehandled\(perinstruction,‘NULLIF‘\-styleguard\),roundedto4decimals\.

\*\*Default/mappingrules\(verifiedagainstactualNULLcounts\):\*\*

\-‘UTM\_SOURCE‘NULL→‘’direct’‘:387/5053sessionshaveNULL‘UTM\_SOURCE‘\.

\-‘UTM\_MEDIUM‘NULL→‘’none’‘:402/5053sessionshaveNULL‘UTM\_MEDIUM‘\.

\-‘UTM\_CAMPAIGN‘NULL→‘’none’‘:3663/5053sessionshaveNULL‘UTM\_CAMPAIGN‘\(majorityofsessionsareuncampaigned\)\.

\*\*Fallback/precedencerule\(verified,notassumed\):\*\*Instructionsaysprefer‘t2\_event\_value‘from‘int\_sessions\_events\_joined‘,elseuse‘event\_value‘from‘stg\_ga\_\_events‘\.Inthisenvironment,‘t2\_event\_value‘isconfirmedabsent\(seeStructuresection\)—thefallbackbranchto‘stg\_ga\_\_events\.EVENT\_VALUE‘istheonethatapplieshere\.

\*\*Dedup/grainrule:\*\*Instructionrequiressummingrevenuepersessionfirst,thenaggregatingtothefinal\(date,channel,medium,campaign\)grain,toavoidinflationfromtheconfirmed1:~3\.1session→event\-rowfanoutin‘int\_sessions\_events\_joined‘\.

\*\*Verifieddiscrepancies\(factsonly,notresolvedhere\):\*\*

\-Instruction’sconversionrulestates‘is\_converted=1‘\.Actualstoredtokensin‘STG\_GA\_\_SESSIONS\.IS\_CONVERTED‘\(VARCHARcolumn,confirmedvia‘GROUPBY‘\):‘’0’‘=1532,‘’1’‘=47,‘’N’‘=997,‘’Y’‘=35,‘’false’‘=2364,‘’true’‘=78\(total5053\)\.Truthy\-tokencandidates\(‘’1’‘,‘’Y’‘,‘’true’‘\)sumto160sessions\(~3\.2%\);falsy\-tokencandidates\(‘’0’‘,‘’N’‘,‘’false’‘\)sumto4893\.Aliteral‘=1‘comparisonagainstthisVARCHARcolumnwillnotmatchthe‘’Y’‘/‘’true’‘rows\.

\-Instruction’srevenue/conversion\-eventrulestates‘event\_name=’purchase’‘\(and‘t2\_event\_name=’purchase’‘\)\.Acase\-insensitivescan\(‘ILIKE’%purchas%’‘\)acrossall15047rowsof‘STG\_GA\_\_EVENTS\.EVENT\_NAME‘returned\*\*0matchingrows\*\*\.Distinct‘EVENT\_NAME‘valuesactuallypresent\(51distinct,1216rowsliteralNULL\):‘’ViewProduct’‘\(1420\),‘’Search’‘\(1418\),‘’Filter’‘\(1410\),‘’VideoPlay’‘\(1406\),‘’Click’‘\(1391\),‘’FormSubmit’‘\(1384\),‘’AddToCart’‘\(1357\),‘’Sort’‘\(1344\),‘’Scroll’‘\(1333\),‘’RemoveFromCart’‘\(1321\),plus~40additionallow\-frequencydirtytokensoftheform‘’event\_N\.0’‘\(eachcount1–2,~46rowstotal\)\.Thesameabsenceof‘’purchase’‘holdsin‘INT\_SESSIONS\_EVENTS\_JOINED\.T2\_EVENT\_NAME‘\(11distinctgroupsobserved,same10namedcategoriesplusNULL,no‘’purchase’‘\)\.

\-‘STG\_GA\_\_EVENTS\.EVENT\_VALUE‘isstoredasVARCHARcontainingdecimalnumericstrings\(e\.g\.‘’86\.52592478731349’‘\)—requiresexplicitcasttoanumerictypebeforesumming\.

\-‘SESSION\_START‘,‘SESSION\_END‘,‘EVENT\_TIMESTAMP‘areallstoredasVARCHAR,notnativeDATE/TIMESTAMP,inthisenvironment—confirmedvia‘DESCRIBEVIEW‘\.‘TRY\_CAST\(SESSION\_STARTASTIMESTAMP\)‘succeededonsampledrows;‘TRY\_CAST\(EVENT\_TIMESTAMPASTIMESTAMP\)‘failedon3801/15047rows\(dirtytimestampstringsintheeventstablespecifically,notthesessionstable\)\.

\-‘STG\_GA\_\_SESSIONS\.CUSTOMER\_ID‘isNULLon3632/5053rows\(guest/anonymoussessions\)and‘ORDER\_ID‘isNULLonall5053rowsinthisenvironment\.

\#\#Environment&ExecutionNotes

\*\*Fresh\-session,zero\-overridecheck:\*\*‘SELECTCURRENT\_WAREHOUSE\(\),CURRENT\_ROLE\(\),CURRENT\_DATABASE\(\),CURRENT\_SCHEMA\(\),CURRENT\_USER\(\),CURRENT\_ACCOUNT\(\)‘runinabrand\-new‘sql\_execute‘sessionwithnoprior‘USE…‘statementsissuedinthatsessionandnooverrideofthearchitect’soranyprioragent’scontext→literalresult:‘WH=KG\_AUTHOR\_WH,ROLE=PUBLIC,DB=DBT\_BENCH\_RETAIL,SCHEMA=MAIN,USER=DEX\_EVAL\_USER,ACCT=UOB68132‘\(query\_id‘01c72a54\-030b\-8aae\-0032\-d0c3016224b6‘\)\.Thedefaultconnectionresolvedcleanly—noNULL/none,nofailuretorecordorfix\.

\*\*Computeverification:\*\*‘SELECTCOUNT\(\*\)FROMDBT\_BENCH\_RETAIL\.MAIN\.STG\_GA\_\_SESSIONS‘onthatsameresolvedcontext→‘5053‘rowsreturnedsuccessfully\(query\_id‘01c72a54\-030b\-8aae\-0032\-d0c3016224e6‘\)\.Thisconfirms‘KG\_AUTHOR\_WH‘isnotjustanamereturnedby‘CURRENT\_WAREHOUSE\(\)‘—itisactuallyauthorizedandusablebyrole‘PUBLIC‘forrealcomputeinthisenvironment\.

\*\*Schemaalignment\(checked,notassumed\):\*\*ThetaskinstructionrequirestheSnowflakeoutputmodeltolandinschema‘main‘viaa‘generate\_schema\_name‘macrooverride\.Thelivesession’s‘CURRENT\_SCHEMA\(\)‘alreadyresolvesto‘MAIN‘withzerooverrides\.Theseagreeinthisenvironment—thereisnolivediscrepancybetweentheinstruction’shardcodedschemaliteralandthesession’sdefaultschematoflaghere\.

\*\*Instructiontodownstreamsolver:\*\*NoenvironmentfixwasneededtoproducethisKG—thedefaultwarehouse/role/database/schemaallresolvedandworkedwithzeromanualoverrides\.If,inyourownsession,youfindyoumustoverrideanyofthese\(e\.g\.toget‘dbtrun‘oryourownself\-teststopass\),leavethatoverrideinthe\*\*persisted\*\*profile/environmentusedforfinalscoring\(e\.g\.‘profiles\.yml‘,environmentvariables,orwhateverconfigfilesuppliesthesevaluesatgradingtime\)—doNOTrevertitbackafterwardunderthebeliefthatrevertingis”cleanup\.”Theindependentgradingconnectionopensitsownfreshsessionandinheritsonlythepersisteddefaults,notanythingyoufixedtransientlyinyourlivesession\.Immediatelybeforefinishing,verifywithasecond,genuinelyfreshconnection/session\(nottheoneyoujustpatched\)thatthebuildstillsucceedsundertheexactenvironmentstatethatwillbegraded\.Donotblanket\-overridetheschema/databaseliteralin‘profiles\.yml‘/macrosbasedsolelyonthissession’senvvarreading—thealignmentcheckabovealreadyconfirmstheyagreeinthisenvironment,sonooverrideiswarrantedherewithoutnewevidence\.

\#\#KnownAmbiguities&RiskFlags

\-\*\*‘is\_converted‘truthydefinition\*\*\(realCOUNTqueriesagainst‘DBT\_BENCH\_RETAIL\.MAIN\.STG\_GA\_\_SESSIONS‘,5053totalrows\):Instructionstatesasessionisconvertedif‘is\_converted=1‘\.InterpretationA—literal‘IS\_CONVERTED=’1’‘→\*\*47\*\*sessions\.InterpretationB—truthy\-tokenset‘IS\_CONVERTEDIN\(’1’,’Y’,’true’\)‘→\*\*160\*\*sessions\(‘’1’‘=47,‘’Y’‘=35,‘’true’‘=78\)\.Complementaryfalsy\-tokenset‘IS\_CONVERTEDIN\(’0’,’N’,’false’\)‘→\*\*4893\*\*sessions\(‘’0’‘=1532,‘’N’‘=997,‘’false’‘=2364\)\.Disagreement:47vs160—a~3\.4xdifferenceinhowmanysessionscountasconverted\.Affects:‘conversions‘numerator,‘conversion\_rate‘,andwhichsessions’revenueiseligibletobesummed\.Notresolvedhere—solvermustchooseaninterpretationandbepreparedtojustifyit;bothcountsarerealandverified,notmodeled\.

\-\*\*‘event\_name=’purchase’‘literalmatch\*\*\(realCOUNTqueryagainst‘DBT\_BENCH\_RETAIL\.MAIN\.STG\_GA\_\_EVENTS‘,15047totalrows,1216NULL‘EVENT\_NAME‘\):Instructionstatesconversion/revenueeventsareidentifiedby‘event\_name=’purchase’‘\(or‘t2\_event\_name=’purchase’‘inthejoinedview\)\.Literalmatch‘EVENT\_NAME=’purchase’‘→\*\*0\*\*rows\.Case\-insensitivefuzzymatch‘EVENT\_NAMEILIKE’%purchas%’‘→\*\*0\*\*rows\.Distinct\-valuescanconfirms51distinctnon\-NULL‘EVENT\_NAME‘valuesactuallypresent,noneresemblingpurchase\(topvalues:‘’ViewProduct’‘=1420,‘’Search’‘=1418,‘’Filter’‘=1410,‘’VideoPlay’‘=1406,‘’Click’‘=1391,‘’FormSubmit’‘=1384,‘’AddToCart’‘=1357,‘’Sort’‘=1344,‘’Scroll’‘=1333,‘’RemoveFromCart’‘=1321,plus~40low\-frequencydirty‘’event\_N\.0’‘tokens\)\.Risk:undertheliteralinstructionrule,‘attributed\_revenue‘willbe\*\*0foreveryrow\*\*ofthefinaloutput,becausethenamedconversion\-eventliteraldoesnotexistanywhereinthisenvironment’sdataunderanycasingtried\.Thisisastructuralrisk,notatwo\-waycountdisagreement—flaggingitratherthansubstitutingadifferentevent\-nameliteralonthesolver’sbehalf\.

Runtimeskillfilesauthoredperassignment\(fullprotocols,generic/reusable,notask\-specificnamesinsidethem\):

\-‘runtime\_skills/kg\-agent\-usage\-guide\.md‘

\-‘runtime\_skills/kg\-metrics\-and\-business\-rules\.md‘

\-‘runtime\_skills/kg\-structure\.md‘

\-‘runtime\_skills/kg\-ambiguity\-guardrails\.md‘

\-‘runtime\_skills/kg\-environment\-handoff\.md‘

The suite below turns KG statements into executable checks\. Some tests verify recorded facts against the live data warehouse, while others recompute business rules from source data and compare them with the solver’s output\. Their failure messages point the solver to the relevant KG statement, helping it locate and fix mismatches\.

”””Self\-correctiontestsuitefortask:dbt\-fix\-marketing\-attribution\.

\[KG\-ADHERENCE\]testsencodespecificfactsstatedintheKnowledgeGraph\(KG\):

requiredoutputmodel,schema,grain,columns,metricformulas,filterpredicates,

UTMdefaultmappings,dedup\-before\-aggregaterules,andtheconfirmednumeric

anchors\(160truthyis\_convertedsessions,1216NULLevent\_namerows,5053sessions\)\.

\[UNIVERSAL\]testsencodeoutput\-qualityinvariantstheKG’sOutputrequirements

imply\(noNULLrequiredcolumns,conversion\_ratein\[0,1\],conversions<=sessions\)

andcross\-sourcereconciliationagainsttheconfirmedsources\.

ThesetestsarederivedEXCLUSIVELYfromtheKG—notfrominstruction\.md,notfrom

anysolutionSQL,andnotfromthehiddengrader\.Passingthemisstrongevidencethe

agentfollowedtheKG\.AfailingtestpointstoaspecificKGsentencetore\-read

\(quotedineachfailuremessage\)\.

”””

importos

importsys

\_here=os\.path\.dirname\(os\.path\.abspath\(\_\_file\_\_\)\)

for\_pin\[\_here,os\.path\.normpath\(os\.path\.join\(\_here,”\.\.”,”\.\.”\)\)\]:

ifos\.path\.exists\(os\.path\.join\(\_p,”test\_backbone\.py”\)\):

sys\.path\.insert\(0,\_p\)

break

fromtest\_backboneimportexecute\_query,execute\_scalar,make\_conn\_fixture,require

\#──Taskconstants────────────────────────────────────────────────────────────

AGENT\_PROJECT\_DIR=”/app/dbt\_project”

\#Outputschema:main\(Snowflake\)oranalytics\(DuckDB\)\.Weresolveperdb\_type\.

SCHEMA\_DUCKDB=”analytics”

SCHEMA\_SNOWFLAKE=”main”

MODEL=”rpt\_attribution\_fixed”

MODEL\_FILE=f”\{AGENT\_PROJECT\_DIR\}/models/marts/marketing/rpt\_attribution\_fixed\.sql”

\#Confirmedsources\(databaseDBT\_BENCH\_RETAIL,schemaMAIN\)\.

SRC\_SCHEMA=”main”

STG\_SESSIONS=”stg\_ga\_\_sessions”

STG\_EVENTS=”stg\_ga\_\_events”

INT\_JOINED=”int\_sessions\_events\_joined”

\#ExpectedgraincolumnsperKG\(”onerowper\(attribution\_date,channel,medium,campaign\)”\)\.

GRAIN\_COLS=\[”attribution\_date”,”channel”,”medium”,”campaign”\]

conn=make\_conn\_fixture\(check\_paths=\[MODEL\_FILE\]\)

def\_out\_schema\(db\_type\):

returnSCHEMA\_DUCKDBifdb\_type==”duckdb”elseSCHEMA\_SNOWFLAKE

def\_fqn\(schema,table\):

returnschema\+”\.”\+table

def\_columns\(c,db\_type,schema,table\):

ifdb\_type==”duckdb”:

rows=execute\_query\(

c,db\_type,

”SELECTnameFROMpragma\_table\_info\(’”\+schema\+”\.”\+table\+”’\)”,

\)

returnset\(str\(r\[0\]\)\.lower\(\)forrinrows\)

rows=execute\_query\(

c,db\_type,

”SELECTcolumn\_nameFROMinformation\_schema\.columns”

”WHERElower\(table\_schema\)=lower\(’”\+schema\+”’\)”

”ANDlower\(table\_name\)=lower\(’”\+table\+”’\)”,

\)

returnset\(str\(r\[0\]\)\.lower\(\)forrinrows\)

\#──Prerequisite────────────────────────────────────────────────────────────

deftest\_dbt\_run\_prerequisite\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

cnt=execute\_scalar\(

c,db\_type,

”SELECTCOUNT\(\*\)FROM”\+\_fqn\(schema,MODEL\),

\)

require\(

cntisnotNoneandcnt\>0,

”\[KG\-ADHERENCE\]Requiredoutputdbtmodel’marts/marketing/”

”rpt\_attribution\_fixed\.sql’mustbebuiltandpopulated—rundbtfirst\.”,

\)

\#──Sourcesanityanchors\(confirmedinKG\)─────────────────────────────────

deftest\_source\_sessions\_row\_count\(conn\):

c,db\_type=conn

cnt=execute\_scalar\(

c,db\_type,

”SELECTCOUNT\(\*\)FROM”\+\_fqn\(SRC\_SCHEMA,STG\_SESSIONS\),

\)

require\(

cnt==5053,

”\[UNIVERSAL\]KG:’STG\_GA\_\_SESSIONS\(VIEW,5053rows\)’\.”,

\)

deftest\_source\_sessions\_distinct\_session\_id\(conn\):

c,db\_type=conn

cnt=execute\_scalar\(

c,db\_type,

”SELECTCOUNT\(DISTINCTsession\_id\)FROM”\+\_fqn\(SRC\_SCHEMA,STG\_SESSIONS\),

\)

require\(

cnt==5053,

”\[UNIVERSAL\]KG:’5053distinctSESSION\_ID,0NULLSESSION\_ID’\.”,

\)

deftest\_source\_events\_row\_count\(conn\):

c,db\_type=conn

cnt=execute\_scalar\(

c,db\_type,

”SELECTCOUNT\(\*\)FROM”\+\_fqn\(SRC\_SCHEMA,STG\_EVENTS\),

\)

require\(

cnt==15047,

”\[UNIVERSAL\]KG:’STG\_GA\_\_EVENTS\(VIEW,15047rows\)’\.”,

\)

deftest\_source\_int\_joined\_fanout\(conn\):

c,db\_type=conn

total=execute\_scalar\(

c,db\_type,

”SELECTCOUNT\(\*\)FROM”\+\_fqn\(SRC\_SCHEMA,INT\_JOINED\),

\)

distinct=execute\_scalar\(

c,db\_type,

”SELECTCOUNT\(DISTINCTsession\_id\)FROM”\+\_fqn\(SRC\_SCHEMA,INT\_JOINED\),

\)

require\(

total==15775anddistinct==5053,

”\[KG\-ADHERENCE\]KG:’INT\_SESSIONS\_EVENTS\_JOINED…15775rowsvs5053”

”distinctSESSION\_ID\-\>fanout~3\.1xrowspersession’\.Thisfanoutis”

”whyrevenuemustbesummedpersessionBEFOREfinalaggregation\.”,

\)

deftest\_int\_joined\_lacks\_t2\_event\_value\(conn\):

c,db\_type=conn

cols=\_columns\(c,db\_type,SRC\_SCHEMA,INT\_JOINED\)

require\(

”t2\_event\_value”notincols,

”\[KG\-ADHERENCE\]KG:’T2\_EVENT\_VALUEDOESNOTEXISTinthisviewinthis”

”environment’—thefallbacktostg\_ga\_\_events\.EVENT\_VALUEapplieshere\.”,

\)

\#──is\_convertedtruthyanchor\(160\)─────────────────────────────────────────

deftest\_is\_converted\_truthy\_anchor\_160\(conn\):

c,db\_type=conn

cnt=execute\_scalar\(

c,db\_type,

”SELECTCOUNT\(\*\)FROM”\+\_fqn\(SRC\_SCHEMA,STG\_SESSIONS\)\+

”WHEREis\_convertedIN\(’1’,’Y’,’true’\)”,

\)

require\(

cnt==160,

”\[KG\-ADHERENCE\]KG:truthy\-tokensetIS\_CONVERTEDIN\(’1’,’Y’,’true’\)”

”\-\>160sessions\(’1’=47,’Y’=35,’true’=78\)\.”,

\)

deftest\_is\_converted\_literal\_one\_anchor\_47\(conn\):

c,db\_type=conn

cnt=execute\_scalar\(

c,db\_type,

”SELECTCOUNT\(\*\)FROM”\+\_fqn\(SRC\_SCHEMA,STG\_SESSIONS\)\+

”WHEREis\_converted=’1’”,

\)

require\(

cnt==47,

”\[KG\-ADHERENCE\]KG:literalIS\_CONVERTED=’1’\-\>47sessions”

”\(InterpretationAoftheis\_convertedambiguity\)\.”,

\)

deftest\_is\_converted\_falsy\_anchor\_4893\(conn\):

c,db\_type=conn

cnt=execute\_scalar\(

c,db\_type,

”SELECTCOUNT\(\*\)FROM”\+\_fqn\(SRC\_SCHEMA,STG\_SESSIONS\)\+

”WHEREis\_convertedIN\(’0’,’N’,’false’\)”,

\)

require\(

cnt==4893,

”\[KG\-ADHERENCE\]KG:falsy\-tokensetIS\_CONVERTEDIN\(’0’,’N’,’false’\)”

”\-\>4893sessions\.”,

\)

\#──event\_name=’purchase’structuralrisk──────────────────────────────────

deftest\_event\_name\_null\_anchor\_1216\(conn\):

c,db\_type=conn

cnt=execute\_scalar\(

c,db\_type,

”SELECTCOUNT\(\*\)FROM”\+\_fqn\(SRC\_SCHEMA,STG\_EVENTS\)\+

”WHEREevent\_nameISNULL”,

\)

require\(

cnt==1216,

”\[KG\-ADHERENCE\]KG:’STG\_GA\_\_EVENTS…1216NULLEVENT\_NAME’\.”,

\)

deftest\_event\_name\_purchase\_absent\(conn\):

c,db\_type=conn

cnt=execute\_scalar\(

c,db\_type,

”SELECTCOUNT\(\*\)FROM”\+\_fqn\(SRC\_SCHEMA,STG\_EVENTS\)\+

”WHEREevent\_name=’purchase’”,

\)

require\(

cnt==0,

”\[KG\-ADHERENCE\]KG:literalEVENT\_NAME=’purchase’\-\>0rows\.”

”Structuralrisk:attributed\_revenuemaybe0foreveryoutputrow\.”,

\)

\#──UTMdefault\-mappingNULLanchors─────────────────────────────────────────

deftest\_utm\_source\_null\_anchor\_387\(conn\):

c,db\_type=conn

cnt=execute\_scalar\(

c,db\_type,

”SELECTCOUNT\(\*\)FROM”\+\_fqn\(SRC\_SCHEMA,STG\_SESSIONS\)\+

”WHEREutm\_sourceISNULL”,

\)

require\(

cnt==387,

”\[KG\-ADHERENCE\]KG:’UTM\_SOURCENULL\-\>direct:387/5053sessions’\.”,

\)

deftest\_utm\_medium\_null\_anchor\_402\(conn\):

c,db\_type=conn

cnt=execute\_scalar\(

c,db\_type,

”SELECTCOUNT\(\*\)FROM”\+\_fqn\(SRC\_SCHEMA,STG\_SESSIONS\)\+

”WHEREutm\_mediumISNULL”,

\)

require\(

cnt==402,

”\[KG\-ADHERENCE\]KG:’UTM\_MEDIUMNULL\-\>none:402/5053sessions’\.”,

\)

deftest\_utm\_campaign\_null\_anchor\_3663\(conn\):

c,db\_type=conn

cnt=execute\_scalar\(

c,db\_type,

”SELECTCOUNT\(\*\)FROM”\+\_fqn\(SRC\_SCHEMA,STG\_SESSIONS\)\+

”WHEREutm\_campaignISNULL”,

\)

require\(

cnt==3663,

”\[KG\-ADHERENCE\]KG:’UTM\_CAMPAIGNNULL\-\>none:3663/5053sessions’\.”,

\)

\#──Outputmodel:existence,columns,grain,uniqueness─────────────────────

deftest\_model\_exists\_as\_table\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

ifdb\_type==”duckdb”:

rows=execute\_query\(

c,db\_type,

”SELECTtable\_nameFROMinformation\_schema\.tables”

”WHERElower\(table\_schema\)=lower\(’”\+schema\+”’\)”

”ANDlower\(table\_name\)=lower\(’”\+MODEL\+”’\)”,

\)

else:

rows=execute\_query\(

c,db\_type,

”SELECTtable\_nameFROMinformation\_schema\.tables”

”WHERElower\(table\_schema\)=lower\(’”\+schema\+”’\)”

”ANDlower\(table\_name\)=lower\(’”\+MODEL\+”’\)”,

\)

require\(

len\(rows\)\>=1,

”\[KG\-ADHERENCE\]KG:Requiredoutputdbtmodel’marts/marketing/”

”rpt\_attribution\_fixed\.sql’materializingtoschema’”\+schema\+”’\.”,

\)

deftest\_model\_has\_grain\_columns\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

cols=\_columns\(c,db\_type,schema,MODEL\)

missing=\[gforginGRAIN\_COLSifgnotincols\]

require\(

len\(missing\)==0,

”\[KG\-ADHERENCE\]KGgrain:’onerowper\(attribution\_date,channel,medium,”

”campaign\)’\.Missinggraincolumns:”\+str\(missing\),

\)

deftest\_model\_has\_metric\_columns\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

cols=\_columns\(c,db\_type,schema,MODEL\)

needed=\[”sessions”,”conversions”,”conversion\_rate”\]

missing=\[nforninneededifnnotincols\]

require\(

len\(missing\)==0,

”\[KG\-ADHERENCE\]KGmetrics:sessions,conversions,conversion\_rate”

”\(=conversions/sessions\)\.Missing:”\+str\(missing\),

\)

deftest\_model\_has\_revenue\_column\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

cols=\_columns\(c,db\_type,schema,MODEL\)

require\(

”attributed\_revenue”incols,

”\[KG\-ADHERENCE\]KG:attributed\_revenuederivedfromt2\_event\_valueelse”

”stg\_ga\_\_events\.EVENT\_VALUE,summedpersessionfirst\.”,

\)

deftest\_model\_grain\_is\_unique\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

dup=execute\_query\(

c,db\_type,

”SELECTattribution\_date,channel,medium,campaign,COUNT\(\*\)ASn”

”FROM”\+\_fqn\(schema,MODEL\)\+

”GROUPBYattribution\_date,channel,medium,campaignHAVINGCOUNT\(\*\)\>1”,

\)

require\(

len\(dup\)==0,

”\[KG\-ADHERENCE\]KGgrain:’onerowper\(attribution\_date,channel,medium,”

”campaign\)’—thegrainkeymustbeunique\(noduplicatecombinations\)\.”,

\)

deftest\_model\_row\_count\_positive\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

cnt=execute\_scalar\(

c,db\_type,

”SELECTCOUNT\(\*\)FROM”\+\_fqn\(schema,MODEL\),

\)

require\(

cntisnotNoneandcnt\>0,

”\[UNIVERSAL\]Outputmodelmustproduceatleastonegrainrow\.”,

\)

\#──Dataqualityinvariants\(Outputrequirements\)───────────────────────────

deftest\_no\_null\_required\_columns\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

rows=execute\_query\(

c,db\_type,

”SELECT\*FROM”\+\_fqn\(schema,MODEL\)\+

”WHEREattribution\_dateISNULLORchannelISNULLORmediumISNULL”

”ORcampaignISNULLORsessionsISNULLORconversionsISNULL”

”ORconversion\_rateISNULL”,

\)

require\(

len\(rows\)==0,

”\[UNIVERSAL\]OutputrequirementsforbidNULLinrequiredcolumns”

”\(grainkeys,sessions,conversions,conversion\_rate\)\.”,

\)

deftest\_conversion\_rate\_in\_range\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

rows=execute\_query\(

c,db\_type,

”SELECTconversion\_rateFROM”\+\_fqn\(schema,MODEL\)\+

”WHEREconversion\_rate<0\.0ORconversion\_rate\>1\.0”,

\)

require\(

len\(rows\)==0,

”\[UNIVERSAL\]KG:’conversion\_rate=conversions/sessions’mustbein”

”\[0\.0,1\.0\]\.”,

\)

deftest\_conversions\_le\_sessions\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

rows=execute\_query\(

c,db\_type,

”SELECTsessions,conversionsFROM”\+\_fqn\(schema,MODEL\)\+

”WHEREconversions\>sessions”,

\)

require\(

len\(rows\)==0,

”\[KG\-ADHERENCE\]KG:convertedsessionsareasubsetofallsessions—”

”conversionsmustneverexceedsessions\.”,

\)

deftest\_sessions\_nonnegative\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

rows=execute\_query\(

c,db\_type,

”SELECTsessionsFROM”\+\_fqn\(schema,MODEL\)\+

”WHEREsessionsISNULLORsessions<0”,

\)

require\(

len\(rows\)==0,

”\[UNIVERSAL\]sessionsmustbeanon\-negativecountofdistinctsessions\.”,

\)

deftest\_conversions\_nonnegative\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

rows=execute\_query\(

c,db\_type,

”SELECTconversionsFROM”\+\_fqn\(schema,MODEL\)\+

”WHEREconversionsISNULLORconversions<0”,

\)

require\(

len\(rows\)==0,

”\[UNIVERSAL\]conversionsmustbeanon\-negativecount\.”,

\)

\#──conversion\_rateformularecomputation────────────────────────────────────

deftest\_conversion\_rate\_formula\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

\#divide\-by\-zeroguard:ratemustbe0\(orNULL\-guarded\)whensessions=0\.

rows=execute\_query\(

c,db\_type,

”SELECTsessions,conversions,conversion\_rateFROM”\+\_fqn\(schema,MODEL\)\+

”WHEREsessions\>0AND”

”ABS\(conversion\_rate\-ROUND\(CAST\(conversionsASDOUBLE\)/sessions,4\)\)\>0\.0001”,

\)

require\(

len\(rows\)==0,

”\[KG\-ADHERENCE\]KG:’conversion\_rate=conversions/sessions…rounded”

”to4decimals’withdivide\-by\-zerohandled\.”,

\)

deftest\_conversion\_rate\_zero\_sessions\_guard\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

rows=execute\_query\(

c,db\_type,

”SELECT\*FROM”\+\_fqn\(schema,MODEL\)\+

”WHEREsessions=0ANDconversion\_rateISNULL”,

\)

require\(

len\(rows\)==0,

”\[KG\-ADHERENCE\]KG:’divide\-by\-zeromustbehandled\(NULLIF\-styleguard\)’—”

”conversion\_rateshouldnotbeNULL/undefinedwhensessions=0\.”,

\)

\#──Cross\-modelreconciliationagainstconfirmedsources─────────────────────

deftest\_total\_sessions\_reconciles\_to\_90day\_window\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

\#KG:90\-daytrailingwindowanchoredtoMAX\(CAST\(session\_startASDATE\)\)

\#fromstg\_ga\_\_sessionsitself,countingdistinctsession\_id\.

src\_total=execute\_scalar\(

c,db\_type,

”WITHanchoredAS\(”

”SELECTsession\_id,CAST\(session\_startASDATE\)ASsd,”

”\(SELECTMAX\(CAST\(session\_startASDATE\)\)FROM”\+

\_fqn\(SRC\_SCHEMA,STG\_SESSIONS\)\+”\)ASmax\_sd”

”FROM”\+\_fqn\(SRC\_SCHEMA,STG\_SESSIONS\)\+

”\)SELECTCOUNT\(DISTINCTsession\_id\)FROManchored”

”WHEREsd\>max\_sd\-90”,

\)

model\_total=execute\_scalar\(

c,db\_type,

”SELECTSUM\(sessions\)FROM”\+\_fqn\(schema,MODEL\),

\)

require\(

src\_totalisnotNoneandmodel\_totalisnotNone

andabs\(int\(model\_total\)\-int\(src\_total\)\)<=0,

”\[UNIVERSAL\]KG:’90\-daytrailingwindowanchoredto”

”MAX\(CAST\(session\_startASDATE\)\)’withdistinctsessioncounting—”

”SUM\(sessions\)inthemartmustequaldistinctsessionsinthatwindow”

”\(model=”\+str\(model\_total\)\+”,source=”\+str\(src\_total\)\+”\)\.”,

\)

deftest\_attribution\_date\_within\_90day\_window\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

max\_sd=execute\_scalar\(

c,db\_type,

”SELECTMAX\(CAST\(session\_startASDATE\)\)FROM”\+

\_fqn\(SRC\_SCHEMA,STG\_SESSIONS\),

\)

rows=execute\_query\(

c,db\_type,

”SELECTattribution\_dateFROM”\+\_fqn\(schema,MODEL\)\+

”WHERECAST\(attribution\_dateASDATE\)<=”

”\(SELECTMAX\(CAST\(session\_startASDATE\)\)\-90FROM”\+

\_fqn\(SRC\_SCHEMA,STG\_SESSIONS\)\+”\)”,

\)

require\(

len\(rows\)==0,

”\[KG\-ADHERENCE\]KG:90\-daytrailingwindowanchoredto”

”MAX\(CAST\(session\_startASDATE\)\)\(maxobserved2026\-01\-03\)\.”

”Allattribution\_datevaluesmustfallinsidethatwindow\(max\_sd=”\+

str\(max\_sd\)\+”\)\.”,

\)

deftest\_channel\_defaults\_direct\_present\_or\_valid\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

\#KG:UTM\_SOURCENULL\-\>’direct’default\.NoNULL/emptychannelshouldremain\.

rows=execute\_query\(

c,db\_type,

”SELECTchannelFROM”\+\_fqn\(schema,MODEL\)\+

”WHEREchannelISNULL”,

\)

require\(

len\(rows\)==0,

”\[KG\-ADHERENCE\]KG:’UTM\_SOURCENULL\-\>direct’—channelmustneverbe”

”NULLintheoutput\.”,

\)

deftest\_medium\_defaults\_none\_present\_or\_valid\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

rows=execute\_query\(

c,db\_type,

”SELECTmediumFROM”\+\_fqn\(schema,MODEL\)\+

”WHEREmediumISNULL”,

\)

require\(

len\(rows\)==0,

”\[KG\-ADHERENCE\]KG:’UTM\_MEDIUMNULL\-\>none’—mediummustneverbeNULL\.”,

\)

deftest\_campaign\_defaults\_none\_present\_or\_valid\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

rows=execute\_query\(

c,db\_type,

”SELECTcampaignFROM”\+\_fqn\(schema,MODEL\)\+

”WHEREcampaignISNULL”,

\)

require\(

len\(rows\)==0,

”\[KG\-ADHERENCE\]KG:’UTM\_CAMPAIGNNULL\-\>none’—campaignmustneverbe”

”NULL\.”,

\)

\#──Revenue:dedup\-before\-aggregate&non\-negativity─────────────────────────

deftest\_attributed\_revenue\_nonnegative\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

rows=execute\_query\(

c,db\_type,

”SELECTattributed\_revenueFROM”\+\_fqn\(schema,MODEL\)\+

”WHEREattributed\_revenue<0”,

\)

require\(

len\(rows\)==0,

”\[UNIVERSAL\]attributed\_revenuemustbenon\-negative\.”,

\)

deftest\_revenue\_not\_inflated\_by\_fanout\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

\#KG:sumrevenuepersessionfirst,thenaggregate—toavoid1:~3\.1fanout

\#inflation\.Totalattributed\_revenuemustnotexceedthetotalper\-session

\#summedEVENT\_VALUEupperboundacrossthewholeeventstable\.

total\_model=execute\_scalar\(

c,db\_type,

”SELECTCOALESCE\(SUM\(attributed\_revenue\),0\)FROM”\+\_fqn\(schema,MODEL\),

\)

\#Per\-sessionsummedeventvalue\(numericcast\),asafeupperbound\.

ifdb\_type==”duckdb”:

cast\_expr=”TRY\_CAST\(event\_valueASDOUBLE\)”

else:

cast\_expr=”TRY\_CAST\(event\_valueASDOUBLE\)”

upper=execute\_scalar\(

c,db\_type,

”WITHper\_sessionAS\(”

”SELECTsession\_id,SUM\(”\+cast\_expr\+”\)ASrev”

”FROM”\+\_fqn\(SRC\_SCHEMA,STG\_EVENTS\)\+

”WHEREsession\_idISNOTNULLGROUPBYsession\_id”

”\)SELECTCOALESCE\(SUM\(rev\),0\)FROMper\_session”,

\)

require\(

total\_modelisnotNoneandupperisnotNone

andfloat\(total\_model\)<=float\(upper\)\+0\.01,

”\[KG\-ADHERENCE\]KG:’sumrevenuepersessionfirst,thenaggregate…to”

”avoidinflationfromtheconfirmed1:~3\.1session\-\>event\-rowfanout’\.”

”Totalattributed\_revenue\(”\+str\(total\_model\)\+”\)mustnotexceedthe”

”per\-sessionsummedEVENT\_VALUEbound\(”\+str\(upper\)\+”\)\.”,

\)

deftest\_revenue\_zero\_when\_purchase\_absent\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

\#KGstructuralrisk:EVENT\_NAME=’purchase’has0matchesunderanycasing\.

\#Aspec\-faithful\(literal’purchase’\)solutionyieldsattributed\_revenue=0

\#foreveryrow\.Iftheagentusedasubstituteeventtoken,thistestflags

\#itsothechoiceisdeliberate&justifiedratherthanaccidental\.

purchase\_events=execute\_scalar\(

c,db\_type,

”SELECTCOUNT\(\*\)FROM”\+\_fqn\(SRC\_SCHEMA,STG\_EVENTS\)\+

”WHEREevent\_name=’purchase’”,

\)

total\_model=execute\_scalar\(

c,db\_type,

”SELECTCOALESCE\(SUM\(attributed\_revenue\),0\)FROM”\+\_fqn\(schema,MODEL\),

\)

ifpurchase\_events==0:

require\(

float\(total\_model\)==0\.0,

”\[KG\-ADHERENCE\]KGstructuralrisk:literalEVENT\_NAME=’purchase’\-\>”

”0rows,sounderthespec’spurchaseruleattributed\_revenueis0for”

”everyoutputrow\.Gottotal=”\+str\(total\_model\)\+”—ifnon\-zero,”

”theagentsubstitutedadifferenteventtoken\(mustbejustified\)\.”,

\)

else:

require\(True,””\)

\#──Metricconsistency:conversionsmustreflectconvertedsessions─────────

deftest\_conversions\_consistent\_with\_truthy\_bound\(conn\):

c,db\_type=conn

schema=\_out\_schema\(db\_type\)

\#KG:convertednumeratordependsonis\_convertedinterpretation\(47literal

\#or160truthy\)\.Totalconversionsinthemartmustnotexceedthetruthy

\#upperboundof160\(withinthe90\-daywindowitis<=160\)\.

total\_conv=execute\_scalar\(

c,db\_type,

”SELECTCOALESCE\(SUM\(conversions\),0\)FROM”\+\_fqn\(schema,MODEL\),

\)

require\(

total\_convisnotNoneandint\(total\_conv\)<=160,

”\[KG\-ADHERENCE\]KGis\_convertedambiguity:truthyset\(’1’,’Y’,’true’\)=”

”160sessionsisthemaximumeligible\.SUM\(conversions\)=”\+

str\(total\_conv\)\+”mustnotexceed160\.”,

\)

## Appendix IMath & System Benchmark Optimization

### I\.1Detailed Methodology

All math and ADRS runs share a 100\-iteration budget and a matched backbone \(claude\-sonnet\-5oropenai\-gpt\-5\.6\-terra\)\. AdaEvolve, OpenEvolve, and GEPA mutate through a stateless completion; CoCoEvolve uses the agentic loop of Appendix[C\.1](https://arxiv.org/html/2610.12794#A3.SS1), skipping the InfoGain branch because these tasks have no instance pool\. The AdaEvolve row in Table[4](https://arxiv.org/html/2610.12794#S3.T4)reruns AdaEvolve’s control loop with the same backbone model as CoCoEvolve\. Matching the backbone removes model choice as an explanation for the observed gap between these search setups\. We report the best objective reached within the shared budget\.

##### Task instances and metrics\.

Table[17](https://arxiv.org/html/2610.12794#A9.T17)records the instance size, the scalar reported in the main tables, and the Human/SOTA source\. Higher is better except Cloudcast, which reports total broadcast cost\.

TaskInstanceReported metricHuman/SOTACircle Packing \(Square\)N=26N\{=\}26, unit squaresum of radii2\.63598Circle Packing \(Rect\)N=21N\{=\}21, perimeter 4sum of radii2\.36583Heilbronn \(Triangle\)N=11N\{=\}11min triangle area / host area0\.0365Heilbronn \(Convex\)N=13N\{=\}13min triangle area / hull area0\.0309MinMax Distanced=3d\{=\}3,N=14N\{=\}14\(min/max\)2\(\\min/\\max\)^\{2\}0\.24004Signal Processingcausal window filteroverall\_scorecomposite0\.718Cloudcast \(↓\\downarrow\)multi\-cloud broadcasttotal cost626\.2EPLBMoE expert placementbalancedness/speed mean0\.1265PrismGPU model placement1/avg​\_​kvpr1/\\mathrm\{avg\\\_kvpr\}\+ success21\.89LLM\-SQLcolumn reorder0\.95⋅0\.95\\cdothit\-rate\+0\.05⋅\{\+\}0\.05\\cdotruntime0\.692TXNschedulingcombined throughput\-style objective2,725

Table 17:Math and ADRS evaluator settings\. Math Human/SOTA is the better of the published human reference and the best previously published automated result, from AlphaEvolve\([Novikov et al\., 2025](https://arxiv.org/html/2610.12794#bib.bib23)\)and AdaEvolve\([Cemri et al\., 2025](https://arxiv.org/html/2610.12794#bib.bib4)\)\. Systems Human/SOTA values are the single AdaEvolve references\.

### I\.2Higher\-precision math and ADRS tables

The main\-text table rounds values for readability\. Tables[18](https://arxiv.org/html/2610.12794#A9.T18)and[19](https://arxiv.org/html/2610.12794#A9.T19)report values at greater precision and retain rankings computed from the underlying values\.

Table 18:Mathematical optimization benchmarks\. We reportMean±\\pmStdandBestobjective values\. Higher is better for all metrics\. \(NNvalues: Square=2626, Rect=2121, Triangle=1111, Convex=1313; MinMaxd=3d\{=\}3,N=14N\{=\}14\)\. See Table[17](https://arxiv.org/html/2610.12794#A9.T17)\. Human/SOTA is the better of the published Human reference and the best previously published automated result on each problem, drawn from AlphaEvolve[Novikov et al\. \(2025\)](https://arxiv.org/html/2610.12794#bib.bib23)and AdaEvolve[Cemri et al\. \(2025\)](https://arxiv.org/html/2610.12794#bib.bib4)\. Values at greater precision corresponding to the main\-paper Table[4](https://arxiv.org/html/2610.12794#S3.T4)\. Bold/underline follow the same within\-backbone ranking on the underlying values\.Table 19:ADRS systems benchmarks\. We reportMean±\\pmStdandBestobjective values\. Higher is better, except forCloudcastwhere lower \(↓\\downarrow\) is better\. Instance sizes and metrics are in Table[17](https://arxiv.org/html/2610.12794#A9.T17)\. Human/SOTA values are from[Cemri et al\. \(2025\)](https://arxiv.org/html/2610.12794#bib.bib4), which reports only a single reference value\. Values at greater precision corresponding to the main\-paper Table[4](https://arxiv.org/html/2610.12794#S3.T4)\. Bold/underline follow the same within\-backbone ranking on the underlying values\.Signal Processing’soverall\_scoreis a weighted blend of a compositeJ⁡\(θ\)J\(\\theta\)\(slope changes, lag, error, false reversals\), smoothness, correlation, noise reduction, and success rate\.

##### Signal Processing strategy convergence\.

Surprisingly, AdaEvolve and CoCoEvolve independently converge to essentially the same champion strategy: both fit the complete signal with an L1 data term and an L1 second\-difference penalty, expressed as the same sparse linear program, and return the aligned tail of the resulting piecewise\-linear trend\. Their material difference is only how the regularization weight is chosen\. CoCoEvolve scales a fixed coefficient by a robust noise estimate, whereas AdaEvolve uses a clipped noise\-to\-motion rule\. In the reported exchange experiment, swapping the regularization rules also swaps the measured performance\. The two champions therefore share the same broad filtering approach, and the regularization rule appears to explain their score difference in this comparison\.

### I\.3Per\-task analysis

Across both backbones, CoCoEvolve is best or tied best in 13 of 24 math cells and 12 of 20 ADRS cells when Avg and Best are counted separately\. It leads 25 of 44 cells overall, compared with 18 for GEPA, 13 for AdaEvolve, and 6 for OpenEvolve\.

On Sonnet\-5, CoCoEvolve has the highest Heilbronn Convex mean and ties GEPA on Best\. In ADRS, it leads both Avg and Best on Cloudcast, EPLB, and TXN, and it leads LLM\-SQL on Avg\. AdaEvolve leads Prism, while GEPA leads LLM\-SQL on Best\.

On GPT\-5\.6\-terra, the leaders are distributed across all four methods\. CoCoEvolve leads Prism and LLM\-SQL and has the highest Heilbronn Triangle Best\. AdaEvolve leads Signal Processing and Circle Packing \(Rect\)\. GEPA leads Heilbronn Convex and TXN and has the best Cloudcast Avg\. OpenEvolve has the best Circle Packing \(Square\) mean\. Circle Packing at both instances and MinMax Distance produce several ties at or near the reference values, so their means rather than their Bests are the informative column\.

The traces below examine two patterns in the table\. Heilbronn Convex tests whether stalled trials recover near a known reference score; TXN shows a structural rewrite with room for a larger gain\. Circle Packing provides a saturated case where final best scores are tied, and Cloudcast shows several methods finishing close together\. These traces explain selected runs, not every aggregate result\. InfoGain is inactive here: these tasks have no instance pool\.

##### Figures\.

Heilbronn Convex layouts and best\-so\-far: Figures[4](https://arxiv.org/html/2610.12794#A9.F4)–[5](https://arxiv.org/html/2610.12794#A9.F5)\(Appendix[I\.3\.1](https://arxiv.org/html/2610.12794#A9.SS3.SSS1)\)\. Circle Packing \(Square\) layouts and best\-so\-far: Figures[6](https://arxiv.org/html/2610.12794#A9.F6)–[7](https://arxiv.org/html/2610.12794#A9.F7)\(Appendix[I\.3\.2](https://arxiv.org/html/2610.12794#A9.SS3.SSS2)\)\.

#### I\.3\.1Heilbronn Convex \(N=13N\{=\}13\)

##### Setup\.

The objective is the ratio of the smallest triangle area amongN=13N\{=\}13points to the area of their convex hull; Human/SOTA is0\.03090\.0309\. The seed returns1313uniform random points \(0\.000320\.00032\)\. All four methods run the matched100100\-iteration Sonnet\-5 protocol of Appendix[I\.1](https://arxiv.org/html/2610.12794#A9.SS1); CoCoEvolve’s Avg in Table[4](https://arxiv.org/html/2610.12794#S3.T4)is three retained trials\. GEPA ties CoCoEvolve on Best, so the analysis below is not a unique\-champion story: it is why Avg stays high when two of those three trials start badly\. Figure[4](https://arxiv.org/html/2610.12794#A9.F4)shows each method’s Best\-cell geometry\. Figure[5](https://arxiv.org/html/2610.12794#A9.F5)overlays one Best\-cell trace per method and a second retained CoCoEvolve trial \(dotted\) that recovers from a stall; both CoCoEvolve traces enter the published Avg\.

Figure 4:Heilbronn Convex \(N=13N\{=\}13\) on Sonnet\-5\. Point sets in the unit square: hull vertices \(navy\), interior points \(lighter\), hull outlined, smallest triangle in red\. Seed is unstructured\. OpenEvolve reaches a weaker1313\-gon \(0\.029000\.02900\); GEPA, AdaEvolve, and CoCoEvolve all reach structured1313\-gons at their Best\-cell scores\.Figure 5:Heilbronn Convex best\-so\-far on Sonnet\-5, cropped just below the first rewrite\. Solid traces are each method’s Best\-cell run; Human/SOTA is the dashed green line\. The dotted navy trace is a second retained CoCoEvolve trial: it stalls near0\.02810\.0281and recovers to0\.030780\.03078\. Solid CoCoEvolve reaches Human/SOTA at iteration99\. The claim is Avg, not the tied Best\.
##### Search\.

The three retained CoCoEvolve finals are0\.030940\.03094,0\.030780\.03078, and0\.030250\.03025\(Table[4](https://arxiv.org/html/2610.12794#S3.T4):0\.03066±0\.000360\.03066\\pm 0\.00036\)\. Two of the three spend most of the budget near0\.0280\.028and still finish at the ceiling, which is what keeps Avg above GEPA and the standard deviation a quarter of GEPA’s\.

The solid trace is the trial that does not need a recovery\. Iterations11–22replace the random seed with from\-scratch solvers \(0\.01820\.0182, then0\.02690\.0269\)\. The reflector then keeps a constrained\-NLP family \(scipy\.optimize\.minimize, SLSQP\) and a hillclimb of that family lands Human/SOTA at iteration99, with a scratch self\-check before each commit\. GEPA reaches the same0\.030940\.03094at iteration3939\. That is sample efficiency on one trial, not the claim for the column\.

The dotted trial is the CMA\-ES recovery and contributes to the three\-run Avg\. Its first rewrite scores0\.02340\.0234and stalls near0\.02810\.0281from iteration3131while the mutator retries SciPy global solvers\. At iteration7575, the reflector marks that family as exhausted and proposes a NumPy CMA\-ES\. A scratch prototype reaches0\.030350\.03035at iteration7878\. At iteration8989, the mutator returns to a dual\-annealing program from iteration1919, which scored0\.026480\.02648, rather than using the current best CMA\-ES child\. It queries the ranked archive and adds CMA\-ES with an exact\-bilinear SLP polish, producing the0\.030780\.03078champion\.

The third retained trial \(not plotted\) is the same mechanism with a different family\. It crawls from0\.02440\.0244to0\.02750\.0275over7373iterations of scipy global solvers, then implements Tutte barycentric embedding \(scipy\.sparse\.linalg\.spsolve\) at iteration8686and jumps to0\.030250\.03025\.

Either recovery is load\-bearing for the Avg win\. If the CMA\-ES trial had remained at0\.02810\.0281and the Tutte trial at its pre\-jump0\.02750\.0275, the three\-run mean would be0\.02890\.0289, below GEPA\. If only one of the two recovered, the mean would still sit near0\.02970\.0297\. Heilbronn therefore shows how historical synthesis improves reliability near a ceiling\. The ledger triggers a new family after a stall, and the archive query transfers that family to a viable island\. This mechanism explains the high Avg rather than a uniquely high Best\.

#### I\.3\.2Circle Packing \(Square\)

##### Setup\.

The objective is the sum of radii ofN=26N\{=\}26non\-overlapping circles in the unit square; Human/SOTA is2\.635982\.63598\. The seed is a two\-ring heuristic \(0\.960\.96\)\. This instance saturates: every method in Table[4](https://arxiv.org/html/2610.12794#S3.T4)reaches Human/SOTA in at least one Sonnet\-5 trial, so Best cannot rank them\. The traces below are therefore a budget\-use comparison, not a final\-cell comparison\. Figure[6](https://arxiv.org/html/2610.12794#A9.F6)shows one ceiling packing per method\. Figure[7](https://arxiv.org/html/2610.12794#A9.F7)overlays the Best\-cell best\-so\-far of each method, cropped near the ceiling, under the same100100\-iteration window\.

Figure 6:Circle Packing \(Square,N=26N\{=\}26\) on Sonnet\-5\. Circles in the unit square, labeled00–2525\. The seed is a two\-ring heuristic \(0\.960\.96\); OpenEvolve, GEPA, AdaEvolve, and CoCoEvolve are size\-biased ceiling packings at Human/SOTA \(2\.635982\.63598\)\.Figure 7:Circle Packing \(Square,N=26N\{=\}26\) best\-so\-far on Sonnet\-5, cropped near the ceiling\. Solid traces are each method’s Best\-cell run; Human/SOTA is the dashed green line\. GEPA first reaches2\.635982\.63598at iteration88, CoCoEvolve at iteration2424, AdaEvolve at iteration9191\. OpenEvolve is at2\.62982\.6298at iteration100100and only hits the ceiling at iteration111111, outside the shared window\.
##### Search\.

This is a saturation control, not a CoCoEvolve win\. Once every method can hit the ceiling, the remaining signal is how the shared budget is spent getting there\. On the traces in Figure[7](https://arxiv.org/html/2610.12794#A9.F7), GEPA is first \(iteration88\), CoCoEvolve is second \(iteration2424\), AdaEvolve reaches the ceiling at iteration9191, and OpenEvolve is still at2\.62982\.6298at iteration100100\(its Best of2\.635982\.63598is from iteration111111, outside the shared window\)\. We do not read first\-to\-ceiling as a method ranking: GEPA winning that race on a saturated packing instance is compatible with Heilbronn, where the same protocol separates the methods on Avg rather than Best\.

#### I\.3\.3Cloudcast

##### Setup\.

The objective is the simulator’s true union\-of\-edges cost of a multi\-cloud broadcast tree \(lower is better\); Human/SOTA is626\.2626\.2\. The seed enumerates Azure single\- and two\-hub plans ranked by a node\-count/bottleneck proxy, with an arborescence fallback, and scores974974\. The analysis below is the CoCoEvolve Best\-cell Sonnet\-5 run under the same100100\-iteration protocol\. OpenEvolve remains far from the others; GEPA, AdaEvolve, and CoCoEvolve all finish in a tight618618cluster \(Table[4](https://arxiv.org/html/2610.12794#S3.T4)\), so this subsection is about the construction that reached that cluster, not about ranking inside it\.

##### Search\.

The champion no longer searches only for hubs\. It evaluates six structurally distinct tree families: terminal MSA, relay hub, capacitated min\-cost flow, Dreyfus–Wagner DP, resistance ranking, and TSP relay\. A proxy replays the simulator’s cost model and selects the cheapest candidate\.

The first two iterations reduce cost from974974to636636by reading the cost graph and instance configuration and by testing scratch candidates against the live evaluator\. Later cross\-lineage grafts add Dreyfus–Wagner ranking and a TSP\-relay construction, which produce the remaining improvement\. The trace documents a substantial rewrite of the seed’s search procedure\. Because GEPA, AdaEvolve, and CoCoEvolve finish close together, Cloudcast gives little evidence for ranking their final scores\. TXN provides a clearer example of a large score change after a structural rewrite\.

#### I\.3\.4TXN scheduling

##### Setup\.

The objective is a combined throughput\-style score of a lock\-aware transaction schedule \(higher is better\); Human/SOTA is2,7252\{,\}725\. The seed is tabu search over swap and or\-opt\-1 from Copeland / MDS / assignment\-problem / greedy starts, and scores2,7252\{,\}725\. The reported runs show substantial headroom above that seed and reference value\. Both the mean and best scores can therefore distinguish methods here\. The analysis below is the CoCoEvolve Best\-cell Sonnet\-5 run that produced4,3484\{,\}348, under the same100100\-iteration protocol\.

##### Search\.

The champion replaces the seed’s tabu loop with a three\-stage pipeline\. An assignment\-problem relaxation and a single\-path greedy construction from the top Copeland candidate each build a starting schedule, and the cheaper one is kept; simulated annealing then applies double\-bridge reheating, and first\-improvement local search performs the final cleanup\.

Copeland ranking and an EDA over pairwise order probabilities first move the score above the seed\. The EDA also queries the ranked archive instead of editing only the parent\. The final program uses assignment/greedy construction followed by simulated annealing, not a retuned tabu tenure\.

TXN is therefore the far\-from\-ceiling counterpart to Heilbronn\. In Heilbronn, the ledger improves reliability by rescuing stalled trials\. In TXN, the same keep\-or\-switch loop replaces the seed metaheuristic and improves both Avg and Best beyond the other methods\.

## Appendix JPrompt Optimization

### J\.1Detailed Methodology

We evaluate three tasks from GEPA’s benchmark suite\. All three compare an unmodified seed, GEPA, and CoCoEvolve from the same initial program; each CoCoEvolve run uses 100 iterations\.

##### HotpotQA\.

The artifact is a four\-stage retrieve\-summarize\-answer pipeline over a BM25 index of Wikipedia abstracts, with prompts for first\-hop summarization \(summarize1\), second\-hop query generation \(create\_query\_hop2\), second\-hop summarization \(summarize2\), and final answering \(final\_answer\)\. Each mutation changes one prompt while retrieval remains fixed, and GEPA’s round\-robin selector mutates the same four components in the same cycle\. We use GEPA’s official 150/300/300 train, validation, and held\-out\-test split\. The GEPA search budgets were chosen to be broadly comparable to the 100\-iteration CoCoEvolve runs, although the two methods account for evaluations differently\. The frozen Sonnet\-5 backbone serves as both mutator and student\.

GEPA uses the sameBenchmarkbase class for AIME, HotpotQA, and IFBench\. The artifact runner passes individual query instances in separate training and validation sets for all three tasks\. GEPA samples three training examples by default for each reflective step\. A proposed edit must strictly improve its parent’s score on that minibatch\. Each accepted candidate then triggers a full validation\-set evaluation, whose per\-example scores inform the Pareto selector’s choice of future parents\.

##### AIME\.

The evolved artifact is a single chain\-of\-thought system prompt that a frozen student model reads with each competition math problem to produce an integer answer\. GEPA’s fixed\-seed split draws 45 training and 45 validation problems fromAI\-MO/aimo\-validation\-aime; its disjoint test set contains 30 problems fromMathArena/aime\_2025\. AIME usesopenai\-gpt\-5\.6\-lunaas both mutator and student\. An initial Sonnet\-5 run gave the seed a perfect training score, leaving no signal for optimization\.

##### IFBench\.

The artifact has two prompt\-driven stages,generate\_responseandensure\_correct\_response\. The evaluator scores the fraction of each query’s instruction constraints that the response satisfies\. It supports 83 checker types: 25 from IFEval and 58 IFBench\-native types, including word and letter counts, formatting, and keyword use\. Eight leniency variants account for markdown emphasis and optional leading or trailing lines\. Neither stage sees the constraint list\. GEPA’s fixed split uses the first 300 rows of its training file for validation and samples 150 training rows from the next 300 with a fixed seed\. We use its bundled 294\-row test file unchanged; it has six fewer rows than the public IFBench release\. IFBench also usesopenai\-gpt\-5\.6\-lunaas both mutator and student\.

The IFBench run uses the defaultFullEvaluationPolicyand 100 concurrent workers\.

##### Repeated evaluation\.

Withopenai\-gpt\-5\.6\-luna, we evaluated the same evolved AIME prompt twice on the 45 training problems: once during optimization and once afterward\. The second evaluation got five fewer answers correct, even though the prompt had not changed\. The student’s nonzero decoding temperature can make its answers vary between evaluations\. Each optimizer ran once and produced one final candidate\. We evaluated that unchanged candidate three times on each AIME and IFBench split and report the mean and standard deviation in Table[5](https://arxiv.org/html/2610.12794#S3.T5)\. These repeats measure evaluation variability, not search variability\. On AIME test, five baseline questions, eight GEPA questions, and six CoCoEvolve questions changed correctness across repeats, even though the baseline’s aggregate test score was25/3025/30every time\.

### J\.2An efficiency variant: sim\-graph targeting and cascaded evaluation on HotpotQA

The HotpotQA run of Appendix[J\.1](https://arxiv.org/html/2610.12794#A10.SS1)scores its*entire*150150\-question training pool and300300\-question validation pool on every one of its100100iterations, unconditionally –450450real model\-backed questions per iteration regardless of how informative any single one is\. DABStep’s own evaluator \(Appendix[C\.2](https://arxiv.org/html/2610.12794#A3.SS2), Appendix[C\.3](https://arxiv.org/html/2610.12794#A3.SS3)\) instead scores only a bounded, informativeness\-targeted subset per iteration and extrapolates the rest from the parent’s own history\. We test a HotpotQA variant that combines similarity\-based target selection, P0/P1/P2 cascaded evaluation, and mutator self\-testing\. The comparison measures this combined configuration\. It cannot attribute any score or time difference to one of the three changes alone\.

##### Setup\.

We keep the seed, 100\-iteration budget, and backbone from Appendix[J\.1](https://arxiv.org/html/2610.12794#A10.SS1)\. We add three mechanisms\.

*Target selection\.*A similarity graph and InfoGain selector operate over the 150\-question training pool \(Appendix[C\.3](https://arxiv.org/html/2610.12794#A3.SS3)\)\. Each mutation receives one selected focus question instead of the full undifferentiated pool\.

*Cascaded evaluation\.*The P0/P1/P2 cascade mirrors the DABStep evaluator \(Appendix[F\.1](https://arxiv.org/html/2610.12794#A6.SS1)\)\. P0 evaluates only the selected target and requires a score of0\.60\.6\. If P0 passes, P1 evaluates a stratified sample of 20 regression questions, or about13%13\\%of the pool\. One third of this sample is selected by similarity\-weighted risk to the target, and two thirds are sampled uniformly from the parent’s tested history\. P1 requires at least0\.9×0\.9\\timesthe parent’s score on the same sample\. If P1 passes, P2 explores similarity\-ranked frontier tiers of 10 questions each\. One random scout represents each tier, and the next tier opens only if the current tier scores at least6/106/10\. The training cascade evaluates at most⌊150/3⌋=50\\lfloor 150/3\\rfloor\{=\}50questions per iteration, excluding the one\-time seed bootstrap, mutator self\-tests, and separately sampled validation passes\. When the parent is already at ceiling on the target, a live pretest can bypass P0 and P1 and proceed directly to P2\.

The child’s score combines its updated per\-question record with the parent’s inherited record and divides the total by 150\. We do not use DABStep’s additive formula because that formula assumes sparse initial coverage\. HotpotQA has dense coverage from the seed’s full\-pool bootstrap, so the additive formula would double\-count observations\.

*Mutator self\-test\.*The mutator receivestest\_draft, with a budget of three calls per turn\. It can score an in\-progress edit directly on the selected focus question\. The mutator prompt also instructs the model to verify its draft before finishing\. Both features reproduce DABStep’s self\-verification affordance \(Appendix[C\.2](https://arxiv.org/html/2610.12794#A3.SS2)\)\.

Validation is sampled independently with probability0\.10\.1per iteration\. When validation is skipped, the child inherits the parent’s validation score and per\-question record unchanged\.

##### Results\.

Table 20:HotpotQA, Sonnet\-5, CoCoEvolve: the published full\-pool\-every\-iteration search versus the sim\-graph\-targeted/cascaded\-evaluation variant of this appendix\. Both rows are a single100100\-iteration search, freshly re\-scored against the identical train/val/test pools\.Across 100 iterations, the cascaded evaluator has a median wall\-clock time of121121s per iteration\. A full 450\-question pass at the same concurrency takes roughly900900–11001100s\. The seed’s one\-time full\-pool bootstrap takes11151115s, which is consistent with that estimate\.

In the first 50 logged iterations, seven \(14%14\\%\) took400400–11001100s; one was the seed bootstrap, and the others included a sampled pass over all 300 validation questions\. Extrapolating that mix to 100 iterations gives an estimated5\.55\.5–66hours of evaluator time, versus an estimated2525–3030hours for full\-pool evaluation at every iteration\. The implied reduction is about44–5×5\\times\. These are estimates from the observed timing mix, not measured total run times\.

##### Analysis\.

The combined variant matches the full\-pool run’s observed train accuracy \(0\.8000\.800\) and has0\.0070\.007higher validation accuracy, but its test accuracy is0\.0260\.026lower, about eight questions out of 300\. The timing estimate suggests lower evaluator cost\. With one search and one final re\-score per variant, the result does not establish equal held\-out quality or isolate which added mechanism produced the difference\. It also does not replace or improve the main HotpotQA result in Table[5](https://arxiv.org/html/2610.12794#S3.T5)\.

## Appendix KComplete Base and Final Programs

The following unabridged listings pair the seed and final program from one run each of Heilbronn Convex, Cloudcast, and TXN\. The Cloudcast and TXN listings contain only the evolved function; module\-level imports \(e\.g\.,networkx as nx,random\) and harness classes such asBroadCastTopologyare supplied by the surrounding benchmark file\.

### K\.1Heilbronn Convex

Heilbronn Convex: base program

Heilbronn Convex: final evolved program

### K\.2Cloudcast

Cloudcast: base program

Cloudcast: final evolved program

### K\.3TXN

TXN: base program

TXN: final evolved program