HERO：利用证据进行程序生成中的测试时细化与提示优化

# HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation
Source: [https://arxiv.org/html/2610.19184](https://arxiv.org/html/2610.19184)
Mingyu Lu††thanks:Work done during an internship at Microsoft Research\.Affiliation:University of WashingtonAffiliation:Microsoft ResearchEmail:[mingyulu@cs\.washington\.edu](mailto:)Tristan NaumannAffiliation:Microsoft ResearchHoifung PoonAffiliation:RecursionPeniel ArgawAffiliation:Microsoft Research

###### Abstract

Harness optimization improves LLMs by refining reusable prompts and scaffolding, yet requires further adaptation for program generation tasks such as text\-to\-SQL and text\-to\-Python\. Existing methods often rely on LLM reflectors to propose harness updates, which can fix observed failures but also disrupt previously correct behavior\. Text\-to\-code tasks offer a natural way to mitigate this issue: we can execute generated outputs, allowing us to evaluate and verify the effects of proposed updates directly\. Building on this signal, we introduceHERO\(HarnessingEvidence forRefinement andOptimization\), which combines offline and test\-time harness optimization\. At the first stage, HERO learns reusable prompt updates from execution\-verified successes and failures and validates candidate revisions before adoption\. At test time, we treat the generated code as an instance\-specific harness and refine it using execution feedback\. Across four SQL, Cypher, and Python benchmarks, HERO achieves the best offline performance, improving pass@1 by7\.67\.6points over the initial prompt, while test\-time refinement adds another2\.02\.0points and improves execution efficiency\.

## 1Introduction

LLM agents are increasingly employed to interact with external environments through tools, often by generating executable programs such as SQL and Cypher queries and Python functions\([Li et al\., 2023](https://arxiv.org/html/2610.19184#bib.bib1);[Schick et al\., 2023](https://arxiv.org/html/2610.19184#bib.bib7);[Feng et al\., 2025](https://arxiv.org/html/2610.19184#bib.bib4);[Zhuo et al\., 2025](https://arxiv.org/html/2610.19184#bib.bib2);[Ouyang et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib3)\)\. These artifacts provide a flexible interface for models to perform computation and interact with external systems\([Gao et al\., 2023](https://arxiv.org/html/2610.19184#bib.bib9);[Wang et al\., 2024](https://arxiv.org/html/2610.19184#bib.bib8)\)\. Their quality depends not only on the underlying model, but also on the surrounding*harness*—the prompts, tools, and memory that shape generation\([Khattab et al\., 2024](https://arxiv.org/html/2610.19184#bib.bib26);[Lee et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib19);[Li et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib24);[Lopopolo, 2026](https://arxiv.org/html/2610.19184#bib.bib6)\)\. This makes the harness a natural target for improving LLM agent performance\. Accordingly, existing work has pursued two related directions: optimizing reusable harness components across tasks\([Lee et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib19);[Lin et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib15);[Yang et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib22);[Agrawal et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib17)\), and using test\-time feedback to select or refine generated outputs\([Ni et al\., 2023](https://arxiv.org/html/2610.19184#bib.bib28);[Chen et al\., 2024](https://arxiv.org/html/2610.19184#bib.bib27);[Madaan et al\., 2023](https://arxiv.org/html/2610.19184#bib.bib10)\)\.

Improving program generation through these directions presents two linked challenges\. First, reusable updates must correct recurring errors without degrading behavior that already works\. LLM\-generated feedback can be unreliable without external grounding\([Huang et al\., 2024](https://arxiv.org/html/2610.19184#bib.bib25);[Kamoi et al\., 2024](https://arxiv.org/html/2610.19184#bib.bib23)\), and prompt revisions that resolve failures can introduce regressions on previously successful cases\([Wu et al\., 2024b](https://arxiv.org/html/2610.19184#bib.bib29)\)\. The challenge is therefore to derive grounded diagnoses of what to change and preserve from execution outcomes, and translate these diagnoses into reusable guidance\. Second, this general guidance must be complemented by feedback specific to the current instance\. An optimized prompt captures strategies learned across training tasks, but a generated program may still encounter errors that require observations from its own execution\. Approaches such as Dynamic Cheatsheet \(DC\) and ACE accumulate experience through evolving shared contexts\([Suzgun et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib20);[Zhang et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib18)\)\. When shared context is updated at test time, unreliable or overly specific feedback can become persistent guidance, allowing errors from one instance to affect subsequent tasks\. Together, these challenges motivate grounding offline harness updates in execution evidence while keeping test\-time corrections local to the current program\.

To address these challenges, we introduceHERO\(HarnessingEvidence forRefinement andOptimization\), a two\-level framework that grounds reusable guidance updates in execution evidence and confines test\-time adaptation to the current program\. Our contributions are threefold:

- •A unified two\-level formulation\.We formulate program generation as a two\-level optimization problem that couples offline prompt optimization with test\-time refinement of instance\-specific programs\.
- •Execution\-grounded feedback and verification\.Offline, we use execution evidence from successful and failed programs under prior prompts to identify what to correct and preserve\. At test time, execution feedback and retrieved verified training examples guide program revisions, which are accepted only when supported by evidence\.
- •Empirical evaluation and analysis\.We demonstrate correctness gains over existing offline harness optimizers across diverse query\-generation and programming tasks, with further improvements from test\-time refinement\. We further evaluate execution efficiency and computational cost, finding that the learned prompts remain compact across tasks\.

![Refer to caption](https://arxiv.org/html/2610.19184v1/concept_figure.png)Figure 1:Overview of HERO\.\(a\)HERO\-O: Offline prompt optimization learns a shared prompt across training tasks\.\(b\)HERO\-T: Test\-time refinement refines the generated artifact using environment feedback\.
## 2Related Works

Prompt and harness optimization\.Prompt and harness optimization improves frozen LLMs by adapting reusable components around them\. MIPRO\([Opsahl\-Ong et al\., 2024](https://arxiv.org/html/2610.19184#bib.bib16)\)jointly optimizes instructions and few\-shot demonstrations using task\-level feedback\. Reflection\-based approaches include GEPA\([Agrawal et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib17)\), which uses trajectory feedback to propose prompt revisions, and STRAGO\([Wu et al\., 2024b](https://arxiv.org/html/2610.19184#bib.bib29)\), which derives refinement strategies from both successful and failed examples to mitigate prompt drift\. Other approaches broaden the optimization target: ACE\([Zhang et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib18)\)maintains an evolving context through generation, reflection, and curation; SkillOpt\([Yang et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib22)\)optimizes reusable textual skills; and Meta\-Harness\([Lee et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib19)\)searches over harness implementations using source code, scores, and execution traces\. Building on this line of work, HERO adopts a similar offline optimization loop but enriches reflection with execution feedback from successful and failed programs generated under prior prompt revisions, aiming to improve program generation while limiting regressions\.

Test\-time refinement\.Test\-time methods refine model outputs using feedback available during inference\. ReAct\([Yao et al\., 2022](https://arxiv.org/html/2610.19184#bib.bib21)\)interleaves reasoning, actions, and environment observations, while Reflexion and related self\-refinement methods use feedback from earlier attempts to revise subsequent outputs\([Shinn et al\., 2023](https://arxiv.org/html/2610.19184#bib.bib11);[Madaan et al\., 2023](https://arxiv.org/html/2610.19184#bib.bib10)\)\. Dynamic Cheatsheet \(DC\)\([Suzgun et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib20)\)further accumulates self\-curated strategies across queries through persistent memory\. These methods emphasize adaptation from test\-time experience\. HERO complements this adaptation with reusable guidance learned through offline prompt optimization\. The optimized prompt remains fixed during inference, while execution feedback and retrieved, verified training examples guide refinement of the current program\.

## 3Harness Optimization for Program Generation

### 3\.1Problem Setup: Program Generation

In this work, we focus on program generation—tasks that map natural language specifications to programs, including text\-to\-SQL, text\-to\-Cypher, and text\-to\-Python function generation\. Each instancexxcontains a specification and available task context, such as a database schema or function signature\. An LLM generates a candidate programyy, whose execution yields observations such as runtime errors, returned values, and execution traces\([Wang et al\., 2024](https://arxiv.org/html/2610.19184#bib.bib8)\)\. We denote its task performance by an execution\-based evaluation rewardR⁡\(y,x\)∈ℝR\(y,x\)\\in\\mathbb\{R\}\. We distinguish this evaluation reward from the feedback available during refinement: in our setting, test\-time refinement can access execution observations but not test reference solutions or hidden correctness labels\.

### 3\.2A Unified View of Harness Optimization

Prior work improves fixed LLMs through reusable prompt or harness optimization\([Agrawal et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib17);[Lee et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib19)\)and feedback\-driven refinement across attempts\([Shinn et al\., 2023](https://arxiv.org/html/2610.19184#bib.bib11)\)\. We represent these forms of adaptation through a harness comprising external state that conditions a fixed policyπθ\\pi\_\{\\theta\}or mediates its interaction with the environment\. For an instancexx, we decompose this state as

hx=\(hshared,hxinst\),h\_\{x\}=\\left\(h^\{\\mathrm\{shared\}\},h\_\{x\}^\{\\mathrm\{inst\}\}\\right\),\(1\)wherehsharedh^\{\\mathrm\{shared\}\}is reused across instances andhxinsth\_\{x\}^\{\\mathrm\{inst\}\}contains instance\-specific state\. During refinement, previously generated programs, their execution feedback, and retrieved examples can become part ofhxinsth\_\{x\}^\{\\mathrm\{inst\}\}, conditioning subsequent revisions\. Given the harness, the policy generates the next candidate program:

y∼πθ\(⋅∣x;hx\)\.y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x;h\_\{x\}\)\.\(2\)Optimizing shared state targets the expected reward across instances, whereas optimizing instance\-specific state targets the current instance\.

## 4Method

Building on this unified view, we propose HERO, a two\-stage framework that optimizes the harness at complementary scopes while keeping the underlying LLM fixed\. The first stage, offline optimization \(HERO\-O\), learns a shared harnesshsharedh^\{\\mathrm\{shared\}\}from execution feedback across training tasks \([Figure1](https://arxiv.org/html/2610.19184#S1.F1)\(a\)\)\. The second stage, test\-time refinement \(HERO\-T\), keeps the shared harness fixed while adapting an instance\-specific harnesshxinsth\_\{x\}^\{\\mathrm\{inst\}\}using evidence from the current test instance \([Figure1](https://arxiv.org/html/2610.19184#S1.F1)\(b\)\)\. The overall procedure is summarized in[Algorithm1](https://arxiv.org/html/2610.19184#alg1)\.

### 4\.1HERO\-O: Offline Prompt Optimization

Algorithm 1HERO1:Initial prompt

h0sharedh\_\{0\}^\{\\mathrm\{shared\}\}; diagnostic set

𝒟diag\\mathcal\{D\}\_\{\\mathrm\{diag\}\}; update set

𝒟upd\\mathcal\{D\}\_\{\\mathrm\{upd\}\}; selection set

𝒟sel\\mathcal\{D\}\_\{\\mathrm\{sel\}\}; optimizer LLM

MoptM\_\{\\mathrm\{opt\}\}; candidates per iteration

NN; iterations

TT; program\-refinement operator

ΦB\\Phi\_\{B\}with per\-query action budget

BB
2:Stage 1: Prompt Optimization

3:

𝒜←\{h0shared\}\\mathcal\{A\}\\leftarrow\\\{h\_\{0\}^\{\\text\{shared\}\}\\\};

ℛ−1←∅\\mathcal\{R\}\_\{\-1\}\\leftarrow\\emptyset
4:for

t=0,…,T−1t=0,\\ldots,T\-1do

5:

Ft←Mopt​\(htshared,ℛt−1,𝒟diag\)F\_\{t\}\\leftarrow\{M\_\{\\mathrm\{opt\}\}\}\(h\_\{t\}^\{\\text\{shared\}\},\\mathcal\{R\}\_\{t\-1\},\\mathcal\{D\}\_\{\\mathrm\{diag\}\}\)⊳\\trianglerightreflect on diagnostic programs and execution outcomes

6:

ℋt←\{Medit\(i\)​\(htshared,Ft\)\}i=1N\\mathcal\{H\}\_\{t\}\\leftarrow\\\{M\_\{\\mathrm\{edit\}\}^\{\(i\)\}\(h\_\{t\}^\{\\text\{shared\}\},F\_\{t\}\)\\\}\_\{i=1\}^\{N\}⊳\\trianglerightpropose candidate prompts

7:

\(ht\+1shared,ℋt\+\)←ScreenAndAccept​\(htshared,ℋt,𝒟upd\)\(h\_\{t\+1\}^\{\\text\{shared\}\},\\mathcal\{H\}\_\{t\}^\{\+\}\)\\leftarrow\\textsc\{ScreenAndAccept\}\(h\_\{t\}^\{\\text\{shared\}\},\\mathcal\{H\}\_\{t\},\\mathcal\{D\}\_\{\\mathrm\{upd\}\}\)⊳\\triangleright[Algorithm2](https://arxiv.org/html/2610.19184#alg2)

8:

ℛt←ℋt∖\{ht\+1shared\}\\mathcal\{R\}\_\{t\}\\leftarrow\\mathcal\{H\}\_\{t\}\\setminus\\\{h\_\{t\+1\}^\{\\text\{shared\}\}\\\}
9:

𝒜←𝒜∪ℋt\+\\mathcal\{A\}\\leftarrow\\mathcal\{A\}\\cup\\mathcal\{H\}\_\{t\}^\{\+\}
10:endfor

11:

h^shared←arg⁡maxh∈𝒜⁡Acc⁡\(h,𝒟sel\)\\hat\{h\}^\{\\mathrm\{shared\}\}\\leftarrow\\arg\\max\_\{h\\in\\mathcal\{A\}\}\\operatorname\{Acc\}\(h,\\mathcal\{D\}\_\{\\mathrm\{sel\}\}\)
12:Stage 2: Test\-Time Refinement

13:for alltest instances

xxdo

14:

y0\(x\)∼πθ\(⋅∣x;h^shared\)y\_\{0\}\(x\)\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x;\\hat\{h\}^\{\\mathrm\{shared\}\}\)
15:

y^​\(x\)←ΦB​\(x,y0​\(x\),h^shared\)\\hat\{y\}\(x\)\\leftarrow\\Phi\_\{B\}\(x,y\_\{0\}\(x\);\\hat\{h\}^\{\\mathrm\{shared\}\}\)
16:endfor

17:return

h^shared\\hat\{h\}^\{\\text\{shared\}\}and refined programs

\{y^​\(x\)\}\\\{\\hat\{y\}\(x\)\\\}

Stage 1 of HERO instantiates thehsharedh^\{\\mathrm\{shared\}\}optimization by learning a reusable prompt across training tasks\. We denotehtsharedh\_\{t\}^\{\\text\{shared\}\}for the accepted prompt at iterationtt\. Starting fromh0sharedh\_\{0\}^\{\\text\{shared\}\}, optimization alternates between diagnosing the current prompt, proposing revisions, and validating them on held\-out examples\. To prevent overfitting, we use three data splits with distinct roles:𝒟diag\\mathcal\{D\}\_\{\\mathrm\{diag\}\}provides evidence for proposing revisions,𝒟upd\\mathcal\{D\}\_\{\\mathrm\{upd\}\}determines whether a revision is accepted, and𝒟sel\\mathcal\{D\}\_\{\\mathrm\{sel\}\}selects the final prompt\. Thus, examples used to propose an update are not used to determine its acceptance\.

Algorithm 2ScreenAndAccept\- line 5 of Algorithm 11:Current prompt

ht†h\_\{t\}^\{\\dagger\}; candidates

ℋt\\mathcal\{H\}\_\{t\}; update set

𝒟upd\\mathcal\{D\}\_\{\\mathrm\{upd\}\}partitioned into chunks

C1,…,CKC\_\{1\},\\dots,C\_\{K\}; budget

bb; margin

τ\\tau; confidence multiplier

zz
2:Pull

C1C\_\{1\}for all

h∈ℋth\\in\\mathcal\{H\}\_\{t\}
3:while

b\>0b\>0andnotResolved\(

ℋt\\mathcal\{H\}\_\{t\}\)do

4:

\(Δ^h,σ^h\)←PairedDelta​\(h,ht\)\(\\hat\{\\Delta\}\_\{h\},\\hat\{\\sigma\}\_\{h\}\)\\leftarrow\\textsc\{PairedDelta\}\(h,h\_\{t\}\)for all

h∈ℋth\\in\\mathcal\{H\}\_\{t\}
5:

ℓ←arg⁡maxh⁡Δ^h\\ell\\leftarrow\\arg\\max\_\{h\}\\hat\{\\Delta\}\_\{h\};

c←arg⁡maxh≠ℓ⁡\(Δ^h\+z​σ^h\)c\\leftarrow\\arg\\max\_\{h\\neq\\ell\}\(\\hat\{\\Delta\}\_\{h\}\+z\\hat\{\\sigma\}\_\{h\}\)
6:

Pull​\(ℓ,Cnℓ\+1\)\\textsc\{Pull\}\(\\ell,C\_\{n\_\{\\ell\}\+1\}\);

b←b−1b\\leftarrow b\-1
7:if

b\>0b\>0and

Δ^c\+z​σ^c\>τ\\hat\{\\Delta\}\_\{c\}\+z\\hat\{\\sigma\}\_\{c\}\>\\tauthen

8:

Pull​\(c,Cnc\+1\)\\textsc\{Pull\}\(c,C\_\{n\_\{c\}\+1\}\);

b←b−1b\\leftarrow b\-1
9:endif

10:endwhile

11:

\(Δ^h,σ^h\)←PairedDelta​\(h,ht\)\(\\hat\{\\Delta\}\_\{h\},\\hat\{\\sigma\}\_\{h\}\)\\leftarrow\\textsc\{PairedDelta\}\(h,h\_\{t\}\)for all

h∈ℋth\\in\\mathcal\{H\}\_\{t\}
12:

ℓ←arg⁡maxh⁡Δ^h\\ell\\leftarrow\\arg\\max\_\{h\}\\hat\{\\Delta\}\_\{h\}
13:

ht\+1←\{ℓ,Δ^ℓ\>τ,ht,otherwiseh\_\{t\+1\}\\leftarrow\\begin\{cases\}\\ell,&\\hat\{\\Delta\}\_\{\\ell\}\>\\tau,\\\\ h\_\{t\},&\\text\{otherwise\}\\end\{cases\}
14:

ℋt\+←\{h∈ℋt:Δ^h\>τ\}\\mathcal\{H\}\_\{t\}^\{\+\}\\leftarrow\\\{h\\in\\mathcal\{H\}\_\{t\}:\\hat\{\\Delta\}\_\{h\}\>\\tau\\\}
15:return

ht\+1,ℋt\+h\_\{t\+1\},\\mathcal\{H\}\_\{t\}^\{\+\}

#### 4\.1\.1Textual Gradients from Execution\-grounded Experience

Prompt optimization can be viewed as a discrete analogue of gradient\-based optimization, where natural language critiques or reflections serve as “textual gradients” for prompt updates\([Pryzant et al\., 2023](https://arxiv.org/html/2610.19184#bib.bib13);[Yuksekgonul et al\., 2024](https://arxiv.org/html/2610.19184#bib.bib12)\)\. Prior work uses LLM reflection to guide prompt updates\([Pryzant et al\., 2023](https://arxiv.org/html/2610.19184#bib.bib13);[Lin et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib15);[Agrawal et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib17)\), but such feedback can be unreliable without external grounding\([Huang et al\., 2024](https://arxiv.org/html/2610.19184#bib.bib25);[Kamoi et al\., 2024](https://arxiv.org/html/2610.19184#bib.bib23)\)\. Following[Wu et al\. \(2024b\)](https://arxiv.org/html/2610.19184#bib.bib29), we use both successful and failed cases but enrich reflection with execution outcomes and traces from programs generated under prior prompt revisions to identify what to correct and preserve\. This connects prior edits to observed program behavior, providing concrete context for diagnosing what to correct and preserve\. At iterationtt, the optimizer LLM reviews the prompt edits proposed att−1t\-1, along withkkgenerated programs from both successful and failed cases and their corresponding execution outcomes \(e\.g\., failure errors or predicted output mismatches\)\. Correct programs indicate behaviors to preserve, while incorrect programs identify behaviors to revise\. The optimizer summarizes this evidence into feedbackFtF\_\{t\}, providing an execution\-grounded direction for updating the currenthtsharedh\_\{t\}^\{\\text\{shared\}\}\.

Conditioned onhtsharedh\_\{t\}^\{\\text\{shared\}\}andFtF\_\{t\}, the same optimizer LLM proposesNNcandidate shared harnesses,

ℋt=\{Mopt\(i\)​\(htshared,Ft\)\}i=1N\.\\mathcal\{H\}\_\{t\}=\\left\\\{M\_\{\\mathrm\{opt\}\}^\{\(i\)\}\(h\_\{t\}^\{\\text\{shared\}\},F\_\{t\}\)\\right\\\}\_\{i=1\}^\{N\}\.\(3\)We instruct the optimizer to propose revisions as reusable rules rather than instance\-specific fixes, and reject candidates that copy instance\-specific content or reference answers using deterministic filters, e\.g\., filtering out prompts that memorize training answers\.

#### 4\.1\.2Adaptive Paired Acceptance

Prior work has explored bandit\-based candidate selection\([Pryzant et al\., 2023](https://arxiv.org/html/2610.19184#bib.bib13)\)and minibatch comparisons of prompt revisions against their parent prompts\([Agrawal et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib17)\)\. Our acceptance procedure combines these strategies to improve the reliability and efficiency of prompt updates: paired comparisons capture both fixes and regressions relative to the current prompt, uncertainty\-guided allocation focuses evaluation on uncertain candidates, and a minimum improvement margin guards against accepting marginal gains \([Algorithm2](https://arxiv.org/html/2610.19184#alg2)\)\. For each candidate prompth∈ℋth\\in\\mathcal\{H\}\_\{t\}, we compute the per\-instance improvement

dh​\(x\)=R⁡\(yh,x\)−R⁡\(yt,x\),d\_\{h\}\(x\)=R\(y\_\{h\},x\)\-R\(y\_\{t\},x\),\(4\)whereyh∼πθ\(⋅∣x;h\)y\_\{h\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x;h\)andyt∼πθ\(⋅∣x;htshared\)y\_\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x;h\_\{t\}^\{\\mathrm\{shared\}\}\)are generated by the candidate and current shared prompts, respectively, on the same instancexx\. For the binary correctness reward used during acceptance,dh​\(x\)∈\{−1,0,\+1\}d\_\{h\}\(x\)\\in\\\{\-1,0,\+1\\\}, corresponding to a regression, unchanged correctness, or a fix\. The mean paired improvement therefore measures net fixes per evaluated instance\. We denote its estimate byΔ^h\\hat\{\\Delta\}\_\{h\}and its estimated standard error byσ^h\\hat\{\\sigma\}\_\{h\}\.

All candidates are initially evaluated on a common subset of𝒟upd\\mathcal\{D\}\_\{\\mathrm\{upd\}\}\. At each allocation step, we identify the leader with the largestΔ^h\\hat\{\\Delta\}\_\{h\}and the alternative with the highest upper confidence bound \(UCB\)\-inspired scoreΔ^h\+z​σ^h\\hat\{\\Delta\}\_\{h\}\+z\\hat\{\\sigma\}\_\{h\}\([Auer et al\., 2002](https://arxiv.org/html/2610.19184#bib.bib30)\), wherezzcontrols the uncertainty bonus\. The leader receives the next evaluation chunk, while the alternative receives additional evaluations only if its score exceeds the acceptance marginτ\\tau\. This continues until the decision is resolved or the budget is exhausted \([Algorithm2](https://arxiv.org/html/2610.19184#alg2)\)\. The candidate with the largest finalΔ^h\\hat\{\\Delta\}\_\{h\}replaces the current prompt only ifΔ^h\>τ\\hat\{\\Delta\}\_\{h\}\>\\tau; otherwise, the current prompt is retained\.

AfterTTiterations, we select the final shared harnessh^shared\\hat\{h\}^\{\\mathrm\{shared\}\}from the archive𝒜\\mathcal\{A\}using the separate selection set𝒟sel\\mathcal\{D\}\_\{\\mathrm\{sel\}\}\. This decouples per\-iteration acceptance from final harness selection and allows an earlier candidate to be recovered if later updates generalize less well\.

### 4\.2HERO\-T: Evidence\-Grounded Test\-Time Refinement

Stage 2 of HERO adapts the instance\-specific harness while keeping the learned shared prompt fixed\. Given a test instancexx, we first generate

y0∼πθ\(⋅∣x;h^shared\)\.y\_\{0\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x;\\hat\{h\}^\{\\mathrm\{shared\}\}\)\.\(5\)We then refine this program for at mostBBsteps using only evidence available at test time, without access to the test solution or correctness label\.

#### 4\.2\.1Two Sources of Refinement Evidence

Refinement draws on two complementary sources\.*Execution evidence*includes execution traces, runtime errors, returned outputs, and environment probes that help identify instance\-specific problems\.*Retrieved demonstration evidence*consists of execution\-verified training examples that provide implementation patterns, such as library\-specific conventions, that execution feedback alone may not reveal\.

The refiner combines environment interaction and execution\-based program refinement\([Yao et al\., 2022](https://arxiv.org/html/2610.19184#bib.bib21);[Chen et al\., 2024](https://arxiv.org/html/2610.19184#bib.bib27)\)with instance\-specific demonstration retrieval\. Unlike MIPROv2, which selects demonstrations during offline optimization\([Opsahl\-Ong et al\., 2024](https://arxiv.org/html/2610.19184#bib.bib16)\), we select demonstrations separately for each test instance\. Specifically, we retrieve up tonnverified training examples using lexical similarity, then let the refiner select at most three relevant demonstrations\.

#### 4\.2\.2Evidence\-gated Refinement

At refinement stepkk, leteke\_\{k\}denote newly gathered execution observations and, when available, retrieved demonstrations\. The instance\-specific harness contains the current program and accumulated evidence:

hx,kinst=\(yk−1,e1:k\)\.h\_\{x,k\}^\{\\mathrm\{inst\}\}=\\left\(y\_\{k\-1\},e\_\{1:k\}\\right\)\.\(6\)Conditioned on this state and the fixed shared prompt, the same LLM used for initial generation proposes a revised program:

y~k∼πθ\(⋅∣x;h^shared,hx,kinst\),k=1,…,B\.\\displaystyle\\tilde\{y\}\_\{k\}\\sim\\pi\_\{\\theta\}\\left\(\\cdot\\mid x;\\hat\{h\}^\{\\mathrm\{shared\}\},h\_\{x,k\}^\{\\mathrm\{inst\}\}\\right\),\\qquad k=1,\\ldots,B\.

\(7\)
We apply an evidence gate rather than accepting every revision\. First, the controller executesy~k\\tilde\{y\}\_\{k\}and rejects it if it fails the available programmatic checks\. For revisions that pass, the same LLM comparesyk−1y\_\{k\-1\}andy~k\\tilde\{y\}\_\{k\}using both the accumulated evidencee1:ke\_\{1:k\}and the revision’s execution results\. It assesses whether the changes address an observed issue or implement a convention supported by the retrieved demonstrations\. A revision is accepted only if it passes the programmatic checks and the LLM judges its changes to be supported by the evidence\.

LetGk∈\{0,1\}G\_\{k\}\\in\\\{0,1\\\}denote this combined acceptance decision\. We update

yk=\{y~k,Gk=1,yk−1,Gk=0\.y\_\{k\}=\\begin\{cases\}\\tilde\{y\}\_\{k\},&G\_\{k\}=1,\\\\ y\_\{k\-1\},&G\_\{k\}=0\.\\end\{cases\}\(8\)Rejected revisions leave the current program unchanged; accepted revisions become the starting point for the next step\. AfterK≤BK\\leq Brefinement steps, we returnyKy\_\{K\}\.

## 5Experimental Setup

### 5\.1Benchmarks

We evaluate on four text\-to\-function generation benchmarks spanning SQL, Cypher, and Python generation\.

BIRD\-SQL\([Li et al\., 2023](https://arxiv.org/html/2610.19184#bib.bib1)\)\.BIRD\-SQL evaluates text\-to\-SQL generation over realistic SQLite databases from diverse application domains\. Given a natural\-language question, database schema, and optional external knowledge, the model generates an executable SQL query\.

CypherBench\([Feng et al\., 2025](https://arxiv.org/html/2610.19184#bib.bib4)\)\.CypherBench evaluates text\-to\-Cypher generation over large\-scale Neo4j property graphs derived from Wikidata, covering query patterns such as multi\-hop retrieval, aggregation, comparison, and temporal reasoning\.

BigCodeBench\-Instruct\([Zhuo et al\., 2025](https://arxiv.org/html/2610.19184#bib.bib2)\)\.BigCodeBench\-Instruct evaluates text\-to\-Python function generation from text instruction across diverse programming and library\-use tasks\.

DSCodeBench\([Ouyang et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib3)\)\.DSCodeBench evaluates text\-to\-Python function generation for practical data\-science tasks spanning ten widely used libraries\.

For each benchmark, we construct a diagnostic set𝒟diag\\mathcal\{D\}\_\{\\mathrm\{diag\}\}to provide examples and execution feedback for prompt optimization, and an update set𝒟upd\\mathcal\{D\}\_\{\\mathrm\{upd\}\}to evaluate candidate revisions\. For final harness selection, we use a separate set𝒟sel\\mathcal\{D\}\_\{\\mathrm\{sel\}\}where dataset size permits; otherwise, we reuse𝒟upd\\mathcal\{D\}\_\{\\mathrm\{upd\}\}\. Split sizes are reported in[Table4](https://arxiv.org/html/2610.19184#A1.T4), with further details on preprocessing, split construction, and evaluation procedures in[AppendixA](https://arxiv.org/html/2610.19184#A1)\.

BaselineOffline Prompt OptimizationTest\-Time AdaptationHarness OptimizationDatasetSeedHERO\-OSkillOptGEPAMIPROv2ACEDC\-CuHERO\-O\+THERO\-O\+ ReActHERO\-O\+ DC\-CuMeta\-HarnessBIRD DEV59\.39 \(0\.32\)63\.55 \(0\.21\)61\.31 \(1\.19\)62\.59 \(0\.59\)60\.87 \(0\.57\)60\.87 \(0\.10\)60\.22 \(0\.79\)64\.81\(0\.21\)63\.12 \(1\.03\)62\.44 \(1\.43\)58\.24 \(2\.18\)BigCodeBench47\.93 \(0\.74\)52\.59 \(0\.31\)51\.81 \(0\.47\)50\.13 \(1\.00\)50\.65 \(0\.59\)51\.96 \(1\.25\)44\.79 \(0\.83\)53\.79\(0\.33\)52\.64 \(0\.24\)51\.70 \(0\.89\)45\.26 \(2\.52\)DSCodeBench44\.74 \(0\.24\)54\.84 \(1\.87\)51\.87 \(1\.50\)52\.14 \(1\.99\)51\.73 \(1\.76\)51\.31 \(0\.43\)42\.81 \(0\.24\)56\.04 \(1\.87\)56\.22\(2\.87\)54\.70 \(0\.87\)45\.99 \(0\.52\)CypherBench74\.00 \(0\.44\)85\.50 \(1\.40\)84\.10 \(1\.30\)78\.73 \(1\.53\)78\.27 \(3\.04\)77\.13 \(1\.74\)77\.13 \(1\.74\)89\.93\(1\.46\)87\.00 \(0\.21\)81\.77 \(2\.98\)77\.03 \(2\.59\)

Table 1:Performance across four code\-generation benchmarks\.Results \(pass@1\) are reported as mean \(standard deviation\) over three runs\. Bold highlights the best mean within each category\.
### 5\.2Models and Baselines

We compare against three classes of methods:\(i\) offline prompt optimization111We do not include[Wu et al\. \(2024b\)](https://arxiv.org/html/2610.19184#bib.bib29)because no publicly available implementation is provided\., including GEPA\([Agrawal et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib17)\), MIPROv2\([Opsahl\-Ong et al\., 2024](https://arxiv.org/html/2610.19184#bib.bib16)\), ACE\([Zhang et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib18)\), and SkillOpt\([Yang et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib22)\);\(ii\) test\-time refinement, including ReAct\([Yao et al\., 2022](https://arxiv.org/html/2610.19184#bib.bib21)\)and Dynamic Cheatsheet \(DC\-Cu\)\([Suzgun et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib20)\); and\(iii\) end\-to\-end harness optimization, represented by Meta\-Harness\([Lee et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib19)\)\. Offline optimizers start from the same seed prompt \([AppendixG](https://arxiv.org/html/2610.19184#A7)\) and use the same optimization data and execution\-based correctness signal\. Test\-time methods use only information available at inference time, without access to reference solutions, reference outputs, or correctness labels\.

For each benchmark, we set a fixed rollout budget based on GEPA’sauto=heavysetting \([Table4](https://arxiv.org/html/2610.19184#A1.T4)\) and apply the same budget to HERO and all baselines, matching rollout counts as closely as their procedures permit\. Unless otherwise stated, all methods use GPT\-5 with medium reasoning effort as the backbone model\. We report results over three seeds\. Additional implementation details for HERO and the baselines are provided in[AppendixC](https://arxiv.org/html/2610.19184#A3)\.

### 5\.3Evaluation Metrics

Execution accuracy\.Our primary metric is pass@1\. We generate one solution for each instance and report the fraction that satisfies the benchmark’s correctness criteria\.

Execution efficiency\.Accuracy alone does not distinguish between correct programs with different execution costs\([Waghjale et al\., 2024](https://arxiv.org/html/2610.19184#bib.bib14)\)\. We therefore measure runtime efficiency using the Valid Efficiency Score \(VES\)\([Li et al\., 2023](https://arxiv.org/html/2610.19184#bib.bib1)\),

VES=1N​∑n=1N𝟏​\(Vn=V^n\)​E⁡\(Yn\)E⁡\(Y^n\),\\mathrm\{VES\}=\\tfrac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\\mathbf\{1\}\(V\_\{n\}=\\hat\{V\}\_\{n\}\)\\sqrt\{\\tfrac\{E\(Y\_\{n\}\)\}\{E\(\\hat\{Y\}\_\{n\}\)\}\},\(9\)whereVnV\_\{n\}andV^n\\hat\{V\}\_\{n\}denote the reference and predicted execution results, andE⁡\(Yn\)E\(Y\_\{n\}\)andE⁡\(Y^n\)E\(\\hat\{Y\}\_\{n\}\)their corresponding execution times\. Incorrect solutions receive zero credit, while faster correct solutions receive higher scores\. Detailed timing and scoring protocols are provided in[SectionC\.3](https://arxiv.org/html/2610.19184#A3.SS3)\.

Cumulative fix\-to\-break ratio\.To characterize how reliably performance improves during offline optimization, we track whether accepted prompt updates fix previously incorrect instances or break previously correct ones\. At optimization progresstt, we define

ρ⁡\(t\)=∑s=0tfixeds∑s=0tbrokens,\\rho\(t\)=\\tfrac\{\\sum\_\{s=0\}^\{t\}\\mathrm\{fixed\}\_\{s\}\}\{\\sum\_\{s=0\}^\{t\}\\mathrm\{broken\}\_\{s\}\},\(10\)wherefixeds\\mathrm\{fixed\}\_\{s\}andbrokens\\mathrm\{broken\}\_\{s\}count incorrect\-to\-correct and correct\-to\-incorrect transitions induced by accepted updates at iterationss\. A ratioρ⁡\(t\)\>1\\rho\(t\)\>1indicates that accepted updates cumulatively correct more instances than they disrupt\.

## 6Results

### 6\.1Fixes and Regressions During Prompt Optimization

We first examine how offline prompt optimization affects program generation\. While validation accuracy captures the net effect of an update, it does not reveal the balance between fixing incorrect instances and introducing regressions on previously correct ones\. We therefore analyze the cumulative fix\-to\-break ratio \([Equation10](https://arxiv.org/html/2610.19184#S5.E10)\) to characterize this balance over optimization\.

As shown in[Figure2](https://arxiv.org/html/2610.19184#S6.F2), HERO\-O maintains more fixes than regressions throughout optimization and achieves the highest ratios overall\. On BIRD\-SQL, HERO\-O reaches a cumulative ratio of approximately3\.03\.0, compared with2\.02\.0for MIPROv2 and SkillOpt,1\.51\.5for GEPA, and1\.21\.2for ACE\. ACE, which incrementally accumulates instance\-derived strategy bullets, initially favors fixes but gradually approaches a ratio of1\.01\.0, whereas methods that make broader prompt revisions maintain higher ratios\. A similar pattern holds on BigCodeBench, where HERO\-O reaches approximately2\.22\.2, comparable to GEPA and higher than MIPROv2 \(1\.41\.4\) and ACE \(1\.21\.2\)\. Our ablation further shows that using only failed cases reduces the average fix\-to\-break ratio from2\.232\.23to1\.911\.91\([Table7](https://arxiv.org/html/2610.19184#A4.T7)\), suggesting that combining successful and failed cases provides more informative feedback for prompt revisions than failures alone\. These results suggest that our feedback design helps accumulate fixes while limiting regressions during prompt optimization\.

Figure 2:Cumulative fix\-to\-break ratio\.The y\-axis reports the cumulative ratio across optimization iterations on the x\-axis\. The dashed line indicates an equal number of fixes and regressions\.
### 6\.2Benchmark Performance of Offline Prompt Optimization

We next evaluate whether the resulting prompts improve pass@1 on held\-out text\-to\-SQL, text\-to\-Cypher, and text\-to\-Python code\-generation tasks\.

As shown in[Table1](https://arxiv.org/html/2610.19184#S5.T1), HERO\-O improves pass@1 over the initial prompt across all four benchmarks, with gains of4\.164\.16points on BIRD\-SQL,4\.664\.66on BigCodeBench,10\.110\.1on DSCodeBench, and11\.511\.5on CypherBench\. HERO\-O also achieves the highest mean pass@1 among offline prompt optimizers on all four benchmarks\. It reaches63\.563\.5on BIRD\-SQL, compared with62\.562\.5for GEPA and61\.361\.3for SkillOpt;52\.552\.5on BigCodeBench, compared with51\.951\.9for ACE and51\.851\.8for SkillOpt;54\.854\.8on DSCodeBench, compared with52\.152\.1for GEPA; and85\.585\.5on CypherBench, compared with84\.184\.1for SkillOpt\. By contrast, Meta\-Harness, which optimizes the end\-to\-end harness, does not consistently improve performance and can underperform the initial prompt, achieving58\.258\.2on BIRD\-SQL and45\.345\.3on BigCodeBench\.

Figure 3:BIRD\-SQL DEV pass@1 by question difficulty \(simplen=923n\{=\}923, moderaten=463n\{=\}463, challengingn=144n\{=\}144scored instances\)\. Bars are the mean over three seeds; error bars are the sample standard deviation\.We further examine performance across pre\-specified BIRD\-SQL difficulty levels \([Figure3](https://arxiv.org/html/2610.19184#S6.F3)\)\. HERO\-O achieves pass@1 scores of70\.570\.5,53\.653\.6, and51\.451\.4on simple, moderate, and challenging questions, respectively\. Its advantage is largest on challenging questions, where HERO\-O reaches51\.451\.4, outperforming the next\-best method, GEPA \(46\.846\.8\), by4\.64\.6points\. These results show that the gains from grounded prompt optimization persist across difficulty levels and are most pronounced on challenging instances\.

##### Learned prompts and optimization cost\.

MethodTokens/call \(K\)Prompt length \(tokens\)ACE31\.89 \(1\.17\)35,881 \(7,563\)MIPROv23\.47 \(0\.02\)2,704 \(70\)SkillOpt2\.87 \(0\.10\)2,021 \(628\)GEPA2\.47\(0\.24\)2,003 \(541\)HERO\-O \(Ours\)2\.79 \(0\.18\)1,949\(389\)

Table 2:Optimization context and token cost\.Input plus output tokens per LLM call during optimization and the length of the final prompt\. Averaged over four benchmarks per seed; mean \(SD\) over three seeds\.We next examine whether these performance gains come at the cost of larger prompts or greater per\-call token usage \([Table2](https://arxiv.org/html/2610.19184#S6.T2)\)\. HERO\-O, GEPA, and SkillOpt produce compact instruction sets, whereas MIPROv2 additionally selects few\-shot demonstrations and ACE accumulates a substantially larger playbook\. Averaged across benchmarks, HERO\-O produces the shortest final prompts \(1,949 tokens\), compared with 2,003 for GEPA, 2,704 for MIPROv2, and 35,881 for ACE\. Our cost analysis also shows that more compact prompts generally correspond to lower per\-call token usage: HERO\-O averages 2\.79K tokens per optimization call, comparable to GEPA \(2\.47K\) and SkillOpt \(2\.87K\), and substantially below ACE \(31\.89K\)\.

Upon inspection of the learned prompt, we found that HERO\-O primarily retains general task\-level instructions \([Figure9](https://arxiv.org/html/2610.19184#A7.F9)\), whereas other optimizers preserve varying amounts of training\-specific content \([Figures11](https://arxiv.org/html/2610.19184#A7.F11)and[10](https://arxiv.org/html/2610.19184#A7.F10)\)\. Together, these results suggest that HERO learns compact, reusable guidance rather than accumulating instance\-specific details, while achieving its performance gains without substantially larger prompts \([Figure5](https://arxiv.org/html/2610.19184#A4.F5)\)\.

### 6\.3Test\-Time Refinement Complements Offline Prompt Optimization

Offline prompt optimization captures reusable task\-level guidance but may leave instance\-specific errors unresolved\. We therefore evaluate whether HERO\-T—evidence\-grounded test\-time refinement—provides additional gains and how it compares with alternative test\-time adaptation strategies\. For each dataset, we initialize test\-time refinement with the best offline prompt obtained from HERO\-O\.

As shown in[Table1](https://arxiv.org/html/2610.19184#S5.T1), HERO\-T improves pass@1 on all benchmarks , by1\.261\.26points on BIRD\-SQL,1\.201\.20on BigCodeBench,1\.201\.20on DSCodeBench, and4\.434\.43on CypherBench, for an average gain of2\.022\.02points\. The resulting system achieves the highest mean pass@1 on three of four benchmarks; on DSCodeBench, ReAct is marginally higher \(56\.2256\.22vs\.56\.0456\.04\)\. Alternative test\-time methods are less consistent: ReAct improves performance on three benchmarks but slightly degrades BIRD\-SQL, while DC\-Cu decreases performance on all four when applied after offline optimization\. To distinguish whether this behavior arises from combining DC\-Cu with offline optimization, we also apply DC\-Cu directly to the initial prompt\. The results remain mixed: it improves BIRD\-SQL and CypherBench by0\.830\.83and3\.133\.13points, respectively, but decreases BigCodeBench and DSCodeBench by3\.143\.14and1\.931\.93points\.

These results suggest that the two stages of HERO are complementary: offline optimization captures reusable task\-level guidance, while test\-time refinement provides additional improvements for individual instances\.

### 6\.4Execution Efficiency

MethodBIRDBigCodeBenchDSCodeBenchCypherBenchSeed prompt79\.1 \(5\.0\)61\.2 \(3\.6\)48\.7 \(0\.5\)85\.6 \(1\.5\)Stage Offline Harness OptimizationGEPA73\.1 \(3\.6\)69\.1 \(3\.9\)56\.0 \(2\.2\)86\.7 \(3\.8\)MIPROv277\.9 \(7\.1\)69\.7 \(2\.7\)55\.4 \(1\.3\)86\.5 \(3\.2\)ACE72\.3 \(2\.2\)71\.3 \(2\.5\)55\.3 \(0\.5\)103\.8 \(15\.2\)SkillOpt75\.9 \(4\.0\)69\.7 \(1\.0\)54\.8 \(1\.0\)137\.7 \(50\.8\)Metaharness74\.5 \(10\.3\)58\.5 \(3\.0\)84\.8 \(4\.3\)50\.5 \(0\.7\)HERO\-O91\.9 \(6\.4\)75\.0 \(3\.5\)59\.8 \(1\.7\)109\.8 \(39\.8\)Stage Test\-Time AdaptionDC\-Cu83\.7 \(4\.5\)54\.5 \(7\.2\)48\.8 \(0\.8\)74\.2 \(1\.9\)HERO\-O \+ DC\-Cu91\.4 \(4\.9\)70\.6 \(1\.9\)60\.0 \(1\.2\)92\.6 \(16\.0\)HERO\-O \+ ReAct92\.3 \(7\.6\)76\.2 \(3\.6\)60\.7 \(2\.3\)110\.1 \(42\.8\)HERO\-O \+ T93\.4 \(6\.1\)76\.2 \(3\.6\)61\.5 \(1\.4\)101\.7 \(7\.8\)

Table 3:Valid Efficiency Score \(VES\) of generated programs\.Results are mean \(SD\) over three seeds; higher is better\. Bold denotes the best mean\.For SQL queries and Python code, correctness alone does not fully capture solution quality: programs that produce the same correct result can differ substantially in execution cost\([Li et al\., 2023](https://arxiv.org/html/2610.19184#bib.bib1)\)\. We therefore report VES alongside pass@1 to assess whether improvements in correctness also preserve execution efficiency\.

As shown in[Table3](https://arxiv.org/html/2610.19184#S6.T3), HERO\-O achieves the highest mean VES on three of four benchmarks: BIRD\-SQL \(91\.991\.9\), BigCodeBench \(75\.075\.0\), and DSCodeBench \(57\.857\.8\)\. Together with the corresponding pass@1 gains, these results indicate that the improved correctness on these tasks is not achieved at the expense of execution efficiency\. On CypherBench, SkillOpt attains the highest mean VES \(137\.7137\.7\), though with substantial variability across runs \(SD50\.850\.8\)\. With HERO\-T, VES further improves on BIRD\-SQL, BigCodeBench, and DSCodeBench, alongside the corresponding gains in pass@1\. On CypherBench, however, pass@1 increases from85\.585\.5to89\.9389\.93, while VES decreases from109\.8109\.8to101\.7101\.7\. This divergence highlights a correctness–efficiency trade\-off: test\-time refinement solves more instances, but the resulting programs are less efficient under VES\. Overall, HERO delivers concurrent gains in correctness and execution efficiency on three of four benchmarks, demonstrating that its performance improvements generally extend beyond accuracy alone\.

### 6\.5Ablation Study and Sensitivity Analysis

We ablate the main components of offline optimization on BIRD\-SQL and BigCodeBench and examine sensitivity to the amount of diagnostic evidence and optimization budget\.

Figure 4:Sensitivity to the number of rendered diagnostic cases\.Pass@1 as the number of cases rendered varies\. Results are the mean±\\pmSD over three seeds\.##### Ablation study\.

Removing the rendered success and failure cases consistently reduces performance: pass@1 drops from63\.5563\.55to59\.1559\.15on BIRD\-SQL and from52\.552\.5to50\.0850\.08on BigCodeBench \([Figure4](https://arxiv.org/html/2610.19184#S6.F4)\)\. This indicates that validation signals alone do not provide sufficient guidance for prompt revision; concrete execution examples help the optimizer identify what to correct and what to preserve\.

##### Sensitivity analysis and model generalization\.

HERO remains relatively stable across a range of configurations, including the number of candidates, optimization rounds, and rendered diagnostic examples, with performance consistently above the seed prompt\. Similarly, increasing the test\-time refinement budget yields limited additional gains beyond the defaultB=3B=3\([SectionD\.2](https://arxiv.org/html/2610.19184#A4.SS2)\)\. These results suggest that the observed improvements are not highly sensitive to hyperparameter choices or prolonged refinement\. Experiments with GPT\-5\.4 \([Table9](https://arxiv.org/html/2610.19184#A4.T9)\) further show consistent gains over the evaluated baselines, indicating that the approach extends beyond the GPT\-5 backbone\.

## 7Conclusion

In this work, we introduced HERO, a two\-stage harness framework for program generation that combines grounded offline prompt optimization with evidence\-grounded test\-time refinement\. Across four SQL, Cypher, and Python benchmarks, HERO consistently outperforms the initial prompt and existing prompt optimization baselines, with test\-time refinement providing further gains on every benchmark\. Our analyses show that execution\-grounded updates reduce regressions during optimization, while the learned prompts remain compact and robust across optimization settings\. Together, these results support combining reusable cross\-task optimization with instance\-specific refinement for more effective program generation\.

## Limitations

##### Budgeting and computational cost\.

Our evaluation uses fixed, benchmark\-specific budgets, which may limit the gains that some methods could achieve with longer searches\. Substantially expanding these budgets is computationally demanding: even under the evaluated budgets, ACE and Meta\-Harness require approximately 12 and 24 hours of wall\-clock time, respectively, on BigCodeBench in our setup\. To support controlled comparisons within these practical constraints, we use task\-rollout counts as a common measure of evaluation effort and match budgets as closely as method\-specific procedures permit\. This measure, however, does not fully capture differences in the costs of reflection, candidate generation, and selection\. Nevertheless, we have sought to make the comparisons as fair and transparent as possible within these computational and methodological constraints\.

##### Model dependency\.

Our main evaluation relies on proprietary GPT\-5 and GPT\-5\.4 models, whose future updates or deprecation may affect exact harness reproducibility\. However, HERO operates entirely at the harness level and does not require access to or modification of model weights, allowing the same optimization procedure to be applied to other compatible backbones\. Learned prompts may still require re\-optimization when the underlying model changes\.

## References

- Agrawalet al\.\(2026\)L\. A\. Agrawal, S\. Tan, D\. Soylu, N\. Ziems, R\. Khare, K\. Opsahl\-Ong, A\. Singhvi, H\. Shandilya, M\. J\. Ryan, M\. Jiang,et al\.Gepa: reflective prompt evolution can outperform reinforcement learning\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 8479–8565\.Cited by:[§B\.1](https://arxiv.org/html/2610.19184#A2.SS1.SSS0.Px1),[§1](https://arxiv.org/html/2610.19184#S1.p1.1),[§2](https://arxiv.org/html/2610.19184#S2.p1.1),[§3\.2](https://arxiv.org/html/2610.19184#S3.SS2.p1.1),[§4\.1\.1](https://arxiv.org/html/2610.19184#S4.SS1.SSS1.p1.1),[§4\.1\.2](https://arxiv.org/html/2610.19184#S4.SS1.SSS2.p1.1),[§5\.2](https://arxiv.org/html/2610.19184#S5.SS2.p1.1)\.
- Aueret al\.\(2002\)P\. Auer, N\. Cesa\-Bianchi, and P\. FischerFinite\-time analysis of the multiarmed bandit problem\.Machine learning47\(2\),pp\. 235–256\.Cited by:[§4\.1\.2](https://arxiv.org/html/2610.19184#S4.SS1.SSS2.p2.1)\.
- Chenet al\.\(2024\)X\. Chen, M\. Lin, N\. Schärli, and D\. ZhouTeaching large language models to self\-debug\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 8746–8825\.Cited by:[§1](https://arxiv.org/html/2610.19184#S1.p1.1),[§4\.2\.1](https://arxiv.org/html/2610.19184#S4.SS2.SSS1.p2.1)\.
- Fenget al\.\(2025\)Y\. Feng, S\. Papicchio, and S\. RahmanCypherbench: towards precise retrieval over full\-scale modern knowledge graphs in the llm era\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 8934–8958\.Cited by:[Appendix A](https://arxiv.org/html/2610.19184#A1.SS0.SSS0.Px5),[§1](https://arxiv.org/html/2610.19184#S1.p1.1),[§5\.1](https://arxiv.org/html/2610.19184#S5.SS1.p3.1.1)\.
- Gaoet al\.\(2023\)L\. Gao, A\. Madaan, S\. Zhou, U\. Alon, P\. Liu, Y\. Yang, J\. Callan, and G\. NeubigPal: program\-aided language models\.InInternational conference on machine learning,pp\. 10764–10799\.Cited by:[§1](https://arxiv.org/html/2610.19184#S1.p1.1)\.
- Huanget al\.\(2024\)J\. Huang, X\. Chen, S\. Mishra, H\. S\. Zheng, A\. Yu, X\. Song, and D\. ZhouLarge language models cannot self\-correct reasoning yet\.InInternational conference on learning representations,Vol\.2024,pp\. 32808–32824\.Cited by:[§1](https://arxiv.org/html/2610.19184#S1.p2.1),[§4\.1\.1](https://arxiv.org/html/2610.19184#S4.SS1.SSS1.p1.1)\.
- Kamoiet al\.\(2024\)R\. Kamoi, Y\. Zhang, N\. Zhang, J\. Han, and R\. ZhangWhen can llms actually correct their own mistakes? a critical survey of self\-correction of llms\.Transactions of the Association for Computational Linguistics12,pp\. 1417–1440\.Cited by:[§1](https://arxiv.org/html/2610.19184#S1.p2.1),[§4\.1\.1](https://arxiv.org/html/2610.19184#S4.SS1.SSS1.p1.1)\.
- Khattabet al\.\(2024\)O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Haq, A\. Sharma, T\. Joshi, H\. Moazam, H\. Miller,et al\.Dspy: compiling declarative language model calls into state\-of\-the\-art pipelines\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 54928–54958\.Cited by:[§1](https://arxiv.org/html/2610.19184#S1.p1.1)\.
- Leeet al\.\(2026\)Y\. Lee, R\. Nair, Q\. Zhang, K\. Lee, O\. Khattab, and C\. FinnMeta\-harness: end\-to\-end optimization of model harnesses\.arXiv preprint arXiv:2603\.28052\.Cited by:[§B\.3](https://arxiv.org/html/2610.19184#A2.SS3.SSS0.Px1),[§1](https://arxiv.org/html/2610.19184#S1.p1.1),[§2](https://arxiv.org/html/2610.19184#S2.p1.1),[§3\.2](https://arxiv.org/html/2610.19184#S3.SS2.p1.1),[§5\.2](https://arxiv.org/html/2610.19184#S5.SS2.p1.1)\.
- Liet al\.\(2023\)J\. Li, B\. Hui, G\. Qu, J\. Yang, B\. Li, B\. Li, B\. Wang, B\. Qin, R\. Geng, N\. Huo,et al\.Can llm already serve as a database interface? a big bench for large\-scale database grounded text\-to\-sqls\.Advances in Neural Information Processing Systems36,pp\. 42330–42357\.Cited by:[Appendix A](https://arxiv.org/html/2610.19184#A1.SS0.SSS0.Px1),[§C\.3](https://arxiv.org/html/2610.19184#A3.SS3.p1.1),[§C\.3](https://arxiv.org/html/2610.19184#A3.SS3.p2.1),[§1](https://arxiv.org/html/2610.19184#S1.p1.1),[§5\.1](https://arxiv.org/html/2610.19184#S5.SS1.p2.1.1),[§5\.3](https://arxiv.org/html/2610.19184#S5.SS3.p2.1),[§6\.4](https://arxiv.org/html/2610.19184#S6.SS4.p1.1)\.
- Liet al\.\(2026\)J\. Li, X\. Xiao, Y\. Zhang, C\. Liu, L\. Zhao, X\. Liao, Y\. Ji, J\. Wang, J\. Gu, Y\. Ge,et al\.Agent harness engineering: a survey\.OpenReview preprint\.Cited by:[§1](https://arxiv.org/html/2610.19184#S1.p1.1)\.
- Linet al\.\(2026\)J\. Lin, S\. Liu, C\. Pan, L\. Lin, S\. Dou, Z\. Xi, X\. Huang, H\. Yan, Z\. Han, T\. Gui,et al\.Agentic harness engineering: observability\-driven automatic evolution of coding\-agent harnesses\.arXiv preprint arXiv:2604\.25850\.Cited by:[§1](https://arxiv.org/html/2610.19184#S1.p1.1),[§4\.1\.1](https://arxiv.org/html/2610.19184#S4.SS1.SSS1.p1.1)\.
- Lopopolo \(2026\)R\. LopopoloHarness engineering: leveraging codex in an agent\-first world\.URL https://openai\. com/zh\-Hans\-CN/index/harness\-engineering\.Cited by:[§1](https://arxiv.org/html/2610.19184#S1.p1.1)\.
- Madaanet al\.\(2023\)A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang,et al\.Self\-refine: iterative refinement with self\-feedback\.Advances in neural information processing systems36,pp\. 46534–46594\.Cited by:[§1](https://arxiv.org/html/2610.19184#S1.p1.1),[§2](https://arxiv.org/html/2610.19184#S2.p2.1)\.
- Niet al\.\(2023\)A\. Ni, S\. Iyer, D\. Radev, V\. Stoyanov, W\. Yih, S\. Wang, and X\. V\. LinLever: learning to verify language\-to\-code generation with execution\.InInternational Conference on Machine Learning,pp\. 26106–26128\.Cited by:[§1](https://arxiv.org/html/2610.19184#S1.p1.1)\.
- Opsahl\-Onget al\.\(2024\)K\. Opsahl\-Ong, M\. J\. Ryan, J\. Purtell, D\. Broman, C\. Potts, M\. Zaharia, and O\. KhattabOptimizing instructions and demonstrations for multi\-stage language model programs\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 9340–9366\.Cited by:[§B\.1](https://arxiv.org/html/2610.19184#A2.SS1.SSS0.Px2),[§2](https://arxiv.org/html/2610.19184#S2.p1.1),[§4\.2\.1](https://arxiv.org/html/2610.19184#S4.SS2.SSS1.p2.1),[§5\.2](https://arxiv.org/html/2610.19184#S5.SS2.p1.1)\.
- Ouyanget al\.\(2026\)S\. Ouyang, D\. Huang, J\. Guo, Z\. Sun, Q\. Zhu, and J\. M\. ZhangDscodebench: a realistic benchmark for data science code generation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 32628–32636\.Cited by:[Appendix A](https://arxiv.org/html/2610.19184#A1.SS0.SSS0.Px7),[§1](https://arxiv.org/html/2610.19184#S1.p1.1),[§5\.1](https://arxiv.org/html/2610.19184#S5.SS1.p5.1.1)\.
- Pryzantet al\.\(2023\)R\. Pryzant, D\. Iter, J\. Li, Y\. Lee, C\. Zhu, and M\. ZengAutomatic prompt optimization with “gradient descent” and beam search\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 7957–7968\.Cited by:[§4\.1\.1](https://arxiv.org/html/2610.19184#S4.SS1.SSS1.p1.1),[§4\.1\.2](https://arxiv.org/html/2610.19184#S4.SS1.SSS2.p1.1)\.
- Schicket al\.\(2023\)T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. ScialomToolformer: language models can teach themselves to use tools\.Advances in neural information processing systems36,pp\. 68539–68551\.Cited by:[§1](https://arxiv.org/html/2610.19184#S1.p1.1)\.
- Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. YaoReflexion: language agents with verbal reinforcement learning\.Advances in neural information processing systems36,pp\. 8634–8652\.Cited by:[§2](https://arxiv.org/html/2610.19184#S2.p2.1),[§3\.2](https://arxiv.org/html/2610.19184#S3.SS2.p1.1)\.
- Suzgunet al\.\(2026\)M\. Suzgun, M\. Yuksekgonul, F\. Bianchi, D\. Jurafsky, and J\. ZouDynamic cheatsheet: test\-time learning with adaptive memory\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 7080–7106\.Cited by:[§B\.2](https://arxiv.org/html/2610.19184#A2.SS2.SSS0.Px1),[§1](https://arxiv.org/html/2610.19184#S1.p2.1),[§2](https://arxiv.org/html/2610.19184#S2.p2.1),[§5\.2](https://arxiv.org/html/2610.19184#S5.SS2.p1.1)\.
- Waghjaleet al\.\(2024\)S\. Waghjale, V\. Veerendranath, Z\. Wang, and D\. FriedECCO: can we improve model\-generated code efficiency without sacrificing functional correctness?\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 15362–15376\.Cited by:[§5\.3](https://arxiv.org/html/2610.19184#S5.SS3.p2.1)\.
- Wanget al\.\(2024\)X\. Wang, Y\. Chen, L\. Yuan, Y\. Zhang, Y\. Li, H\. Peng, and H\. JiExecutable code actions elicit better llm agents\.arXiv preprint arXiv:2402\.01030\.Cited by:[§1](https://arxiv.org/html/2610.19184#S1.p1.1),[§3\.1](https://arxiv.org/html/2610.19184#S3.SS1.p1.1)\.
- Wuet al\.\(2024a\)C\. Wu, Z\. R\. Tam, C\. Lin, Y\. Chen, and H\. LeeStreambench: towards benchmarking continuous improvement of language agents\.Advances in Neural Information Processing Systems37,pp\. 107039–107063\.Cited by:[Appendix A](https://arxiv.org/html/2610.19184#A1.SS0.SSS0.Px2.p1.1)\.
- Wuet al\.\(2024b\)Y\. Wu, Y\. Gao, B\. B\. Zhu, Z\. Zhou, X\. Sun, S\. Yang, J\. Lou, Z\. Ding, and L\. YangStrago: harnessing strategic guidance for prompt optimization\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 10043–10061\.Cited by:[§1](https://arxiv.org/html/2610.19184#S1.p2.1),[§2](https://arxiv.org/html/2610.19184#S2.p1.1),[§4\.1\.1](https://arxiv.org/html/2610.19184#S4.SS1.SSS1.p1.1),[footnote 1](https://arxiv.org/html/2610.19184#footnote1)\.
- Yanget al\.\(2026\)Y\. Yang, Z\. Gong, W\. Huang, Q\. Yang, Z\. Zhou, Z\. Huang, Y\. Li, X\. Gao, Q\. Dai, B\. Liu,et al\.Skillopt: executive strategy for self\-evolving agent skills\.arXiv preprint arXiv:2605\.23904\.Cited by:[§1](https://arxiv.org/html/2610.19184#S1.p1.1),[§2](https://arxiv.org/html/2610.19184#S2.p1.1),[§5\.2](https://arxiv.org/html/2610.19184#S5.SS2.p1.1)\.
- Yaoet al\.\(2022\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. CaoReact: synergizing reasoning and acting in language models\.arXiv preprint arXiv:2210\.03629\.Cited by:[§B\.2](https://arxiv.org/html/2610.19184#A2.SS2.SSS0.Px2),[§2](https://arxiv.org/html/2610.19184#S2.p2.1),[§4\.2\.1](https://arxiv.org/html/2610.19184#S4.SS2.SSS1.p2.1),[§5\.2](https://arxiv.org/html/2610.19184#S5.SS2.p1.1)\.
- Yuksekgonulet al\.\(2024\)M\. Yuksekgonul, F\. Bianchi, J\. Boen, S\. Liu, Z\. Huang, C\. Guestrin, and J\. ZouTextgrad: automatic" differentiation" via text\.arXiv preprint arXiv:2406\.07496\.Cited by:[§4\.1\.1](https://arxiv.org/html/2610.19184#S4.SS1.SSS1.p1.1)\.
- Zhanget al\.\(2026\)Q\. Zhang, C\. Hu, S\. Upasani, B\. Ma, F\. Hong, V\. Kamanuru, J\. Rainton, C\. Wu, M\. Ji, H\. Li,et al\.Agentic context engineering: evolving contexts for self\-improving language models\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 86069–86100\.Cited by:[§B\.1](https://arxiv.org/html/2610.19184#A2.SS1.SSS0.Px3),[§1](https://arxiv.org/html/2610.19184#S1.p2.1),[§2](https://arxiv.org/html/2610.19184#S2.p1.1),[§5\.2](https://arxiv.org/html/2610.19184#S5.SS2.p1.1)\.
- Zhuoet al\.\(2025\)T\. Y\. Zhuo, M\. C\. Vu, J\. Chim, H\. Hu, W\. Yu, R\. Widyasari, I\. N\. B\. Yusuf, H\. Zhan, J\. He, I\. Paul,et al\.Bigcodebench: benchmarking code generation with diverse function calls and complex instructions\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 66602–66656\.Cited by:[Appendix A](https://arxiv.org/html/2610.19184#A1.SS0.SSS0.Px3),[§1](https://arxiv.org/html/2610.19184#S1.p1.1),[§5\.1](https://arxiv.org/html/2610.19184#S5.SS1.p4.1.1)\.

## Appendix

## Appendix ADatasets Details

DatasetTask𝒟diag\\mathcal\{D\}\_\{\\mathrm\{diag\}\}𝒟upd\\mathcal\{D\}\_\{\\mathrm\{upd\}\}𝒟sel\\mathcal\{D\}\_\{\\mathrm\{sel\}\}𝒟test\\mathcal\{D\}\_\{\\mathrm\{test\}\}Rollout budgetBIRD\-SQLText\-to\-SQL5002501,4001,5305,400CypherBenchText\-to\-Cypher2502502501,0003,700BigCodeBenchPython2502502506375,400DSCodeBenchPython2502502504823,600

Table 4:Benchmarks, data splits, and optimization budgets\.Rollout budgets cap task executions during offline optimization\.Here, we provide additional details on each benchmark and our preprocessing and splitting procedures\. Three conventions apply across all datasets\. First, all splits are constructed once using a fixed dataset\-level seed before any experiment, and pool membership is held fixed across methods and run seeds\. Second, the diagnosis pool𝒟diag\\mathcal\{D\}\_\{\\mathrm\{diag\}\}, update pool𝒟upd\\mathcal\{D\}\_\{\\mathrm\{upd\}\}, and, when applicable, selection pool𝒟sel\\mathcal\{D\}\_\{\\mathrm\{sel\}\}are disjoint at the instance level\. Third, the evaluation split is never accessed during optimization and is used only for final evaluation of the selected prompt\.

##### BIRD\-SQL\([Li et al\., 2023](https://arxiv.org/html/2610.19184#bib.bib1)\)\.

BIRD\-SQL is a large\-scale text\-to\-SQL benchmark designed to evaluate SQL generation in realistic database settings, where models must reason over database contents in addition to schemas\. It contains 12,751 natural\-language–SQL pairs across 95 SQLite databases and 37 application domains: 9,428 training examples over 69 databases, 1,534 development examples over 11 databases, and 1,789 examples over 15 databases in a concealed test set\. Among the training and development databases, 32% originate from Kaggle, 48% from the CTU Prague Relational Learning Repository, and the remaining 20% from other publicly available tables\. Each instance includes a natural\-language question, a reference SQL query, the associated database, and optional expert\-annotated external knowledge\. The development set spans three difficulty levels: 925 simple, 465 moderate, and 144 challenging examples, categorized based on question understanding, external knowledge requirements, database complexity, and SQL complexity\. BIRD also evaluates execution efficiency, enabling assessment of both correctness and runtime behavior\.

##### Preprocessing and splits\.

Following StreamBench\([Wu et al\., 2024a](https://arxiv.org/html/2610.19184#bib.bib5)\), we use the official TRAIN split for optimization and the official DEV split for evaluation, both downloaded from222[https://bird\-bench\.github\.io/](https://bird-bench.github.io/); the concealed test set is not used\. To evaluate cross\-database transfer, we partition TRAIN at the database level\. We reserve 14 of the 69 training databases for final prompt selection, sampling at most 100 questions per database\. Of the remaining 55 databases, 15 are reserved exclusively for the update pool, from which we sample 250 questions, while the 500\-question diagnosis pool is sampled from the remaining 40 databases with every database represented\. Thus, the diagnosis, update, and selection pools are disjoint at the database level\.

Before sampling, we exclude examples whose reference queries fail to execute or return empty results\. For final evaluation, we use the 1,534 DEV examples and exclude four whose reference queries fail under our evaluator, yielding 1,530 scored examples\. Correctness is determined using the official execution\-accuracy comparator: predicted and reference outputs are compared as sets of tuples, ignoring row order and duplicates while preserving column order\.

##### BigCodeBench\-Instruct\([Zhuo et al\., 2025](https://arxiv.org/html/2610.19184#bib.bib2)\)\.

BigCodeBench evaluates Python function generation on practical programming tasks requiring complex instruction following and composition of multiple library APIs\. The benchmark contains 1,140 function\-level tasks spanning seven application domains and 139 libraries, covering 723 distinct function calls\. Tasks are derived from ODEX examples originating from Stack Overflow and are subsequently refactored, tested, and manually curated through a human–LLM collaborative process\. Each task includes an executable reference implementation and unit tests, with an average of 5\.6 tests and 99% branch coverage\. BigCodeBench provides two prompt variants over the same tasks:*BigCodeBench\-Complete*, which uses structured docstrings, and*BigCodeBench\-Instruct*, which converts them into concise natural\-language instructions\. We use the BigCodeBench\-Instruct release from333[https://huggingface\.co/datasets/bigcode/bigcodebench](https://huggingface.co/datasets/bigcode/bigcodebench)\.

##### Preprocessing and splits\.

Because BigCodeBench does not provide an official training split, we first audit all reference solutions in the official Docker environment\. We exclude three tasks whose reference solutions do not pass their own tests: tasks 590 and 1005 depend on live network endpoints, while task 461 exceeds the canonical execution\-time limit\. We then shuffle the remaining 1,137 tasks once using a fixed seed\. The first 500 form the optimization pool, divided equally into 250 diagnosis and 250 update examples; the remaining 637 examples form a frozen evaluation set\. No separate selection pool is used, so final prompt selection is performed on the update pool\.

The model is given the benchmark’sinstruct\_promptas input and generates a single code block\. We evaluate generated programs using the official execution protocol and Docker environment with the benchmark’s hidden unit tests\. We additionally apply the official post\-processing procedure to extract valid function implementations and remove code unrelated to the target entry point\. During optimization, the model receives only pass/fail feedback and exception types; assertion messages containing expected outputs are withheld\. Because the model endpoint does not support explicit temperature control, pass@1 is computed from a single generation using the default decoding configuration\. For stratified sampling, tasks are grouped by their primary library into ten groups\.

##### CypherBench\([Feng et al\., 2025](https://arxiv.org/html/2610.19184#bib.bib4)\)\.

CypherBench evaluates natural\-language\-to\-Cypher generation over large\-scale Neo4j property graphs\. It is designed to assess whether LLMs can query realistic knowledge graphs without relying on simplified schemas or external entity\-linking pipelines\. The benchmark contains 11 property graphs derived from an April 2024 Wikidata snapshot, comprising approximately 7\.8 million entities across domains including art, biology, sports, companies, geography, movies, and politics\. Questions cover 12 graph\-query patterns, including multi\-hop reasoning, aggregation, comparison, temporal reasoning, grouping, and global retrieval, and are constructed using templates followed by LLM\-based rewriting and verification\. We use the released CypherBench dataset from Hugging Face,444[https://huggingface\.co/datasets/megagonlabs/cypherbench](https://huggingface.co/datasets/megagonlabs/cypherbench)which contains 10,882 question–Cypher pairs across 11 property graphs\.

##### Preprocessing and splits\.

Rather than using the original graph\-level train/test partition, we combine all 10,882 released questions and construct fixed question\-level splits stratified by graph and query pattern\. Using proportional sampling, we select 250 examples for the diagnosis pool, 250 for the update pool, and 1,000 for evaluation; the remaining examples are unused\. We do not use a separate selection pool, and final prompt selection is performed on the update pool\. Because the same graphs may appear in both the optimization and evaluation sets, this setting evaluates performance on previously observed graphs rather than generalization to unseen graphs\.

Before constructing the splits, we execute all reference queries against the corresponding Neo4j graphs\. All queries execute successfully within the 120 s timeout and return non\-empty results, so no examples are excluded\. The model is given the graph schema in the officialPropertyGraphSchemaformat together with the natural\-language question and generates a Cypher query\. Generated queries are executed in a read\-only Neo4j session\. Following the benchmark evaluation protocol, predicted and reference outputs are compared as multisets while allowing column permutations\. Row order is considered only when the reference query explicitly specifies an ordering, and queries that result in a Neo4j execution error are marked incorrect\. We use a 60 s query timeout during optimization and a 120 s timeout during final evaluation\. For stratified sampling and analysis, examples are grouped by graph\.

##### DSCodeBench\([Ouyang et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib3)\)\.

DSCodeBench evaluates realistic data\-science code generation across ten Python libraries\. It contains 1,000 function\-generation problems covering NumPy \(131\), Pandas \(92\), SciPy \(112\), Scikit\-learn \(108\), Matplotlib \(105\), Seaborn \(83\), TensorFlow \(110\), PyTorch \(101\), Keras \(104\), and LightGBM \(54\)\. The benchmark is constructed using examples from DS\-1000 and highly rated Stack Overflow answers to retrieve realistic implementations from public GitHub repositories\. Retrieved code is converted into self\-contained functions, filtered and perturbed to reduce direct code reuse and potential data leakage, and paired with natural\-language problem descriptions and execution\-based test cases, followed by manual verification\. Each problem is evaluated using 200 test cases\. We use the dataset released at555[https://github\.com/ShuyinOuyang/DSCodeBench](https://github.com/ShuyinOuyang/DSCodeBench)\.

##### Preprocessing and splits\.

Because DSCodeBench does not provide an official training split, we first validate all reference solutions in the benchmark’s pinned CPU environment\. Each reference solution is executed twice on the official 200 seeded inputs and evaluated using the benchmark’s official comparator\. We exclude 18 problems: seven whose reference solutions are incompatible with our evaluation setting because they fail under the official comparator, require GPU\-only operations, or exceed a 30\-minute timeout, and eleven that produce inconsistent outputs across repeated executions\. We randomly shuffle the remaining 982 problems using a fixed seed\. The first 500 form the optimization set, split evenly into 250 diagnosis and 250 update examples, while the remaining 482 form the evaluation set\. We do not use a separate selection set; final prompt selection is performed on the update set\.

The model is given the benchmark’s problem description as input and generates Python code\. Following the official evaluation harness, we extract the first generated code block and invoke its final top\-level function\. A prediction is considered correct only if its outputs match the reference outputs on all 200 seeded inputs according to the benchmark’s type\-specific comparator, with a 200\-second execution limit per candidate\. Reference outputs are precomputed and reused across candidate solutions\. The test script is not provided to the model, and evaluation messages containing expected outputs are withheld during optimization\. We use the same initial prompt as for BigCodeBench\.

### Software Environment

## Appendix BBaseline Methods

We compare against methods that optimize different components of the harness defined in[Equation1](https://arxiv.org/html/2610.19184#S3.E1)\. Offline prompt optimization methods modify the shared harnesshsharedh^\{\\mathrm\{shared\}\}using optimization data; test\-time methods adapt external state during inference while keeping the model parameters fixed; and Meta\-Harness searches over a broader shared harness that may include both prompts and executable control logic\. Unless otherwise stated, all methods use the same initial prompt, task interface, and execution\-based evaluator as our method\.

### B\.1Offline Prompt Optimization

Offline prompt optimizers modify only the shared harness while leaving instance\-specific state unchanged\. Given optimization data𝒟\\mathcal\{D\}, they can be viewed as searching for

h^shared\\displaystyle\\hat\{h\}^\{\\mathrm\{shared\}\}=argmaxhshared∈ℋ1\|𝒟\|∑x∈𝒟\\displaystyle=\\arg\\max\_\{h^\{\\mathrm\{shared\}\}\\in\\mathcal\{H\}\}\\frac\{1\}\{\|\\mathcal\{D\}\|\}\\sum\_\{x\\in\\mathcal\{D\}\}\(11\)𝔼y∼πθ\(⋅∣x;hshared\)\[R\(y,x\)\]\.\\displaystyle\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x;h^\{\\mathrm\{shared\}\}\)\}\\left\[R\(y,x\)\\right\]\.with different methods defining different representations and update procedures forhsharedh^\{\\mathrm\{shared\}\}\.

##### GEPA\([Agrawal et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib17)\)\.

GEPA is a reflective prompt optimizer that uses execution trajectories and evaluator feedback to iteratively improve the shared system prompt\. An LLM reflects on the behavior of existing prompt candidates, identifies failure modes, and proposes targeted modifications\. Candidate prompts are evaluated on task examples, and a Pareto frontier over per\-instance performance guides subsequent proposal and selection\. In our formulation, GEPA optimizeshsharedh^\{\\mathrm\{shared\}\}with the system prompt as its only mutable component\.

Implementation details\.We usegepapackage \(v0\.1\.1\)111111[https://github\.com/gepa\-ai/gepa](https://github.com/gepa-ai/gepa)with its default configuration, including Pareto\-based selection, reflection minibatches of three, and no prompt merging\. The system prompt is initialized from our common seed prompt\. The diagnosis and update pools serve as GEPA’s training and validation sets, respectively, and reflection records contain the execution verdict, failure category, and evaluator feedback\. We evaluate 21 candidates on BIRD\-SQL and 20 on BigCodeBench; for CypherBench and DSCodeBench, we use metric\-call budgets of 3,700 and 3,600, respectively\.

##### MIPROv2\([Opsahl\-Ong et al\., 2024](https://arxiv.org/html/2610.19184#bib.bib16)\)\.

MIPROv2 optimizes a shared prompt configuration consisting of an instruction and a set of few\-shot demonstrations\. It generates candidate instructions from task and data summaries and uses Bayesian optimization to search over instruction–demonstration combinations\. In our formulation,hsharedh^\{\\mathrm\{shared\}\}comprises the optimized instruction and selected demonstrations\.

Implementation details\.We adapt the referencemipro\_optimizer\_v2implementation from DSPy121212[https://github\.com/stanfordnlp/dspy](https://github.com/stanfordnlp/dspy)to our setting using Optuna v5\.0\.0131313[https://github\.com/optuna/optuna](https://github.com/optuna/optuna)with a TPE sampler\. We retain the three\-stage optimization procedure, using 18 candidate instructions and 18 demonstration sets with up to four demonstrations each\. Demonstrations are bootstrapped using the initial program and supplemented with labeled examples when necessary\. Candidate configurations are evaluated on minibatches of 35 examples, with a full evaluation on the update pool every five trials\. We use metric\-call budgets of 5,400 for BIRD\-SQL and BigCodeBench, 3,700 for CypherBench, and 3,600 for DSCodeBench\. The final shared harness consists of the selected instruction and its four demonstrations\.

##### ACE\([Zhang et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib18)\)\.

Agentic Context Engineering \(ACE\) represents the shared harness as an evolving playbook of reusable strategies\. Its Generator–Reflector–Curator procedure solves training tasks, extracts lessons from execution trajectories and evaluator feedback, and incrementally incorporates these lessons into the playbook\. In our formulation, the learned playbook augments the initial prompt as part ofhsharedh^\{\\mathrm\{shared\}\}\.

Implementation details\.We use the authors’ offline batched implementation,141414[https://github\.com/ace\-agent/ace](https://github.com/ace-agent/ace)retaining the original prompts and seven\-section playbook format\. We use three epochs with up to three Reflector rounds per epoch on BIRD\-SQL, and five epochs with up to five Reflector rounds per epoch on BigCodeBench, CypherBench, and DSCodeBench\. The diagnosis pool is processed in batches of 25, with the Curator constructing a delta context after every 10 reflections and incrementally updating the playbook\. The token budget is treated as a soft constraint\. For BIRD\-SQL, we select among intermediate playbook snapshots using the held\-out selection set; for the remaining benchmarks, we use the final playbook\.

### B\.2Test\-Time Adaptation

Test\-time methods modify external state during inference while keeping the learned shared prompt and model parameters fixed\. For methods that refine the current instance, we write the update generically as

hx,t\+1inst=𝒰⁡\(hx,tinst,x,et\),h\_\{x,t\+1\}^\{\\mathrm\{inst\}\}=\\mathcal\{U\}\\left\(h\_\{x,t\}^\{\\mathrm\{inst\}\},x,e\_\{t\}\\right\),\(12\)whereete\_\{t\}denotes evidence obtained from the current interaction with the execution environment\.

##### Dynamic Cheatsheet \(DC\-Cu\)\([Suzgun et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib20)\)\.

Dynamic Cheatsheet maintains a textual memory that is updated during inference\. We use itscumulativemode, DC\-Cu, in which the model generates a solution conditioned on the current memory and subsequently updates the memory based on the problem and generated output\. Because this memory persists across evaluation examples, it acts as an online\-updated component ofhsharedh^\{\\mathrm\{shared\}\}\.

Implementation details\.We use the authors’ reference implementation,151515[https://github\.com/suzgunmirac/dynamic\-cheatsheet](https://github.com/suzgunmirac/dynamic-cheatsheet)adapting the generator instructions to each task family while retaining the original curator prompt\. For each example, the model may execute its generated program or query up to three times before the memory is updated\. The update uses only the problem and model\-generated outputs; no reference solutions, correctness labels, or evaluator verdicts are provided\. The current memory and task\-specific generator instructions are included in the model context for subsequent examples\. We limit the memory to 20,000 characters and maintain five independent memory streams over the evaluation set to reduce sensitivity to example ordering\.

##### ReAct\([Yao et al\., 2022](https://arxiv.org/html/2610.19184#bib.bib21)\)\.

ReAct interleaves language\-model reasoning with actions in an external environment\. We use ReAct as an instance\-level refinement baseline: the shared harnesshsharedh^\{\\mathrm\{shared\}\}remains fixed, while the current program and interaction history constitutehxinsth\_\{x\}^\{\\mathrm\{inst\}\}\. Execution feedback from the current instance is used to iteratively revise the generated program\.

Implementation details\.Because the original ReAct implementation161616[https://github\.com/ysymyth/ReAct](https://github.com/ysymyth/ReAct)uses free\-form prompting for text\-based tasks, we adapt ReAct to our test\-time interaction framework\. An initial candidate is first generated using the fixed shared prompt\. During refinement, the model can invoke three tools through native function calling: a read\-only diagnostic probe, execution of the current candidate, and submission of the final solution\. We allow up toB=3B=3environment interactions, excluding the final submission, and provide no few\-shot ReAct trajectories\. Model reasoning remains internal rather than being emitted as explicit thought–action traces\.

### B\.3End\-to\-End Harness Optimization

##### Meta\-Harness\([Lee et al\., 2026](https://arxiv.org/html/2610.19184#bib.bib19)\)\.

Meta\-Harness optimizes a broader shared harness than prompt\-only methods\. An agentic proposer generates candidate harness implementations using information from previously evaluated candidates, including their source code, execution traces, and validation performance\. In our formulation, Meta\-Harness optimizeshsharedh^\{\\mathrm\{shared\}\}over a search space that includes prompting, memory, multi\-call control flow, and self\-verification, while keeping the underlying policyπθ\\pi\_\{\\theta\}fixed\.

Implementation details\.We build on the authors’ reference implementation,171717[https://github\.com/stanford\-iris\-lab/meta\-harness](https://github.com/stanford-iris-lab/meta-harness)retaining its optimization loop and Pareto\-frontier selection procedure while adapting the data interface to our four benchmarks\. We use GPT\-5 as the proposer through the OpenAI Codex CLI and generate three candidate harnesses per iteration using the original skill instructions\. Candidate harnesses may use read\-only diagnostic tools, while evaluation examples and reference labels are not accessible during optimization\. Each candidate is optimized on 100 diagnosis examples for BIRD\-SQL and BigCodeBench and 50 for CypherBench and DSCodeBench, and evaluated on the full update pool\. Search terminates when the corresponding rollout budget is reached, requiring 4–11 iterations across benchmarks\. The final harness is selected based on update\-pool performance, with the initial harness retained when no candidate achieves better performance\.

### B\.4Software environment\.

All optimization, baseline, and evaluation pipelines run under Python 3\.12\.14\. We use GPT\-5 \(snapshot2025\-08\-07\) as the generator for all methods through Azure OpenAI\. Generation uses a maximum of 16,384 completion tokens and the deployment’s default sampling and reasoning settings\. Additional baseline\-specific dependencies and configurations are described in[AppendixB](https://arxiv.org/html/2610.19184#A2)\.

## Appendix CImplementation Details of HERO

##### Model configuration\.

We use GPT\-5 \(snapshot2025\-08\-07\) for program generation, prompt optimization, and test\-time refinement, with a maximum of 16,384 completion tokens and the deployment’s default sampling and reasoning settings\.

### C\.1Offline Prompt Optimization

We runT=5T=5optimization rounds with no early stopping\. In each round, we evaluate the current prompt on the full diagnosis set and uniformly sample 32 incorrect and 32 correct examples\. For incorrect examples, the reflector receives the task input, generated output, failure category, evaluator feedback, and available reference information; correct examples are included to identify behaviors that should be preserved\. The optimization context also includes aggregate failure statistics and complementary feedback from the previous round\. Based on this information, the reflector independently proposes four candidate prompt revisions, with up to eight edits per candidate\. Proposals are rejected if they cannot be parsed, leave the prompt unchanged, or introduce task\-specific values or identifiers found only in reference solutions\.

##### Candidate evaluation and acceptance\.

In each optimization round, we compare four candidate prompts with the current prompt using adaptive paired evaluation \([Algorithm2](https://arxiv.org/html/2610.19184#alg2)\)\. All candidates are initially evaluated on the same 50 examples from the 250\-example update set, with a budget of two additional 50\-example pulls allocated by the algorithm\. Candidate scores are paired improvements in execution accuracy, macro\-averaged across observed databases or domains\. Standard errors are computed from the sample standard deviation of group means divided by the square root of the number of observed groups, floored by the pooled row\-level standard error with minimum per\-row variance0\.020\.02\. We setz=1z=1for evaluation allocation and early stopping, but accept an update only if the best candidate’s estimated mean improvement exceedsτ=0\.01\\tau=0\.01\. Accepted prompts and any additional candidates exceeding this threshold are archived for final selection\.

##### Final selection\.

After five optimization rounds, we select the final prompt through a second adaptive race over the archive, with a budget equal to75%75\\%of the evaluations needed to score every archived prompt on the full selection set\.

### C\.2Test\-Time Refinement

We apply test\-time refinement independently to each evaluation instance after offline prompt optimization, without access to reference solutions or correctness labels\. Each episode starts from the initial output generated by the selected offline prompt; therefore,B=0B=0recovers the corresponding unrefined prediction\. Refinement is evaluated on the full evaluation set using the same execution\-based evaluator as offline evaluation\.

##### Tools and interaction budget\.

The refiner has access to four tools: \(i\) a read\-only diagnostic probe for inspecting the database, graph, or program behavior; \(ii\) execution of the current candidate; \(iii\) retrieval of verified examples; and \(iv\) submission of the final candidate\. We use an interaction budget ofB=3B=3, counting diagnostic, execution, and retrieval actions; final submission does not consume the budget\. Diagnostic probes and candidate execution do not expose hidden evaluation tests or reference outputs\.

##### Verified\-example retrieval\.

The retrieval store is constructed exclusively from the offline optimization data; evaluation examples are never included\. At the start of each episode, we construct a lexical shortlist of up to 20 examples based on task\-text overlap and, for code tasks, shared libraries\. The refiner may retrieve up to three examples, receiving their task descriptions and verified reference artifacts\. These examples are used to identify reusable conventions rather than instance\-specific solutions, and the refiner is instructed not to copy task\-specific values or identifiers\.

##### Refinement and verification\.

The refiner uses execution evidence and retrieved examples to determine whether the initial candidate should be revised\. Retrieval\-based revisions require consistent evidence for a relevant convention, while execution\-based revisions must be supported by observations from the current instance\. Any revised candidate must execute successfully before submission\. The controller also verifies that the stated evidence for a revision was observed during the episode; unsupported revisions are rejected\. If no supported improvement is identified, the initial candidate is retained\. The ReAct baseline in[AppendixB](https://arxiv.org/html/2610.19184#A2)uses the same interaction framework without verified\-example retrieval or evidence\-based acceptance\.

### C\.3VES Details

We follow the VES definition of[Li et al\. \(2023\)](https://arxiv.org/html/2610.19184#bib.bib1)\. For an instancexxwith reference artifactyyand predictiony^\\hat\{y\}, letE⁡\(⋅\)E\(\\cdot\)denote wall\-clock execution time andR⁡\(x\)=E⁡\(y\)/E⁡\(y^\)R\(x\)=E\(y\)/E\(\\hat\{y\}\)\. The Valid Efficiency Score \(VES\) is

VES=100N​∑x𝟏​\[y^​correct\]​R⁡\(x\),\\mathrm\{VES\}=\\frac\{100\}\{N\}\\sum\_\{x\}\\mathbf\{1\}\[\\hat\{y\}\\text\{ correct\}\]\\sqrt\{R\(x\)\},where correctness follows each benchmark’s official execution\-based evaluation \([AppendixA](https://arxiv.org/html/2610.19184#A1)\); incorrect predictions receive zero reward\. Higher VES therefore reflects both correctness and execution efficiency\.

To estimateE⁡\(⋅\)E\(\\cdot\), we perform a separate timing pass over cached evaluation outputs without additional generation\. For each instance, the reference and correct predictions are timed on the same worker, with repetitions interleaved to reduce machine\-state variation\. We execute each artifactkktimes, wherekkis benchmark\-specific, remove timings outside mean±3\\pm 3standard deviations following[Li et al\. \(2023\)](https://arxiv.org/html/2610.19184#bib.bib1), and average the remaining runs\. Instances whose references cannot be timed reliably are excluded for all methods, while correct predictions that fail during timing receive zero VES\.

##### BIRD\-SQL\.

We evaluate all 1,530 scored DEV questions\. Queries execute with Pythonsqlite3against read\-only databases using the official3030s deadline\. Timing covers query execution through full cursor consumption, with outputs above 200,000 rows treated as failures\. We usek=100k=100repetitions\.

##### BigCodeBench\-Instruct\.

Of the 637 TEST tasks, 440 are included in efficiency evaluation\. Because the hidden tests primarily consist of I/O\-bound assertions, we instead time each task’s docstring example against the generated function; 197 tasks whose examples fail on the canonical solution are excluded for all methods\. Imports occur outside the timer, and the example block is first executed for validity before timing\. We usek=100k=100repetitions with a2020s per\-execution and600600s per\-task limit\.

##### CypherBench\.

We evaluate all 1,000 frozen TEST questions against the 11 Neo4j 5\.20 graphs using the official read\-only execution setup and120120s timeout\. Timing measures client\-side wall time from query submission through full result consumption, including network and serialization overhead for both references and predictions\. We usek=10k=10repetitions\.

##### DSCodeBench\.

Of the 482 TEST problems, 453 are included in the VES evaluation; the remaining 29 are excluded because their reference solutions cannot be timed reliably\. For each problem, DSCodeBench generates the same 200 test inputs from a fixed random seed and compares the predicted outputs with those of the reference solution\. We use this same 200\-input workload for runtime measurement in the benchmark’s CPU\-only environment with the official200​s200\\,\\mathrm\{s\}limit\. The reference and correct predictions are first validated and then timed overk=3k=3interleaved repetitions, withRRcomputed from their cleaned mean runtimes\.

## Appendix DAdditional Results

### D\.1Token Consumption and prompt length

MethodBIRD\-SQLBigCodeBenchCypherBenchDSCodeBenchTokens per call \(K\)ACE38\.46±\\pm1\.0631\.51±\\pm6\.1620\.84±\\pm1\.6636\.63±\\pm1\.97MIPROv22\.17±\\pm0\.112\.41±\\pm0\.331\.82±\\pm0\.107\.47±\\pm0\.12SkillOpt3\.17±\\pm0\.432\.52±\\pm0\.192\.36±\\pm0\.433\.41±\\pm0\.40GEPA2\.42±\\pm0\.291\.86±\\pm0\.122\.24±\\pm0\.223\.36±\\pm0\.49Ours3\.10±\\pm0\.482\.10±\\pm0\.182\.57±\\pm0\.093\.36±\\pm0\.18Prompt length \(tokens\)ACE17,079±\\pm25,73545,702±\\pm11,12528,113±\\pm2,89152,631±\\pm5,026MIPROv21,192±\\pm1252,139±\\pm4151,068±\\pm5386,418±\\pm177SkillOpt2,119±\\pm1,0112,350±\\pm1,4201,905±\\pm6081,710±\\pm459GEPA2,053±\\pm8781,339±\\pm2431,882±\\pm8182,736±\\pm2,258Ours2,223±\\pm7411,655±\\pm5332,538±\\pm801,378±\\pm425

Table 5:Optimization cost across datasets\.Input plus output tokens per LLM call during optimization \(top\) and token length of the final prompt \(bottom\); mean±\\pmSD over three seeds\. Bold marks the lowest value per dataset\.Table[5](https://arxiv.org/html/2610.19184#A4.T5)compares methods along two dimensions: average tokens per LLM call during optimization and final prompt length, which affects recurring inference costs\. ACE incurs substantially higher overhead on both dimensions, requiring 20\.8–38\.5K tokens per optimization call and producing playbooks of 17–53K tokens \(8–38×\\timeslonger than those of other methods\)\. This overhead arises from repeatedly processing the accumulated playbook during reflection and curation\. In contrast, the remaining methods generally require 1\.8–3\.4K tokens per call on three of the four benchmarks, although their relative efficiency varies across tasks\. MIPROv2 has the lowest per\-call cost on BIRD\-SQL and CypherBench, while GEPA is cheapest on BigCodeBench\. However, MIPROv2’s cost increases to 7\.5K tokens per call on DSCodeBench, with a 6\.4K\-token final prompt, due to lengthy bootstrapped demonstrations\.

Our method maintains competitive optimization efficiency, remaining within 1\.1K tokens per call of the cheapest competing method across all benchmarks and achieving the lowest cost on DSCodeBench\. Its additional overhead on BIRD\-SQL and CypherBench primarily reflects the inclusion of 64 diagnosed cases in each whole\-prompt rewrite\. Importantly, our final prompts remain compact \(1\.4–2\.5K tokens\), achieving the shortest length on DSCodeBench and remaining within2×2\\timesof the shortest prompts elsewhere\. Prompt lengths are also stable across seeds \(e\.g\., SD of 80 tokens on CypherBench, compared with 538–818 for MIPROv2 and GEPA\)\. Since prompt length contributes to the input\-token cost of each downstream query, these results indicate that the performance improvements in Table[1](https://arxiv.org/html/2610.19184#S5.T1)are achieved without substantially increasing the shared prompt overhead at inference time\.

Figure 5:Pass@1 versus cost\.Pass@1 performance against optimization token budget \(top\) and final prompt length \(bottom\) per benchmark\. Points show 3\-seed means with standard deviation \(log\-scale x\-axes\)\.
### D\.2Hyperparameter Sensitivity

##### Rendered diagnostic examples\.

HyperparameterValueBIRD\-SQLBigCodeBenchCandidatesper roundKK4†4^\{\\dagger\}63\.55\(0\.21\)51\.96 \(0\.00\)661\.90 \(1\.25\)52\.07\(0\.66\)862\.39 \(1\.52\)51\.38 \(0\.55\)1061\.74 \(1\.61\)51\.65 \(1\.41\)1262\.83 \(0\.81\)51\.81 \(1\.34\)Search roundsTT361\.66 \(1\.52\)51\.86 \(0\.81\)462\.00 \(1\.56\)51\.65 \(0\.96\)5†5^\{\\dagger\}63\.55\(0\.21\)51\.96 \(0\.00\)661\.52 \(0\.37\)52\.33\(0\.24\)762\.90 \(0\.56\)51\.39 \(0\.50\)Rendered casesper outcome059\.15 \(0\.93\)50\.08 \(1\.55\)862\.38 \(1\.19\)51\.54 \(1\.81\)1661\.39 \(1\.62\)50\.97 \(1\.19\)32†32^\{\\dagger\}63\.55\(0\.21\)52\.59\(0\.31\)6463\.16 \(0\.39\)51\.39 \(0\.36\)

Table 6:Offline search hyperparameter sensitivity\.Pass@1 \(mean \(SD\), three seeds\) on BIRD\-SQL DEV and BigCodeBench\. Each section varies one setting;†\\daggermarks defaults \(K=4K=4,T=5T=5, 32 rendered cases per outcome\)\. Bold denotes the best mean per section and benchmark\.Rendered diagnostic cases provide concrete evidence of failures to correct and successful behaviors to preserve\. We examine whether this evidence improves prompt revision and how performance depends on its amount \([Table6](https://arxiv.org/html/2610.19184#A4.T6)\)\. Removing all rendered cases lowers mean pass@1 by 4\.40 percentage points on BIRD\-SQL and 2\.51 points on BigCodeBench relative to the default of 32 failed and 32 passing cases\. With examples included, performance remains above the seed prompt across all tested counts \(8–64 per outcome\), but does not improve monotonically with their number\. The default achieves the highest mean accuracy on both benchmarks, and doubling the counts provides no further gain\. These findings support using concrete diagnostic evidence while indicating that a moderate evidence set is sufficient in the evaluated settings\.

SettingBIRD\-SQLBigCodeBenchDSCodeBenchCypherBenchMeanFull3\.011\.611\.702\.062\.10Failure\-only2\.611\.711\.491\.411\.81

Table 7:Effect of successful diagnostic examples\.Cumulative fix\-to\-break ratios over accepted rounds, averaged over three seeds\. Full: 32 failed \+ 32 passing cases; failure\-only: 32 failed cases\. Mean averages the four benchmarks\.
##### Role of successful examples\.

Beyond the amount of diagnostic evidence, we examine whether successful executions provide useful guidance alongside failures\. Failure cases expose errors to correct, while successful cases illustrate behaviors to preserve\. To assess this balance, Table[7](https://arxiv.org/html/2610.19184#A4.T7)compares the fix\-to\-break ratio under combined success\-and\-failure feedback and failure\-only feedback\. Including successful examples raises the mean ratio from 1\.81 to 2\.10, with improvements on three of four benchmarks\. The largest increase occurs on CypherBench \(1\.41 to 2\.06\), although BigCodeBench slightly favors failure\-only feedback \(1\.71 versus 1\.61\)\. This overall improvement supports including successful examples to guide revisions toward a better balance between fixes and regressions\.

##### Offline search settings\.

We next examine whether the gains depend on specific choices of search breadth and depth \([Table6](https://arxiv.org/html/2610.19184#A4.T6)\)\. Varying the number of candidates per round \(K=4K=4–1212\) and optimization rounds \(T=3T=3–77\) yields mean pass@1 scores of 61\.52–63\.55 on BIRD\-SQL and 51\.38–52\.33 on BigCodeBench\. All tested configurations remain above the seed prompt, but increasing eitherKKorTTdoes not consistently improve accuracy\. The gains therefore persist across multiple search configurations without requiring the largest tested search budgets\.

##### Test\-time refinement budget\.

Refinement budgetBBReActDatasetMetric02𝟑†\\mathbf\{3\}^\{\\dagger\}468𝐁=𝟑\\mathbf\{B=3\}BIRD\-SQLPass@1 \(%\)62\.53 \(1\.82\)63\.38 \(1\.83\)63\.59 \(1\.43\)63\.53 \(1\.53\)63\.77 \(1\.47\)64\.05\(1\.57\)63\.12 \(1\.02\)Env\. actions01\.351\.591\.661\.761\.780\.79LLM calls03\.153\.333\.403\.463\.431\.93BigCodeBenchPass@1 \(%\)52\.59 \(0\.31\)53\.38 \(0\.87\)53\.79 \(0\.33\)53\.95 \(0\.74\)54\.06\(0\.33\)53\.85 \(0\.31\)52\.64 \(0\.24\)Env\. actions01\.882\.312\.402\.492\.501\.91LLM calls03\.914\.334\.444\.524\.523\.84

Table 8:Test\-time refinement budget sensitivity\.BBcaps environment actions per query;†\\daggermarks the default\.B=0B=0denotes no refinement\. Pass@1: mean \(SD\) over three seeds\. Actions and calls: mean environment actions and refinement LLM calls per query\. Bold denotes the highest mean pass@1 per benchmark\.We similarly examine whether refinement gains require a large per\-query interaction budget \([Table8](https://arxiv.org/html/2610.19184#A4.T8)\)\. Refinement improves mean pass@1 over the optimized prompt at every evaluated nonzero budget\. At the defaultB=3B=3, scores increase from 62\.53 to 63\.59 on BIRD\-SQL and from 52\.59 to 53\.79 on BigCodeBench\. The best scores at larger budgets add only 0\.46 and 0\.27 percentage points, respectively, indicating that the default captures much of the observed benefit\.

The resource\-usage figures help interpret these budget comparisons\. Figures[6](https://arxiv.org/html/2610.19184#A4.F6)and[7](https://arxiv.org/html/2610.19184#A4.F7)show that actual computation largely plateaus beyondB=3B=3: increasingBBfrom 3 to 8 adds just 0\.19 environment actions per query on each benchmark, while refinement LLM calls increase by 0\.10 on BIRD\-SQL and 0\.19 on BigCodeBench\. Thus, larger allowances remain partly unused and produce only small changes in average computation\. At the same cap ofB=3B=3, our refinement also exceeds ReAct’s mean pass@1 by 0\.47 and 1\.15 percentage points, respectively, while using more environment actions and LLM calls\.

Figure 6:Refinement resource usage on BigCodeBench\.Mean environment actions \(left\) and LLM calls \(right\) per query versus budgetBB\. Hollow points: optimized prompt only; dashed lines: full budget utilization; gray points: plain ReAct \(B=3B=3\)\.Figure 7:Refinement resource usage on BIRD\-SQL\.Mean environment actions \(left\) and LLM calls \(right\) per query versus budgetBB\. Hollow points: optimized prompt only; dashed lines: full budget utilization; gray points: plain ReAct \(B=3B=3\)\.

### D\.3Model Generalization

BaselineOffline Prompt OptimizationTest\-Time AdaptationHarness OptimizationDatasetSeedOursSkillOptGEPAMIPROv2ACEDC\-CuOurs\+ RefineOurs\+ ReActOurs\+ DC\-CuMeta\-HarnessBIRD DEV58\.91 \(0\.44\)63\.66 \(1\.18\)64\.38 \(1\.21\)63\.01 \(1\.30\)63\.03 \(0\.84\)63\.36 \(1\.57\)62\.14 \(0\.27\)65\.38\(1\.13\)63\.57 \(1\.10\)64\.12 \(0\.69\)58\.52 \(0\.72\)BigCodeBench52\.22 \(0\.77\)52\.43 \(0\.57\)52\.96\(0\.36\)52\.69 \(0\.48\)52\.54 \(1\.94\)52\.49 \(0\.36\)50\.71 \(1\.77\)52\.59 \(0\.57\)53\.48\(0\.50\)53\.38 \(1\.13\)51\.49 \(0\.28\)DSCodeBench50\.83 \(0\.55\)55\.39\(1\.70\)54\.56 \(0\.72\)53\.94 \(0\.55\)54\.84 \(0\.94\)55\.26 \(1\.22\)45\.09 \(1\.69\)54\.63\(0\.32\)54\.08 \(0\.43\)54\.49 \(0\.21\)50\.14 \(2\.80\)CypherBench73\.13 \(0\.45\)83\.67\(1\.10\)79\.00 \(1\.37\)76\.87 \(2\.05\)79\.37 \(2\.36\)82\.17 \(2\.10\)73\.83 \(1\.01\)88\.63\(2\.02\)87\.87 \(1\.42\)81\.13 \(3\.20\)82\.47 \(1\.96\)

Table 9:Performance across four code\-generation benchmarks using GPT\-5\.4\.Results \(pass@1\) are reported as mean \(standard deviation\) over three runs\. Bold highlights the best reported mean within each optimization stage\.##### Results on GPT\-5\.4

To assess generalization across model backbones, we extend our evaluation to GPT\-5\.4 on all four benchmarks\. As shown in[Table9](https://arxiv.org/html/2610.19184#A4.T9), our offline optimization improves over the seed prompt across all benchmarks, with gains of 4\.75, 0\.21, 4\.56, and 10\.54 percentage points on BIRD\-SQL, BigCodeBench, DSCodeBench, and CypherBench, respectively\. Our method achieves the highest offline pass@1 on DSCodeBench \(55\.39\) and CypherBench \(83\.67\), while remaining competitive with the strongest baselines on the other benchmarks\.

Test\-time refinement further improves performance on BIRD\-SQL \(63\.66 to 65\.38\) and CypherBench \(83\.67 to 88\.63\), achieving the highest overall pass@1 on both benchmarks\. However, gains are smaller on BigCodeBench \(\+0\.16 points\), and performance decreases slightly on DSCodeBench \(\-0\.76 points\), where offline optimization alone achieves the best result\. On BigCodeBench, combining our optimized prompt with ReAct yields the highest performance \(53\.48\)\. Overall, these results demonstrate that our offline optimization remains effective with GPT\-5\.4, while the additional benefits of test\-time refinement are benchmark\-dependent\.

## Appendix EAI Use Statement

In this work, we used generative AI tools to assist with figure generation from experimental results, language and grammar editing, code generation for method implementation, experiment execution, and statistical analysis\. All AI\-assisted code was reviewed, verified, and tested for correctness by the authors\. Separately, LLMs are also an explicit object of study within the proposed framework: both the frozen model that is used and the reflector that updates the harness\.

We did not use generative AI to generate research ideas, formulate or refine hypotheses, generate scientific claims, develop theoretical or conceptual frameworks, formulate mathematical claims or proofs, or generate synthetic datasets\. The authors take full responsibility for all text, claims, code, figures, and other artifacts in the final work\.

## Appendix FEthical Statement

HERO aims to improve the reliability and efficiency of automated code generation\. However, improvements in code\-generation capabilities could potentially be misused to automate malicious operations or access sensitive information\. Deploying such systems may also raise privacy and security concerns, particularly when interacting with external tools or databases\. We encourage responsible deployment practices, including sandboxed execution, appropriate access controls, and human oversight in sensitive applications\.

## Appendix GPrompt Templates

BIRD\-SQLThink step by step\.Use the provided database schema\.Return exactly ONE SQLite query in a \`\`\`sql code block, selecting exactly the columns the question asks for, in the order asked\.Check the query before returning it\.

BigCodeBench / DSCodeBenchThink step by step\.Use the provided task specification\.Return exactly ONE self\-contained Python solution in a \`\`\`python code block, starting with exactly the imports and function signature specified in the task\.Check the code before returning it\.

CypherBenchThink step by step\.Use the provided graph schema\.Return exactly ONE Cypher query in a \`\`\`cypher code block, returning exactly the entity names or property values the question asks for \(never whole nodes or relationships\)\.Check the query before returning it\.

Figure 8:Seed system prompts across benchmarks\.All optimization methods are initialized with the same benchmark\-specific seed prompt\. Task\-specific inputs are provided separately in the user message\.### G\.1Optimized Prompt for BIRD\-SQL

HEROPlan the query before writing it\.\- Determine the exact output the question requests: the column names, their order, and how many columns the result should have\. If the request says "full name" and the schema stores name parts separately, treat the full name as those separate parts in multiple columns unless the question explicitly asks for a single combined string or the schema has a single full\-name column\.\- Identify the required grain \(per row, per day, per episode, per team/season, per order, etc\.\)\. If the question mentions a specific grain like "in a single day" or "per year", operate at that grain rather than pre\-aggregating to a higher level\.\- Extract filters and map each to the correct column\(s\) by name \(ids, codes, dates, categories\)\. Keep filters minimal: include only constraints stated or necessary for the computation\.\- Choose the minimal set of tables that contain the needed columns\. Prefer direct join paths indicated by foreign keys over indirect or text\-based joins\.\- Decide the aggregation: COUNT rows for "how many" entities; SUM a numeric measure only when the question asks for a total of that measure \(e\.g\., goals/points/units made\) and the column stores that quantity per row; use AVG only when asked\. Use DISTINCT only when the question says distinct/unique/different\.\- For "most/least/top N" questions, define the ordering expression, add a deterministic secondary tie\-breaker \(e\.g\., id or name ascending\), and use LIMIT appropriately\. Prefer ORDER BY \.\.\. LIMIT over equality to global MIN/MAX to avoid ties and type issues\.\- For multi\-part questions that request different results, plan a single result schema that fits all parts\. If parts share a single column type, combine them with UNION/UNION ALL into one column set\. Do not add extra columns unless the question explicitly asks for them\.\- For percentages, define numerator and denominator explicitly to match the question, and compute numerator \* 100\.0 / denominator with REAL casting to avoid integer division\.\- Prefer the simplest correct formulation \(fewest joins, no unnecessary subqueries/CTEs\) to reduce the chance of timeouts\.Constrain all logic to the provided schema\.\- Use only existing tables/columns\. Select join keys from declared foreign keys and primary keys; include all columns of composite keys in join predicates\.\- When a measure exists in multiple tables, pick the table that is the factual source of that measure \(e\.g\., a detail/fact table\) unless the question explicitly targets a summary/lookup\.\- Map natural\-language attributes to columns with matching semantics:\- Names: if the schema splits a name into parts, match on the appropriate first/last \(and middle if needed\) columns; do not use nickname/alias columns unless the question mentions them\.\- Birth/birthplace→\\rightarrowbirth\_\* columns; high school→\\rightarrowhs\* columns; college→\\rightarrowcollege\* columns; episode/season/series→\\rightarrowthe corresponding columns; addresses used by an order→\\rightarrowthe address referenced by the order, not a customer’s unrelated addresses\.\- Facts related to weather, stations, or locations: when keys exist \(e\.g\., zip or station id\), join on both the date component and the location key rather than text\-only or date\-only joins\.\- Use INNER JOIN by default\. Use LEFT JOIN only when the question requires keeping unmatched entities \(e\.g\., "including those with zero"\)\. Ensure no join omits a necessary key that would create a Cartesian product\.\- Do not introduce DISTINCT, extra GROUP BY, IS NOT NULL, or LIKE patterns unless the question calls for them or they are necessary to faithfully map natural\-language tokens that may vary \(e\.g\., case or pluralization\)\.\- Text comparisons:\- Use case\-insensitive equality \(e\.g\., COLLATE NOCASE or UPPER\(\.\.\.\)=UPPER\(\.\.\.\)\) for names and categories unless the column is a case\-sensitive code\.\- For categorical/unit tokens where the natural\-language term may differ in case or simple pluralization/punctuation from stored values, prefer case\-insensitive equality; if ambiguity remains, use a narrowly targeted LIKE that matches the intended token \(e\.g\., a substring of the unit name\) rather than inventing multiple alternatives\.\- Dates/times: use the stored formats\. If extracting parts \(year/month/day\) is necessary, use SQLite text functions consistently with the stored format\.Return exactly ONE SQLite query in a ‘‘‘sql code block\.\- Select exactly the columns requested, in the order requested\. Do not add extra columns, labels, or computed/decoded values unless explicitly requested\.\- Names: if the schema stores name parts separately and the question requests a "full name", return the separate name\-part columns \(e\.g\., first, middle, last\) as separate output columns; only concatenate into a single string if explicitly asked for a single combined full\-name string or if a single full\-name column exists in the schema\.\- Multi\-part questions: produce a single column set across the whole result\. If different parts request values of the same column type, combine them with UNION or UNION ALL so the output shape remains constant\. Do not append unrelated extra columns to satisfy a later part\.\- Aggregation:\- "How many" entities→\\rightarrowCOUNT over the relevant rows; use COUNT\(DISTINCT \.\.\.\) only if the question says distinct/unique/different or you are counting unique entities across multiple rows\.\- Totals of a numeric measure explicitly mentioned \(e\.g\., goals/points/units made\)→\\rightarrowSUM the measure column over the filtered set\.\- Averages→\\rightarrowAVG over the intended numeric expression; cast to REAL as needed to avoid integer division\.\- For "zero" conditions on an existing numeric column, filter that column equals 0 rather than using NOT EXISTS\.\- Joins:\- Join tables using explicit foreign keys; include all parts of composite keys\.\- Prefer key\-based joins \(ids, codes\) over text/date\-string joins when keys exist\.\- For weather or other time/location facts, join on both date \(or date substring\) and the relevant location key \(e\.g\., zip/station\) when both are available\.\- Granularity:\- Respect the grain implied by the question \(e\.g\., per day/episode/order\)\. For phrases like "in a single day" or per\-episode counts, avoid pre\-aggregating to coarser grains \(e\.g\., per station overall\) before ranking; order or aggregate at the specified grain\.\- Ordering and limits:\- For "top N"/"lowest/highest" return ordered results with LIMIT N and a deterministic secondary tie\-breaker \(e\.g\., id or name ASC\) when ties are unspecified\.\- Prefer ORDER BY with LIMIT to equality against global MIN/MAX to avoid missing rows due to ties or type casting\.\- Text filtering:\- Use case\-insensitive comparisons for names/categories \(COLLATE NOCASE or UPPER\(\.\.\.\)\)\.\- For unit/category tokens susceptible to singular/plural or punctuation differences, if exact equality is uncertain, use a careful LIKE on a stable substring of the token\.\- Keep the query minimal \(avoid unnecessary CTEs/subqueries, DISTINCT, or extra joins\) to reduce runtime and avoid timeouts\.Before returning the query, validate the following:\- Output shape:\- The number of columns and their order exactly match what the question asks\. If a "full name" is requested and the schema has separate name parts, ensure the output includes those parts as separate columns \(not concatenated\) unless a single combined string is explicitly requested\.\- For multi\-part requests, ensure a single, consistent set of columns is returned across all parts\. Use UNION/UNION ALL across SELECTs that project the same columns rather than adding extra columns for later parts\.\- Joins:\- Every join uses the correct key\(s\) as indicated by foreign keys; include all parts of composite keys\.\- There is no missing connecting condition that could create a Cartesian product\.\- Where both a date and a location key are relevant \(e\.g\., facts to weather\), both are part of the join/filter\.\- Filters:\- Each literal maps to the correct column; no speculative alternatives \(no extra ORs, no nickname fallbacks\) unless required by the question\.\- Text comparisons are case\-insensitive where appropriate\. For unit/category tokens where singular/plural or punctuation could differ, consider using a precise LIKE on a meaningful substring if exact equality might fail\.\- Time filters match the specified range/grain and use the stored format correctly \(e\.g\., year extraction, month/day substrings\)\.\- Aggregation and DISTINCT:\- Use COUNT to answer "how many" entities; use SUM only when summing an explicit numeric measure is required by the question; use COUNT\(DISTINCT \.\.\.\) only when uniqueness is requested\.\- The grouping or lack of grouping matches the intended grain\. Do not pre\-aggregate to a higher grain than requested\.\- Ranking and extremes:\- For top/bottom/most/least, verify ORDER BY uses the intended measure and includes a deterministic secondary tie\-breaker; prefer ORDER BY \.\.\. LIMIT over equality to MIN/MAX\.\- Result sanity:\- Avoid transforming requested raw values into derived labels \(e\.g\., do not convert flags into Yes/No unless asked\)\.\- Check that the query is unlikely to return zero rows if the question implies an answer exists; if risk arises from equality to an extreme value or over\-restrictive filters, consider re\-expressing with ORDER BY/LIMIT or removing non\-essential filters\.If any check fails, revise the query before returning it\.

Figure 9:Example optimized system prompt produced by HERO on BIRD\-SQL\.The prompt provides detailed instructions for query planning, schema grounding, SQL generation, and verification\.GEPAYou will be given:\- question: the natural\-language request\- database\_schema: the exact SQLite schema available\- external\_knowledge: optional hints clarifying phrasing or value mappingsYour task:\- Return exactly one SQLite query inside a single ‘‘‘sql code block and nothing else\.\- Select exactly and only the columns the question asks for, in the order asked\.\- Use only tables and columns present in the provided database\_schema\.\- Do not concatenate, compute, alias, or rename columns unless the question explicitly asks for it \(e\.g\., “as a single string”\)\.\- Avoid extraneous filters, joins, DISTINCT, GROUP BY, or ORDER BY clauses unless required by the question\.Reasoning and verification:\- Carefully map any domain\-specific phrasing using external\_knowledge, but ensure it aligns with actual schema columns/values\.\- Prefer the simplest query that satisfies the question to avoid timeouts \(minimize joins and avoid large event tables unless explicitly needed\)\.\- Verify the output shape \(number and order of columns\) matches the question exactly and that the query runs logically on the given schema\.\- Do not add unnecessary conditions like IS NOT NULL unless the question requires them\.\- When computing ratios or percentages, ensure real \(floating\-point\) division using 100\.0 or CAST\(\.\.\. AS REAL\) to avoid integer division\.\- When aggregating over joins, do not add DISTINCT unless the question explicitly requires it; count/sum/avg over the joined rows as\-is\.\- Use explicit INNER JOINs via declared foreign keys instead of EXISTS filters when linking entities mentioned in the question\.General patterns:\- Use LIMIT for “top K” requests\.\- Use ORDER BY only when the question implies an ordering \(e\.g\., “top/most”, “lowest/highest”, “earliest/latest”\)\.\- For “most/least” by a numeric measure, ORDER BY that measure DESC/ASC accordingly and LIMIT 1\.\- Use COUNT or COUNT\(DISTINCT \.\.\.\) only when aggregation or distinctness is explicitly required by the question\.\- Join via declared foreign keys in the schema \(e\.g\., donations\.projectid \-\> projects\.projectid; app\_labels\.label\_id \-\> label\_categories\.label\_id; Crime\.ward\_no \-\> Ward\.ward\_no\)\.\- Do not add secondary sort keys unless specified; sort only as required \(e\.g\., by COUNT DESC for “top/most”\)\.\- When multiple years/records per entity exist and the question does not specify a time period, keep the query simple: select the requested column\(s\) with the minimal required filter and, if an extreme \(“lowest/highest”\) is asked, order accordingly and LIMIT 1\.Domain\-specific guidance and mappings:\- TV/person/credit schemas:\- A “crew member from \[Department\]” maps to Credit\.category = ’<Department Name\>’\.\- For average heights by department, join Person to Credit and filter Credit\.category \(e\.g\., AVG\(Person\.height\_meters\) with INNER JOIN Credit on the appropriate key\)\.\- Depending on schema, Credit may reference Person by name or ID:\- If Credit\.person references Person\.name, join ON Credit\.person = Person\.name\.\- If Credit\.person\_id references Person\.person\_id, join ON Credit\.person\_id = Person\.person\_id\.\- “credited” status refers to Credit\.credited = ’true’ \(string\)\.\- To count how many times someone was credited, join Person to Credit, filter by Person\.name and Credit\.credited = ’true’, and COUNT\(\*\) \(or COUNT of the joined key\)\. Do not join Episode unless required by the question\.\- DonorsChoose projects/resources:\- “main subject area” maps to projects\.primary\_focus\_area\.\- “resource type supplies” maps to resources\.project\_resource\_type = ’Supplies’\.\- “most requested item” refers to the item with the highest resources\.item\_quantity; select resources\.item\_name, ORDER BY item\_quantity DESC, LIMIT 1\.\- “public magnet school” maps to projects\.school\_magnet = ’t’\.\- City filters \(e\.g\., “Brooklyn”\) map to projects\.school\_city = ’Brooklyn’\.\- “projects created by a teacher working in …” generally means filter using project attributes and join donations/resources only if the question references those tables’ fields\.\- Apps/categories dataset:\- “categories” are label\_categories\.category\.\- For “top N categories with the most app users/popularity,” aggregate over app\_labels joined to label\_categories:SELECT category, COUNT\(app\_id\)FROM app\_labels alJOIN label\_categories lc ON al\.label\_id = lc\.label\_idGROUP BY categoryORDER BY COUNT\(app\_id\) DESCLIMIT N\- Do not join events or device tables unless the question explicitly asks about devices or active/installed users\.\- Chicago crime and wards:\- “case No\. <ID\>” maps to Crime\.case\_number = ’<ID\>’\.\- “alderman of the legislative district” means join Crime\.ward\_no to Ward\.ward\_no\.\- When asked for “full name,” return two columns alderman\_first\_name and alderman\_last\_name in that order unless the question explicitly requests a single concatenated string\.\- For “percentage of crimes in <district name\>,” join Crime to District on district\_no and compute the percentage in a single pass to avoid mismatched denominators and integer division:SELECT CAST\(SUM\(CASE WHEN d\.district\_name = ’<District Name\>’ THEN 1 ELSE 0 END\) AS REAL\) \* 100\.0 / COUNT\(c\.case\_number\)FROM Crime cINNER JOIN District d ON c\.district\_no = d\.district\_no\- Use COUNT\(c\.case\_number\) \(not COUNT\(\*\)\) when counting cases to align with case\-based counts\.\- World Bank Indicators\-style datasets:\- Country name is typically Indicators\.CountryName \(or specific ShortName/LongName columns if explicitly required\)\.\- “arable land percentage” maps to IndicatorName LIKE ’Arable land \(% of land area\)’\.\- For “lowest/highest,” use ORDER BY Value ASC/DESC LIMIT 1\.\- Do not add year filters or aggregations unless the question specifies a time period\.Style and correctness:\- Use exact table/column names from database\_schema\. If the schema uses quoted identifiers \(e\.g\., "Episode"\), reference them exactly as defined\.\- Treat booleans as stored in the schema \(commonly ’true’/’false’ or ’t’/’f’ strings\)\.\- Ensure the final query returns exactly the requested columns and nothing else\.\- Output only the SQL query inside a single ‘‘‘sql code block and no additional text\.

Figure 10:Example GEPA\-optimized system prompt on BIRD\-SQL\.ACEThink step by step\. Use the provided database schema\. Return exactly ONE SQLite query in a ‘‘‘sql code block, selecting exactly the columns the question asks for, in the order asked\. Check the query before returning it\.\#\# PLAYBOOKThe playbook below contains strategies and insights accumulated from previous attempts at similar text\-to\-SQL questions\. Read it carefully, apply the relevant bullets, and avoid the mistakes it lists\.\#\#\#\# STRATEGIES & INSIGHTS\[str\-00001\] helpful=606 harmful=22 :: Use the minimal set of predicates needed to answer the question; avoid adding redundant filters that can undercount or overconstrain results\.\[str\-00002\] helpful=233 harmful=16 :: Aggregate at the source of truth for the metric \(the table that records the event\) to avoid join\-induced row multiplication; only join after pre\-aggregating or deduplicating if needed\.\[\.\.\. 68 more bullets omitted \.\.\.\]\#\#\#\# FORMULAS & CALCULATIONS\[calc\-00006\] helpful=34 harmful=0 :: BETWEEN is inclusive of both endpoints \(e\.g\., age BETWEEN 13 AND 19 is equivalent to age \>= 13 AND age <= 19\)\.\[calc\-00007\] helpful=221 harmful=9 :: COUNT\(\*\) counts rows; COUNT\(column\) ignores NULLs\. Use COUNT\(\*\) for total rows, and COUNT\(DISTINCT key\) for unique\-entity counts across joins\.\[\.\.\. 38 more bullets omitted \.\.\.\]\#\#\#\# CODE SNIPPETS & TEMPLATES\[code\-00123\] helpful=27 harmful=0 :: Two percentage patterns \-\- choose based on intended grain: event\-weighted share \(occurrence\-based\) versus entity\-weighted share \(per\-parent existence\)\. Ensure numerator and denominator are derived from the same cohort and grain\.\[\.\.\. 50 more bullets omitted \.\.\.\]\#\#\#\# COMMON MISTAKES TO AVOID\[err\-00013\] helpful=25 harmful=93 :: Overfiltering by adding constraints not required by the question \(e\.g\., adding city/title when zip alone determines the answer\)\.\[err\-00014\] helpful=7 harmful=4 :: Joining on incomplete keys \(e\.g\., date\-only joins to contextual tables\) that widen the join and distort aggregates; include all relevant keys such as location\.\[\.\.\. 172 more bullets omitted \.\.\.\]\#\#\#\# PROBLEM\-SOLVING HEURISTICS\[ph\-00009\] helpful=775 harmful=7 :: Determine the intended grain before writing the query \(row\-level, per\-group list, or single scalar\) and ensure SELECT, GROUP BY, and HAVING reflect that grain\.\[\.\.\. 32 more bullets omitted \.\.\.\]\#\#\#\# CONTEXT CLUES & INDICATORS\[ctx\-00031\] helpful=68 harmful=2 :: Language cues: words like ’total’, ’sum’, ’per’, ’by’, ’each’ imply aggregation; phrases like ’highest/lowest \[column\]’ without ’total’ suggest row\-level ordering rather than grouping\.\[\.\.\. 79 more bullets omitted \.\.\.\]

Figure 11:Example ACE\-optimized system prompt on BIRD\-SQL\.ACE extends the initial system prompt with a playbook of accumulated strategies, formulas, code templates, and common mistakes\. Each entry includes helpful and harmful counts; additional entries and some long templates are omitted\.MIPRO\-v2You are a senior analytics engineer and SQLite text\-to\-SQL specialist working on a cross\-domain NL\-to\-SQL benchmark\. Your job is to translate a natural\-language question, together with the provided SQLite schema and optional external knowledge, into exactly one correct SQLite query\.Follow this process silently \(do not show your reasoning; only output the SQL\):\- Parse the question to identify:\- The exact columns requested and their order\.\- Whether the result should be a scalar, a short list, or pairs/tuples\.\- Filters, time ranges, groupings, aggregates, ratios/percentages, and any “top/maximum/minimum” requirements\.\- Any requested sort order or limit\. Do not add ORDER BY unless explicitly required or needed for “top/first/maximum/minimum”\.\- Map terms to schema:\- Use the provided CREATE TABLE statements and column names only\.\- Use the external knowledge hints for term/column mappings, formulas, and code\-to\-label linkages\.\- Resolve synonyms and paraphrases \(e\.g\., “game”→\\rightarrowgame\_name\) and prefer human\-readable label columns \(name/title\) over IDs when the question asks for names\.\- Plan joins:\- Use the minimal set of tables and equijoins on obvious keys \(foreign keys, id columns, or name/code columns indicated by external knowledge\)\.\- Favor star\-like joins \(fact to dimensions\)\. Avoid unnecessary joins that create duplicates\.\- Implement filters and computations:\- Use WHERE for row filters; use HAVING for filters on aggregated results\.\- For “how many” counts of entities, use COUNT\(DISTINCT appropriate\_id\_or\_name\) if duplicates are possible\.\- For “list” queries that imply uniqueness, use DISTINCT\.\- For ratios/percentages: 100\.0 \* CAST\(numerator AS REAL\) / NULLIF\(denominator, 0\)\. Only ROUND if requested\.\- For conditional logic, use CASE WHEN … END\.\- For maximum/minimum or “highest/lowest”, use ORDER BY … DESC/ASC LIMIT 1, or use MAX/MIN with a subquery when ties or grouping require it\.\- Handle messy data robustly:\- CAST text\-encoded numbers to REAL/INTEGER after cleaning \(e\.g\., REPLACE\(text\_col, ’,’, ’’\)\)\. Use TRIM to remove extra spaces\.\- For dates/timestamps stored as text, extract parts with SUBSTR or use strftime if stored in date\-like formats\. Example: SUBSTR\(date\_col, 1, 4\) = ’2020’ for year filtering when needed\.\- Use COALESCE or IS NULL checks where appropriate\.\- Keep to SQLite\-compatible SQL; CTEs are allowed but the entire answer must be a single executable SELECT query\.Output requirements:\- Return exactly ONE SQLite query inside a single ‘‘‘sql fenced code block\.\- Select exactly and only the columns the question asks for, in the precise order requested\.\- Do not include explanations, comments, or additional text outside the code block\.\- Do not use SELECT \*\.\- Do not add ORDER BY unless explicitly requested by the question or required to pick extremes or “top N”\.Final checklist before output:\- Columns match the question’s requested fields and order\.\- Tables and columns exist in the provided schema\.\- Joins and filters reflect the question and any external knowledge mappings\.\- Aggregations, DISTINCT, and casts are correct; no integer\-division errors\.\- No extraneous columns, ordering, or text beyond the single ‘‘‘sql code block\.\#\# EXAMPLESExample question/SQL pairs from this task \(from other databases; schemas omitted\)\. Use them to calibrate style and conventions\.\#\#\#\# Example 1Question: What is the most requested item under the resource type "Supplies" for projects whose main subject area is Literacy & Language?External knowledge: main subject area refers to primary\_focus\_area = ’Literacy & Language’; resource type supplies refers to project\_resource\_type = ’Supplies’; most requested item refers to Max\(item\_quantity\);SQL:‘‘‘sqlSELECT r\.item\_nameFROM resources rJOIN projects p ON p\.projectid = r\.projectidWHERE r\.project\_resource\_type = ’Supplies’AND p\.primary\_focus\_area = ’Literacy & Language’ORDER BY r\.item\_quantity DESCLIMIT 1‘‘‘\#\#\#\# Example 2Question: What is the first and last name of the player with the highest field goal made rate in 1973?External knowledge: in 1973 refers to year = 1973; player with highest field goal made refers to Max\(Divide\(fgMade, fgAttempted\)\)SQL:‘‘‘sqlSELECT T1\.firstName, T1\.lastName FROM players AS T1 INNER JOIN players\_teams AS T2 ON T1\.playerID = T2\.playerID WHERE year = 1973 ORDER BY CAST\(T2\.fgMade AS REAL\) / T2\.fgAttempted DESC LIMIT 1‘‘‘\#\#\#\# Example 3Question: Please list the IDs of the solutions that contain files within the top 3 followed repositories\.External knowledge: more forks refers to more people follow this repository;SQL:‘‘‘sqlSELECT T2\.Id FROM Repo AS T1 INNER JOIN Solution AS T2 ON T1\.Id = T2\.RepoId ORDER BY T1\.Forks DESC LIMIT 3‘‘‘\#\#\#\# Example 4Question: What is the average download for entertainment apps with size no more than 1\.0 M?External knowledge: downloads and installs are synonyms; entertainment apps refers to Category = ’ENTERTAINMENT’;SQL:‘‘‘sqlSELECT AVG\(CAST\(REPLACE\(REPLACE\(Installs, ’,’, ’’\), ’\+’, ’’\) AS INTEGER\)\) FROM playstore WHERE Category = ’ENTERTAINMENT’ AND Size < ’1\.0M’‘‘‘

Figure 12:Example MIPRO\-v2\-optimized system prompt for BIRD\-SQL\.The prompt includes instructions for SQL generation, schema grounding, query verification, and four example question–SQL pairs\.