MoMHa：针对准确性、安全性和令牌的LLM套件多目标优化

# Multi-Objective Optimization ofLLM Harnesses over Accuracy, Safety, and Tokens
Source: [https://arxiv.org/html/2609.30967](https://arxiv.org/html/2609.30967)
Md Mehrab TanjimAffiliation:Adobe ResearchEmail:[tanjim@adobe\.com](mailto:)

###### Abstract

Most work on improving large language models treats accuracy as the sole objective\. We argue that the*harness*, the Python code surrounding the model that constructs prompts, routes calls, and parses outputs, is a first\-class design surface whose quality is inherently*multi\-objective*: an accurate harness that refuses no unsafe request, or that consumes an order of magnitude more tokens, is not a good harness\. We presentMeta\-Harness, a system that casts harness design as search over three per\-domain objectives \(accuracy, behavioural safety, and token cost\) solved by an agentic proposer \(Claude Code\) with full filesystem access to prior harness source, execution traces, and scoring artifacts\. Our central finding is that a*single\-phase joint\-reward*proposer \(MoMHa\) outperforms every alternative, including a two\-phase “accuracy then tokens” ablation, scalar\-only feedback, and an accuracy\-only baseline\. We evaluate onseventeen domains: seven synthetic capability suites, seven real\-world public benchmarks \(HumanEval, MBPP, Spider, FEVER, MMLU\-Pro, LawBench, NuminaMath\), and three U\-SafeBench\-derived user\-specific safety domains, using a 12\-model fleet spanning four families\. On the synthetic track MoMHa achieves a joint mean of0\.4820\.482versus0\.1980\.198–0\.4220\.422for ten baselines, winning7/107/10per\-domain columns; on the real\-world track it scores0\.4610\.461versus0\.3770\.377for the strongest baseline \(DSPy\), winning5/75/7columns, demonstrating that harness strategies transfer to unseen benchmarks without retraining on 8 of 12 target models\. MoMHa attains the highest measured behavioral safety composite \(U\-SafeBench,0\.7810\.781\) and uses9595fewer tokens per example than the two\-phase alternative\. We will release all harness code, evaluation infrastructure, and cross\-model logs\.

## 1Introduction

The performance of a large language model on a downstream task depends not only on the model itself but also on the*harness*: the surrounding Python code that constructs prompts, routes calls, parses outputs, and orchestrates multi\-turn interactions\. A well\-designed harness can turn a weak model into a strong solver; a poor one can cripple a frontier model\. Yet harness design remains largely manual, guided by intuition and ad\-hoc experimentation, and, more importantly, almost always framed as a single\-objective problem over accuracy\.

#### Harness quality is not a scalar\.

A harness that answers correctly but happily complies with an unsafe user request is not a good harness; nor is one that reaches the right answer while spending an order of magnitude more tokens than a competitor\. Real deployments trade off accuracy, safety, and cost simultaneously\. Prior work on automatic prompt optimization\([Zhou et al\., 2023](https://arxiv.org/html/2609.30967#bib.bib1);[Pryzant et al\., 2023](https://arxiv.org/html/2609.30967#bib.bib2)\)and DSPy\-style program synthesis\([Khattab et al\., 2024](https://arxiv.org/html/2609.30967#bib.bib3)\)has demonstrated that programmatic search over LLM interaction patterns yields consistent accuracy gains, but these systems collapse feedback into a scalar reward and ignore safety and token objectives almost entirely\.

#### Meta\-Harness\.

We presentMeta\-Harness, a system that treats harness design as explicit multi\-objective search over three per\-domain objectives \(accuracy, behavioural safety, and token cost\) and solves it with an agentic proposer \(Claude Code\) given*full filesystem access*to prior harness source, per\-example execution traces, and scoring artifacts\. Because the proposer emits executable Python, we add a three\-layer safety system: AST\-based static analysis \(AST\-Guard\), domain\-specific safety skills with calibrated import whitelists, and sandbox enforcement\. Harnesses are then evaluated on seventeen domains: seven synthetic capability suites, seven real\-world public benchmarks, and three U\-SafeBench\-derived user\-specific safety domains\([In et al\., 2025](https://arxiv.org/html/2609.30967#bib.bib15)\)\) against a 12\-model fleet spanning Claude, GPT, Gemini, and DeepSeek\.

#### Central finding: joint beats two\-phase\.

The natural way to do multi\-objective search with an agentic proposer is*staged*: first optimize accuracy subject to safety, then in a second phase optimize tokens holding accuracy roughly fixed\. We implement this as 2\-phase and treat it as the strongest ablation\. Our headline system,MoMHa, instead exposes all three objectives to the proposer*simultaneously*in a single phase, and rewards a joint utilityR=acc\+λs​safety−λt​tokensR=\\text\{acc\}\+\\lambda\_\{s\}\\,\\text\{safety\}\-\\lambda\_\{t\}\\,\\text\{tokens\}\. MoMHa achieves the highest 3D hypervolume indicator \(HV=0\.481\\mathrm\{HV\}\{=\}0\.481\) among all 15 variants, a\+0\.119\+0\.119gain over MH \(0\.3620\.362\), which contributes zero unique Pareto volume once safety and token efficiency are added as axes\. MoMHa also outperforms the 2\-phase ablation on77of1010domains by joint metricJJ, with a mean accuracy composite of0\.6110\.611versus0\.5840\.584at9595fewer tokens per example\. The joint formulation lets the proposer make cost\-aware structural choices \(e\.g\. a single confidence\-gated verifier instead of two redundant draft\-verify calls\) that a two\-phase pipeline cannot discover, because Phase 1 has already frozen the structure\.

#### MoMHa versus external baselines\.

The closest baseline isMH\([Lee et al\., 2026](https://arxiv.org/html/2609.30967#bib.bib16)\), the only other system that fully rewrites Python harness code via search, but MH optimizes accuracy only, with no safety or token objectives\. Moving from MH to MoMHa \(i\.e\., the MOO extension\) yields\+0\.177\+0\.177in joint mean J \(0\.4820\.482vs\.0\.3050\.305\) and\+21\.2\+21\.2pp in behavioral safety \(0\.7810\.781vs\.0\.5690\.569\)\. The remaining eight baselines \(CoT, APE, OPRO, DSPy, MIPROv2, TextGrad, GEPA, Rand\) are*prompt\-optimization*methods that fix the harness structure and tune prompts only, a fundamentally different search space\. AH\([Lou et al\., 2026](https://arxiv.org/html/2609.30967#bib.bib17)\)\(AutoHarness\) also emits full Python harnesses but is single\-objective \(accuracy only, no safety signal, no Pareto pool\)\.

Against all ten baselines, MoMHa leads on both evaluation tracks\. On thesynthetic trackMoMHa achieves a ten\-domain joint meanJ=0\.482J\{=\}0\.482versus0\.1980\.198–0\.4220\.422for ten baselines, winning7/107/10per\-domain columns; the nearest rival is TextGrad \(0\.4220\.422,\+0\.060\+0\.060in jointJJ\)\. On thereal\-world trackMoMHa scoresJ=0\.461J\{=\}0\.461versus0\.0840\.084–0\.3770\.377, winning5/75/7columns; the nearest rival is DSPy \(0\.3770\.377,\+0\.084\+0\.084in jointJJ\)\. MoMHa attains*the highest U\-SafeBench behavioral safety composite*\(0\.7810\.781\) across all eleven systems and the second\-highest raw capability accuracy \(0\.5390\.539; DSPy reaches0\.5420\.542but at2,2292\{,\}229tokens per example versus MoMHa’s672672, which explains its lower joint score\)\. No external baseline clears0\.750\.75safety\.

#### Cross\-model generalization\.

Strategies discovered on a small Haiku\-class proposer\-evaluator loop transfer without modification to8 of 12target models; on the remaining 4, GPT\-5\-mini, o4\-mini, and DeepSeek\-R1 prefer DSPy’s terse prompts \(which add fewer harness tokens on top of model\-internal CoT\), while GPT\-5\.4 goes to GEPA \([Section4\.4](https://arxiv.org/html/2609.30967#S4.SS4)\)\. Despite these exceptions, the mean cross\-model joint score for MoMHa \(J=0\.434J\{=\}0\.434\) leads all baselines, and the accuracy ordering between MoMHa and the baselines is stable across model families\.

#### Contributions\.

\(1\) Harness search as explicit multi\-objective optimization\.We cast harness design as multi\-objective search over per\-domain accuracy, safety, and token cost, solved via a scalarized joint reward, with an agentic proposer that reads traces and edits source \([Section3](https://arxiv.org/html/2609.30967#S3)\)\.\(2\) Joint reward beats staged optimization\.Our central empirical result is that a single\-phase joint\-reward proposer \(MoMHa\) outperforms a two\-phase*accuracy\-then\-tokens*proposer \(2\-phase\) by\+2\.7\+2\.7overall points at9595fewer tokens per example \([AppendixK](https://arxiv.org/html/2609.30967#A11)\)\. MoMHa leads all 15 variants on the 3D hypervolume indicator \(HV=0\.481\\mathrm\{HV\}\{=\}0\.481vs\. MH’s0\.3620\.362,[AppendixK](https://arxiv.org/html/2609.30967#A11)\)\.\(3\) Trace\-level feedback is measurable\.Replacing per\-example traces with scalar scores \(MoMHa\-scalar\) drops mean overall from0\.6110\.611to0\.6040\.604and safety from0\.7810\.781to0\.7540\.754, confirming that the proposer extracts causal signal from traces beyond what scores carry \([AppendixK](https://arxiv.org/html/2609.30967#A11)\)\.\(4\) Safety as a first\-class objective\.Removing the domain\-specific safety skill \(MoMHa\-ns\) drops safety from0\.7810\.781to0\.7160\.716while capability accuracy is largely unchanged, isolating the skill’s contribution \([Section4\.5](https://arxiv.org/html/2609.30967#S4.SS5)\)\. All ten external baselines score below0\.750\.75on the U\-SafeBench composite\.\(5\) Seventeen\-domain, twelve\-model evaluation\.Seven synthetic capability domains, seven real\-world benchmarks \(LawBench, NuminaMath, FEVER, Spider, HumanEval, MBPP, MMLU\-Pro\), and three U\-SafeBench safety domains on a 12\-model fleet from four families\.\(6\) Open artifacts\.We release all harness source, search logs, per\-example traces, scoring artifacts, and cross\-model evaluation JSONs\.

#### Key findings\.

Overall:MoMHa leads the synthetic\-track joint mean \(J=0\.482J\{=\}0\.482vs\.0\.1980\.198–0\.4220\.422for ten baselines,\+6\.0\+6\.0over TextGrad\) and the real\-world joint mean \(J=0\.461J\{=\}0\.461vs\.0\.0840\.084–0\.3770\.377,\+7\.9\+7\.9over DSPy\)\.Capability:mean accuracy across seven skill domains improves from0\.4780\.478\(MH\) to0\.5390\.539\(MoMHa\), a\+6\.1\+6\.1\-point gain\.Safety:U\-SafeBench behavioral safety composite improves from0\.5690\.569\(MH\) to0\.7810\.781\(MoMHa\), the highest across all eleven systems; the closest external baseline is TextGrad at0\.7470\.747\.Joint vs\. two\-phase:MoMHa beats 2\-phase by\+2\.7\+2\.7overall points while using9595fewer tokens per example\.Hypervolume:MoMHa attainsHV=0\.481\\mathrm\{HV\}\{=\}0\.481\(highest of 15 variants\); MH scores0\.3620\.362and contributes zero unique Pareto volume once safety and efficiency are added as axes\.

## 2Related Work

Prior prompt optimizers \(APE\([Zhou et al\., 2023](https://arxiv.org/html/2609.30967#bib.bib1)\), OPRO\([Yang et al\., 2024](https://arxiv.org/html/2609.30967#bib.bib7)\), DSPy\([Khattab et al\., 2024](https://arxiv.org/html/2609.30967#bib.bib3)\), MIPROv2\([Opsahl\-Ong et al\., 2024](https://arxiv.org/html/2609.30967#bib.bib13)\), TextGrad\([Yüksekgönül et al\., 2024](https://arxiv.org/html/2609.30967#bib.bib5)\), and GEPA\([Agrawal et al\., 2025](https://arxiv.org/html/2609.30967#bib.bib14)\)\) operate within a*fixed harness skeleton*: they tune prompts, demonstrations, or natural\-language gradients but leave the program’s control flow immutable \(*skill optimization*\)\. MH\([Lee et al\., 2026](https://arxiv.org/html/2609.30967#bib.bib16)\)first proposed rewriting*full Python harnesses*via an agentic proposer \(Claude Code\), but optimizes accuracy only \(single objective\); it reports a Pareto frontier only as a*post\-hoc*plot over single\-objective search runs, whereas MoMHa’s joint scalar reward drives the search itself\. AutoHarness\([Lou et al\., 2026](https://arxiv.org/html/2609.30967#bib.bib17)\)similarly self\-synthesizes a rejection\-sampling wrapper but evaluates only on game\-playing benchmarks without safety or token\-cost objectives\. MoMHa extends this foundation to three\-objective search over capability accuracy, behavioral safety \(U\-SafeBench\([In et al\., 2025](https://arxiv.org/html/2609.30967#bib.bib15)\)\), and token cost, adding a three\-layer safety stack and per\-domain safety skills\.

#### Guardrails, soft prompts, and activation steering\.

Three orthogonal LLM\-safety and adaptation paradigms motivate why harness search is a distinct design surface\.*Guardrail sandwich models*\(I/O classifiers around a base model\) are typified by Llama\-Guard\([Inan et al\., 2023](https://arxiv.org/html/2609.30967#bib.bib27)\), ShieldGemma\([Zeng et al\., 2024](https://arxiv.org/html/2609.30967#bib.bib28)\), and NeMo Guardrails\([Rebedea et al\., 2023](https://arxiv.org/html/2609.30967#bib.bib29)\): they add a pre\-call input filter and \(optionally\) a post\-call output filter, but the model’s prompt, retrieval, retry, and routing behavior is untouched\. MoMHa’s harness is a*superset*: an input classifier is one mutation in the action set \([AppendixO](https://arxiv.org/html/2609.30967#A15)\), coexisting with prompt edits, retrieval augmentation, verification cascades, and confidence\-gated routing\.*Soft\-prompt / prefix tuning*\([Lester et al\., 2021](https://arxiv.org/html/2609.30967#bib.bib30);[Li and Liang, 2021](https://arxiv.org/html/2609.30967#bib.bib31)\)optimizes continuous vectors in the LLM’s embedding space by gradient descent through the model, and therefore requires white\-box weight access \(excluding closed API models\), training compute at optimization time, and produces an opaque vector rather than auditable Python\.*Activation steering / representation engineering*\([Turner et al\., 2023](https://arxiv.org/html/2609.30967#bib.bib32);[Zou et al\., 2023](https://arxiv.org/html/2609.30967#bib.bib33)\)inserts learned directions into intermediate residuals, again a white\-box technique; it is complementary rather than competing, since a discovered MoMHa harness could invoke a steered model as one of its LLM\-client targets\.

Full related work, covering prompt transfer, code\-generation safety, sandboxing, and cross\-model transferability, is in[AppendixA](https://arxiv.org/html/2609.30967#A1)\.

## 3Method

### 3\.1Problem Formulation

A*harness*is a Python class that wraps an LLM client and processes one example at a time\. Formally, a harnesshhimplements a functionh\.run:𝒳→𝒴h\.\\texttt\{run\}:\\mathcal\{X\}\\to\\mathcal\{Y\}that maps an input examplex∈𝒳x\\in\\mathcal\{X\}to a predictiony∈𝒴y\\in\\mathcal\{Y\}by issuing one or more calls to an LLM client\. The harness optimization problem seeks:

h∗=argmaxh∈ℋ1\|𝒟search\|∑x∈𝒟searchr\(h\.run\(x\),x\),h^\{\*\}=\\arg\\max\_\{h\\in\\mathcal\{H\}\}\\;\\tfrac\{1\}\{\|\\mathcal\{D\}\_\{\\mathrm\{search\}\}\|\}\\sum\_\{x\\in\\mathcal\{D\}\_\{\\mathrm\{search\}\}\}r\(h\.\\texttt\{run\}\(x\),x\),\(1\)whereℋ\\mathcal\{H\}is the space of valid harnesses,𝒟search\\mathcal\{D\}\_\{\\mathrm\{search\}\}is a search set of 50 examples \(visible to the proposer\), andrris a domain\-specific reward function\. A held\-out test set𝒟test\\mathcal\{D\}\_\{\\mathrm\{test\}\}\(50 examples, hidden from the proposer\) is used for final evaluation\. Every harness must implement\_\_init\_\_\(self, client, config=None\)andrun\(self, example: dict\) \-\> dict, returning a dictionary containing at minimum a"prediction"key\.

### 3\.2System Architecture

Meta\-Harnessoperates in an iterative propose\-evaluate\-log loop \([Algorithm1](https://arxiv.org/html/2609.30967#alg1)\)\. Each iteration, the proposer inspects the filesystem, diagnoses failures, and writes a new harness candidate\. The evaluator runs the candidate on the search set, logging per\-example scores and execution traces\. Each domain undergoes∼100\{\\sim\}100total harness evaluations \(see[Section4\.1](https://arxiv.org/html/2609.30967#S4.SS1)for the full budget breakdown\)\.

#### Proposer\.

The proposer is an instance ofClaude Codewith filesystem access to: \(a\) all prior harness source files \(domains/<d\>/harnesses/candidate\_\*\.py\); \(b\) per\-run artifacts:scores\.json\(per\-example scores\),traces\.jsonl\(execution traces\),meta\.json\(model, parent, timestamp\),summary\.md; \(c\) the search set \(data/search\_set\.jsonl\); and \(d\) CLI tools:list,top,pareto,diff,show\. The proposer reads files via standard tools \(grep,cat,diff\) rather than ingesting all history into a single prompt, enabling inspection of 80\+ files per iteration without exceeding context limits\.

#### Per\-domain proposer skills\.

The proposer prompt is not monolithic; it is assembled per run bycompose\_skill\(\)from*three*markdown files:

Algorithm 1Meta\-Harnessoptimization loop1:Input:domain

dd, search set

𝒟search\\mathcal\{D\}\_\{\\mathrm\{search\}\}, proposer

PP, evaluator

EE, iterations

NN
2:Initialize filesystem with baseline harness and empty run log

3:for

i=1i=1to

NNdo

4:

hi←P\.propose​\(filesystem\)h\_\{i\}\\leftarrow P\.\\texttt\{propose\}\(\\texttt\{filesystem\}\)// reads∼\\sim82 files

5:ifAST\-Guard

\(hi\)\(h\_\{i\}\)failsthen

6:Log rejection;continue

7:endif

8:

\(𝐬i,𝝉i\)←E\.eval​\(hi,𝒟search\)\(\\mathbf\{s\}\_\{i\},\\boldsymbol\{\\tau\}\_\{i\}\)\\leftarrow E\.\\texttt\{eval\}\(h\_\{i\},\\mathcal\{D\}\_\{\\mathrm\{search\}\}\)
9:Log

hih\_\{i\},

𝐬i\\mathbf\{s\}\_\{i\},

𝝉i\\boldsymbol\{\\tau\}\_\{i\}, metadata

10:endfor

11:Return:

arg⁡maxi⁡R⁡\(𝐬i\)\\arg\\max\_\{i\}R\(\\mathbf\{s\}\_\{i\}\)// joint reward \(Eq\.[3](https://arxiv.org/html/2609.30967#S3.E3)\)

\(i\) abase skillcarrying the search objective and generic rules \(proposer\_skill\.mdfor MoMHa’s single\-phase joint search;proposer\_skill\.mdfollowed byproposer\_token\_skill\.md\); \(ii\) aper\-domain skill\(proposer\_<domain\>\.md\) encoding domain knowledge: a strategy library \(e\.g\. draft\-verify, subject\-aware routing, verification cascades\), answer\-normalization recipes, and common failure modes observed on MH\([Lee et al\., 2026](https://arxiv.org/html/2609.30967#bib.bib16)\)runs; \(iii\) aper\-domain safety skill\(safety\_<domain\>\.md\) giving calibrated import whitelists and output\-format constraints at the domain’s risk level \(strict / minimal / light,[Section3\.3](https://arxiv.org/html/2609.30967#S3.SS3)\)\. This composition means the same infrastructure generalizes across all seventeen domains with only the<<500\-line per\-domain file changing; ablating \(iii\) yields theMoMHa\-nsvariant \([Table10](https://arxiv.org/html/2609.30967#A11.T10)\)\. The per\-domain skill lets the proposer skip the “rediscover domain idioms from traces” step that dominates cold\-start iterations on a new task\.

#### Evaluator\.

The evaluator runs each harness on the search set with domain\-specific reward functions:text classification: exact match against ground truth labels;math reasoning: exact match with numeric normalization;agentic coding: syntax validity plus test\-case execution;MCQ: exact match on option letter;fact verification: exact match on stance label;NER: entity\-level F1 with type matching;SQL generation: query execution plus result comparison; anduser\-specific safety\(three U\-SafeBench domains\): a symmetric LLM\-as\-judge reward0\.5×refusalunsafe\+0\.5×compliancehelpful0\.5\{\\times\}\\text\{refusal\}\_\{\\text\{unsafe\}\}\+0\.5\{\\times\}\\text\{compliance\}\_\{\\text\{helpful\}\}, where the judge classifies each response as*refuse*,*comply*, or*partial*\(half credit\), coupling refusal on profile\-conditioned unsafe instructions with helpfulness on matched benign instructions so the proposer cannot trivially maximize reward by refusing everything\.

#### Logger\.

Every evaluation produces machine\-readable artifacts designed forgrep/cataccess by the proposer:harness\.py\(source snapshot\),scores\.json\(per\-example breakdown \+ aggregate\),traces\.jsonl\(one JSON per LLM call\), andmeta\.json\(provenance\)\.

#### Proposer inner loop and screening \(summary\)\.

Each iteration the proposer reads the prior candidate source, a redacted run history \(accuracy, tokens, safety, and joint rewardRRper candidate\), and per\-example traces, then applies a*single*structural mutation from a fixed action set declared in the skill files \(e\.g\. rewrite the prompt, add or remove a verification/retry loop, change model routing, or prune few\-shot/schema context\)\. Every candidate must clear four gates before it can replace the current best:AST\-Guardrejection, mock\-run validation against aMockLLMClient, a budgeted screening evaluation, and a joint\-reward floor, after which only∼30%\{\\sim\}30\\%of the∼100\{\\sim\}100proposer calls per domain reach the acceptance step, which requires strict improvement onRR\(Eq\. \([3](https://arxiv.org/html/2609.30967#S3.E3)\)\)\. A harness may build, rewrite, or reject prompts, route across models, retry, and parse/redact outputs, but cannot modify model weights, escape the sandbox, or exceedAST\-Guard’s per\-domain whitelist \([Section3\.3](https://arxiv.org/html/2609.30967#S3.SS3)\)\.[AppendixO](https://arxiv.org/html/2609.30967#A15)gives the full state representation, mutation catalogue, and a worked example \(fact verification\) of why single\-phase joint search discovers harness structures that two\-phase search cannot reach\.

### 3\.3Safety Architecture \(MoMHa\)

Because the proposer generates executable code, safety is a first\-class concern\. Our MoMHa architecture introduces three layers of defense:

#### Layer 1: AST\-based static analysis \(AST\-Guard\)\.

Before any harness is executed,AST\-Guardparses the code as a Python AST against a*domain\-parameterized*banned\-symbol list \(the list is tightened or relaxed according to the domain risk level set by Layer 2\)\. Checks include: \(a\)banned imports\(os,subprocess,shutil,socket,requests,urllib,http,ctypes,signal,sys,importlib,asyncio,threading,tempfile,glob,pathlib, and others\); \(b\)banned calls\(exec,eval,compile,\_\_import\_\_,open,input,breakpoint,exit,globals,locals,getattr,setattr,delattr\); \(c\)banned attributes\(system,popen,Popen,check\_output,listdir,rmtree,communicate,urlopen\); and \(d\)structural requirements: must define aHarnessclass with\_\_init\_\_andrunmethods\.

#### Layer 2: Domain\-specific safety skills\.

Each domain has a calibrated safety specification that sets the risk level fed into Layer 1\. Domains are classified as:strict\(agentic coding, SQL generation: tight import whitelists, explicit output format constraints\),minimal\(math reasoning: the AST\-GUARD banned\-calls list is relaxed to permiteval\(\)andcompile\(\)for expression parsing, while all other banned calls remain enforced\), orlight\(text classification, MCQ, fact verification, NER: standard banned imports with permissive allowed lists\)\.

#### Layer 3: Sandbox enforcement\.

All harnesses execute within a sandbox with: 30\-second timeout per call, 100K token limit, 200 API call limit, 300\-second wall clock, empty environment \(PATHonly\), and/tmpas the working directory\.

### 3\.4Multi\-Objective Optimization

Harness quality is inherently multi\-objective: accuracy alone is insufficient if the harness is unsafe or prohibitively expensive\. We formalize the optimization as a search over three axes:

ℱ⁡\(h\)=\(accuracy⁡\(h\),safety⁡\(h\),−tokens⁡\(h\)\),\\mathcal\{F\}\(h\)=\\big\(\\mathrm\{accuracy\}\(h\),\\;\\mathrm\{safety\}\(h\),\\;\-\\mathrm\{tokens\}\(h\)\\big\),\(2\)where the goal is to find the Pareto non\-dominated set that maximizes accuracy and safety while minimizing token consumption\.

#### Joint reward \(MoMHa, headline\)\.

Our headline configuration is a*single\-phase*proposer that receives all three objectives simultaneously and optimizes a scalarized utility

R⁡\(h\)=accuracy⁡\(h\)\+λs​safety​\(h\)−λt​tokens​\(h\),R\(h\)=\\mathrm\{accuracy\}\(h\)\+\\lambda\_\{s\}\\,\\mathrm\{safety\}\(h\)\-\\lambda\_\{t\}\\,\\mathrm\{tokens\}\(h\),\(3\)withλs=1\.0\\lambda\_\{s\}\{=\}1\.0andλt\\lambda\_\{t\}normalized so a 1k\-token reduction is worth 1pp of accuracy\. Crucially, the proposer is*not*compressed down to the scalarRR: it reads the three raw per\-example axes fromscores\.json\(accuracy,safety\_score,tokens\_used\) plus the full trace, and only the search ranking is scalarized\. This lets the proposer see*which*axis a proposed edit moves, and prefer structural rewrites that trade a redundant verification call for a single confidence\-gated one\.

#### Token reduction in the joint formulation\.

Token optimization is*emergent*in MoMHa rather than a separate phase: the base skill instructs the proposer to \(a\) read the per\-exampletokens\_usedcolumn inscores\.jsonto isolate the 20% of examples consuming 80% of the budget; \(b\) walktraces\.jsonlto identify*which*LLM call in a multi\-call pipeline is the dominant sink \(system prompt, CoT completion, verification pass, or fix loop\); and \(c\) propose a targeted structural rewrite, e\.g\., skip a verification call when the draft confidence exceeds a threshold, prune the DDL schema to relevant tables only, or cap fix\-loop iterations after a first\-attempt improvement\. Because this diagnosis happens in the*same iteration*as accuracy and safety feedback, the proposer can discover jointly cost\-aware structures \(e\.g\., a single confidence\-gated verifier instead of two sequential draft\-verify calls\) that are*inexpressible*to a two\-phase system whose harness structure is already frozen before the token phase begins\. The ablation variants \(2\-phase, MoMHa\-ns, MoMHa\-noTok, MoMHa\-scalar\) and per\-domain token sink analysis are in[AppendicesK](https://arxiv.org/html/2609.30967#A11)and[D](https://arxiv.org/html/2609.30967#A4)\. The relationship between the search signalRRand the evaluation metricJJused in all tables is discussed whereJJis defined \([Section4\.3](https://arxiv.org/html/2609.30967#S4.SS3)\)\.

## 4Experiments

### 4\.1Setup

#### Domains\.

We evaluate on seventeen domains across two tracks\.Synthetic track\(ten domains, primary evaluation\): seven capability domains \(text classification, math reasoning, agentic coding, MCQ, fact verification, NER, SQL generation\) and three U\-SafeBench\-derived user\-specific safety domains \(illegal\-activity QA, autonomous\-physical\-harm, autonomous\-mental\-harm\)\([In et al\., 2025](https://arxiv.org/html/2609.30967#bib.bib15)\); each capability domain has 100 LLM\-generated examples \(verified by a second model\), split into 50 search \(visible to the proposer\) and 50 test \(held out\)\.Real\-world track\(seven domains, generalization check\): LawBench, NuminaMath, FEVER, Spider, HumanEval, MBPP, and MMLU\-Pro, drawn from HuggingFace \([Section4\.2](https://arxiv.org/html/2609.30967#S4.SS2)\)\.

#### Models\.

We evaluate discovered harnesses on 12 models spanning four families and four capability tiers \([Table1](https://arxiv.org/html/2609.30967#S4.T1)\)\. Harnesses are discovered using

Table 1:12 models for cross\-model evaluation, grouped by tier\.Claude Haiku 4\.5 and evaluated without modification on all 12 models\. Each domain uses 50 examples for search and a*disjoint*50 examples for test evaluation; the proposer never reads test\-set traces\. The real\-world track \([Section4\.2](https://arxiv.org/html/2609.30967#S4.SS2)\) provides an additional out\-of\-distribution generalization check since those benchmarks were not used in the search phase\.

#### Harness search budget\.

Each of the seven synthetic capability domains undergoes∼100\{\\sim\}100harness evaluations \(range: 98–106\), totaling 724 evaluations across those seven domains; the three U\-SafeBench safety domains use a smaller budget \(∼25\{\\sim\}25evaluations each\) given their narrower action space\. The seven real\-world benchmark domains \([Section4\.2](https://arxiv.org/html/2609.30967#S4.SS2)\) incur*zero*search cost: they are evaluated using the harnesses already discovered on the synthetic track, making them a fully held\-out generalization test\. The proposer reads a median of 82 files per iteration\.Cost note:this makes each domain search substantially more proposer\-API\-intensive than prompt\-optimization baselines \(APE, OPRO, DSPy, TextGrad\) which only call the proposer model once per candidate with a short meta\-prompt; a full API\-cost comparison is deferred to future work\.

#### Baselines\.

We compare MoMHa against ten alternatives split into three tiers\.

Harness\-search baseline \(primary\):MH\([Lee et al\., 2026](https://arxiv.org/html/2609.30967#bib.bib16)\)uses the same search infrastructure as MoMHa \(full Python harness rewriting via an LLM proposer\) but optimizes accuracy only \(single\-objective, no safety or token objectives\)\. It is our primary point of comparison: all improvement of MoMHa over MH is attributable to the multi\-objective extension\.

Prompt\-optimization baselines:CoT\(chain\-of\-thought zero\-shot\),APE\([Zhou et al\., 2023](https://arxiv.org/html/2609.30967#bib.bib1)\)\(Monte\-Carlo resampling around top\-KKprompts, 5 rounds×\\times5 candidates\),OPRO\([Yang et al\., 2024](https://arxiv.org/html/2609.30967#bib.bib7)\)\(ascending\-sorted \(prompt, score\) trajectory \+ in\-context exemplars as a meta\-prompt; same budget\),DSPy\([Khattab et al\., 2024](https://arxiv.org/html/2609.30967#bib.bib3)\)\(ChainOfThought/Predict with BootstrapFewShot\),MIPROv2\([Opsahl\-Ong et al\., 2024](https://arxiv.org/html/2609.30967#bib.bib13)\)\(Bayesian instruction proposal optimization, selects among diverse LLM\-generated candidates via a Parzen\-Estimator surrogate\),TextGrad\([Yüksekgönül et al\., 2024](https://arxiv.org/html/2609.30967#bib.bib5)\)\(natural\-language feedback backpropagation\),GEPA\([Agrawal et al\., 2025](https://arxiv.org/html/2609.30967#bib.bib14)\)\(Genetic\-Pareto reflective prompt evolution, maintains an instance\-wise Pareto front of candidates\), andRand\(random composition of a 10\-strategy library\)\. These methods optimize prompts within a fixed harness skeleton, a qualitatively different search space from MoMHa/MH’s full Python rewriting\.

Self\-synthesized harness baseline:AH\([Lou et al\., 2026](https://arxiv.org/html/2609.30967#bib.bib17)\)\(AutoHarness\) lets the LLM iteratively write and refine its own Python harness as a rejection sampler over its own outputs\. Like MoMHa it emits full Python code rather than prompts alone, but unlike MoMHa it is single\-objective \(accuracy\), has no safety signal, and does not maintain a Pareto pool\. All baselines are evaluated cross\-model on the same 12\-model fleet\. The MoMHa\-family ablations are analysed in[AppendixK](https://arxiv.org/html/2609.30967#A11)\. MIPROv2 and GEPA represent two advances in automated prompt optimization from the DSPy ecosystem\. GEPA is the closest baseline in spirit toMeta\-Harness: both use natural\-language reflection over execution traces to propose improvements, and both maintain a Pareto pool rather than a greedy global best\. The key differences are \(i\)*surface of search*: both MIPROv2 and GEPA edit only the module’s instruction string, whereasMeta\-Harnessedits arbitrary harness Python code \(retries, routing, schema pruning, etc\.\); \(ii\)*objectives*: GEPA’s Pareto front is over*task instances*for diversity, whereasMeta\-Harness’s Pareto front is over*objectives*\(accuracy, safety, tokens\); and \(iii\)*safety*: none of MIPROv2, TextGrad, GEPA, APE, or OPRO hasAST\-Guard, safety skill composition, or sandboxed execution, and their measured U\-SafeBench safety composites \(0\.6410\.641,0\.7470\.747,0\.6690\.669,0\.6370\.637,0\.5730\.573respectively\) fall below MoMHa’s behavioral safety composite of0\.7810\.781\(U\-SafeBench\)\.

### 4\.2Real\-World Benchmark Generalization

We evaluate MoMHa and all baselines on seven real\-world public benchmarks from HuggingFace: LawBench \(TC\), NuminaMath \(MathR\), FEVER \(FVFEV\), Spider \(SQLSp\), HumanEval, MBPP, and MMLU\-Pro\. For each \(variant, domain\) cell we run a 12\-model cross\-model sweep and compute the joint scoreJv,d=acc⋅Safev/ln⁡\(tokens\)J\_\{v,d\}=\\mathrm\{acc\}\\cdot\\mathrm\{Safe\}\_\{v\}/\\ln\(\\mathrm\{tokens\}\)\([Equation4](https://arxiv.org/html/2609.30967#S4.E4)\), whereSafev\\mathrm\{Safe\}\_\{v\}is each variant’s mean accuracy across the three U\-SafeBench safety domains\.[Table2](https://arxiv.org/html/2609.30967#S4.T2)reports the result\.MoMHa wins55of77per\-domain columns and leads the 7\-domain mean by\+0\.084\+0\.084points over the strongest baseline \(DSPy,0\.377→0\.4610\.377\\to 0\.461\)\. TextGrad retains TC \(LawBench\) and MMLU\-Pro but collapses on MBPP \(0\.0680\.068\); DSPy is the only non\-MoMHa baseline with a non\-trivial MBPP cell\.

Table 2:Joint per\-domain score on real\-world benchmarksJv,dJ\_\{v,d\}\([Equation4](https://arxiv.org/html/2609.30967#S4.E4)\)\.Bold= column winner;underline= runner\-up\. MoMHa wins5/75/7per\-domain columns and the 7\-domain mean\. Cells show mean±SEM\{\}\_\{\\pm\\text\{SEM\}\}withSEM=σcross\-model/Ntrials\\text\{SEM\}=\\sigma\_\{\\text\{cross\-model\}\}/\\sqrt\{N\_\{\\text\{trials\}\}\}andNtrials≈11×50N\_\{\\text\{trials\}\}\\\!\\approx\\\!11\\times 50per cell\.
### 4\.3Main Results: Synthetic Evaluation

To compare baselines and MoMHa across the ten synthetic domains, we use the same joint per\-domain score:

Jv,d=accv,d⋅Safevln⁡\(tokensv,d\)J\_\{v,d\}\\;=\\;\\tfrac\{\\mathrm\{acc\}\_\{v,d\}\\cdot\\mathrm\{Safe\}\_\{v\}\}\{\\ln\(\\mathrm\{tokens\}\_\{v,d\}\)\}\(4\)whereaccv,d\\mathrm\{acc\}\_\{v,d\}is the mean accuracy of variantvvon domainddacross the 12\-model fleet,Safev\\mathrm\{Safe\}\_\{v\}is the variant’s U\-SafeBench safety composite \(mean accuracy on the three safety domains\), andtokensv,d\\mathrm\{tokens\}\_\{v,d\}is the mean per\-example token cost\. This composite aggregates three axes into a single leaderboard column, following the paradigm of holistic multi\-metric LLM evaluation\([Liang et al\., 2022](https://arxiv.org/html/2609.30967#bib.bib24)\)\. Theln⁡\(⋅\)\\ln\(\\cdot\)denominator is a gentle token penalty: moving from 500 to 2000 tokens shrinks the score by only∼\\sim20%\([Du et al\., 2025](https://arxiv.org/html/2609.30967#bib.bib25);[Kaiser et al\., 2026](https://arxiv.org/html/2609.30967#bib.bib26)\)\.

Metric design note:Safev\\mathrm\{Safe\}\_\{v\}is a variant\-level constant, not domain\-specific, so it acts as a multiplicative safety bonus on every capability cell, structurally rewarding the variant with higher behavioral safety on all domains including unrelated ones \(e\.g\. MBPP\)\. Per\-cell raw accuracy, safety, and token numbers are in[AppendixE](https://arxiv.org/html/2609.30967#A5), allowing readers to evaluate domain\-specific task performance independently of the global safety factor\.

Relationship to search signalRR:MoMHa optimizesR⁡\(h\)=acc\+λs​safety−λt​tokensR\(h\)=\\mathrm\{acc\}\+\\lambda\_\{s\}\\,\\mathrm\{safety\}\-\\lambda\_\{t\}\\,\\mathrm\{tokens\}\(Eq\.[3](https://arxiv.org/html/2609.30967#S3.E3)\) during search, notJJ\.RRuses a*linear*token penalty so the LLM proposer can reason about a concrete trade\-off \(“a 1k\-token reduction is worth 1pp of accuracy”\);JJreplaces this withln⁡\(tokens\)\\ln\(\\mathrm\{tokens\}\)for*cross\-domain aggregation*, since a linear penalty would be dominated by high\-token domains\. Crucially, both metrics rank harnesses in the same Pareto direction: more accurate, safer, and cheaper is better on both, so a harness selected by maximizingRRduring search also scores well onJJat evaluation\. The empirical correlation betweenRR\-rankings andJJ\-rankings across the MoMHa\-family variants is confirmed in[AppendixK](https://arxiv.org/html/2609.30967#A11)\.

Table 3:Joint per\-domain scoreJv,d=acc⋅Safe/ln⁡\(tokens\)J\_\{v,d\}=\\mathrm\{acc\}\\cdot\\mathrm\{Safe\}/\\ln\(\\mathrm\{tokens\}\)\([Equation4](https://arxiv.org/html/2609.30967#S4.E4)\)\. Each cell folds accuracy, safety, and token cost into one number\.Bold= column winner;underline= runner\-up\.MoMHa wins77of1010per\-domain columnsand leads the 10\-domain mean by\+0\.060\+0\.060points over the strongest baseline \(TextGrad\)\. DSPy’s22292229\-token capability harnesses are heavily penalized by theln⁡\(tokens\)\\ln\(\\mathrm\{tokens\}\)denominator\. Cells show mean±SEM\{\}\_\{\\pm\\text\{SEM\}\}withSEM=σcross\-model/Ntrials\\text\{SEM\}=\\sigma\_\{\\text\{cross\-model\}\}/\\sqrt\{N\_\{\\text\{trials\}\}\},Ntrials≈12×50N\_\{\\text\{trials\}\}\\\!\\approx\\\!12\\times 50per cell\.Under the joint metric MoMHa takes column wins on TC, Code, MCQ, NER, SQL, Safephys, and Safement; DSPy retains Math and TextGrad retains FV and Safeill\. The two closest baselines, DSPy \(strongest on capability at0\.5420\.542\) and TextGrad \(strongest on safety at0\.7470\.747\), fall behind once tokens enter each cell: DSPy’s capability harnesses average22292229tokens versus MoMHa’s672672, and TextGrad’s two column wins do not outweigh MoMHa’s wider per\-domain dominance\. OPRO \(0\.2830\.283\) and AH \(0\.1980\.198\) are the joint\-metric laggards: OPRO’s ascending trajectory meta\-prompt produced prompts with the second\-worst safety composite \(0\.5730\.573\) and the weakest SQL cell \(0\.0580\.058\) among prompt optimizers, while AH produces structurally minimal harnesses that score well on safety J cells \(low token cost\) but collapse on capability domains\. This confirms that prompt\-level optimization alone cannot recover safety at the U\-SafeBench frontier, and that single\-objective harness rewriting without a Pareto pool cannot transfer across diverse domain types\. The per\-domain win count is the headline take\-away: the joint metric reveals that*MoMHa’s advantage is not concentrated in a few structural domains but distributed across the evaluation suite once cost is priced in*\.

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/baselines_pareto_3d_surface.png)\(a\)3D Pareto surfaces \(accuracy×\\timessafety×\\timestokens\)\. MoMHa \(green\) occupies the high\-accuracy, high\-safety, low\-token region\.
![Refer to caption](https://arxiv.org/html/2609.30967v1/img/spider_global/multi_objective_global_3axis_K10.png)\(b\)Global three\-axis spider \(accuracy×\\timessafety×\\timestoken efficiency\)\. MoMHa is the only variant covering all three axes\.
![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/spider_composite.png)\(c\)Per\-domain search\-reward compositeJ⁡\(d\)J\(d\)across all 17 axes\. Synthetic axes carry the\-synsuffix\.

Figure 1:Multi\-objective analysis across eleven variants\.\(a\)Non\-MoMHa baselines span a safety band of0\.570\.57–0\.750\.75; TextGrad alone approaches MoMHa’s safety but at lower accuracy\.\(b\)MoMHa covers all three axes; baselines collapse on at least one\.\(c\)Per\-axis search\-reward: MoMHa is the outer envelope on the seven real\-world capability axes and on the two autonomous\-harm safety axes\. Points in \(a,b\) are aggregated over all 17 domains×\\times12 models; per\-domain Pareto panels in[AppendixJ](https://arxiv.org/html/2609.30967#A10)\.
### 4\.4Cross\-Model Generalization

A central question is whether harnesses optimized on one model transfer to others, and whether this transfer holds not just against the MH baseline but against the full field of prior prompt\-optimization methods\.[Table4](https://arxiv.org/html/2609.30967#S4.T4)reports the mean joint rewardJJper model per variant, aggregated over ten domains \(seven hard synthetic \+ three U\-SafeBench safety\)\. Pipelines are discovered on Claude Haiku 4\.5 and then evaluated on 12 target models without modification\.

Table 4:Cross\-model transfer: mean joint rewardJJacross 12 target models and 10 domains\. Each cell is the mean ofJ=acc​Safev/ln⁡\(tokens\)J\{=\}\\mathrm\{acc\}\\,\\mathrm\{Safe\}\_\{v\}/\\ln\(\\mathrm\{tokens\}\)per \(variant, model\)\. MoMHa \(ours\) wins on 8 of 12 models;bold= best per row,underline= runner\-up; “Ours” col\. = MoMHa \(ours\)\. Cells show mean±SEM\{\}\_\{\\pm\\text\{SEM\}\}withSEM=σcross\-domain/Ntrials\\text\{SEM\}=\\sigma\_\{\\text\{cross\-domain\}\}/\\sqrt\{N\_\{\\text\{trials\}\}\},Ntrials≈10×50N\_\{\\text\{trials\}\}\\\!\\approx\\\!10\\times 50per cell\.MoMHa is the strongest variant on 8 of 12 target models and has the highest cross\-model meanJJ\(0\.4340\.434\), exceeding the next\-best baseline DSPy \(0\.3840\.384\) by\+0\.050\+0\.050and the MH baseline \(0\.2790\.279\) by\+0\.155\+0\.155\. The gain is driven more by the safety axis than by accuracy: across the 12 models, MoMHa improves mean safety over MH by a wide margin while task accuracy improves modestly, and this holds across Anthropic, OpenAI, Google, and open\-weight \(DeepSeek\-R1\) model families\. The four cells where MoMHa is not the best split cleanly: GPT\-5\.4 goes to GEPA \(\.502\.502vs\.453\.453\), while GPT\-5\-mini, o4\-mini, and DeepSeek\-R1 all go to DSPy\. These are all reasoning\-trained targets whose models internally generate long chain\-of\-thought sequences regardless of harness instructions, inflating theln⁡\(tokens\)\\ln\(\\mathrm\{tokens\}\)denominator far beyond the harness\-prompt contribution\. DSPy’s lightweightChainOfThought/Predictsignatures add fewer harness tokens on top of this fixed reasoning budget, so theJJratio tilts in DSPy’s favour even though its absolute capability accuracy is comparable to MoMHa on these models\.

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/hypervolume/hv_bars_vs_baselines_final.png)Figure 2:3D hypervolume indicator \(per\-model HV over capability accuracy×\\timesbehavioral safety×\\timestoken efficiency, pymoo\([Blank and Deb, 2020](https://arxiv.org/html/2609.30967#bib.bib18)\)\)\. MoMHa \(ours\) leads all 10 external baselines \(HV=0\.481\\mathrm\{HV\}\{=\}0\.481\)\.On the non\-reasoning targets \(8/12\), MoMHa’s richer harness recovers a consistent\+0\.05\+0\.05to\+0\.10\+0\.10margin over the closest baseline\. Per\-domain breakouts, intra\-family ablations, and feedback granularity ablations are in[AppendicesE](https://arxiv.org/html/2609.30967#A5),[K](https://arxiv.org/html/2609.30967#A11)and[L](https://arxiv.org/html/2609.30967#A12)\. Total search/evaluation token cost, a proposer\-model ablation, and robustness of the headline ranking to alternative scalarizations and a skill\-file ablation are reported in[AppendixN](https://arxiv.org/html/2609.30967#A14)\.

### 4\.5User\-Specific Safety Domains

To test whetherMeta\-Harnessgeneralizes beyond code\-level safety \(sandboxing, banned imports, AST guards\) to*behavioral*safety, we add three domains drawn from U\-SafeBench\([In et al\., 2025](https://arxiv.org/html/2609.30967#bib.bib15)\):safety\_qa\_illegal\(QA×\\timesillegal activity assist, 1,118 source unsafe pairs\),safety\_autonomous\_physical\(autonomous×\\timesphysical health risk, 900 pairs\), andsafety\_autonomous\_mental\(autonomous×\\timesmental health risk, 216 pairs\)\. Each domain pairs an*unsafe*split \(instructions whose safety depends on a user’s disclosed condition or role, e\.g\. “buy a bottle of Patron” with “I have hepatitis”\) against a*helpful*split \(the same action class with an “ordinary member of the general population” profile\)\. The reward is the symmetric0\.5×refusalunsafe\+0\.5×compliancehelpful0\.5\{\\times\}\\text\{refusal\}\_\{\\text\{unsafe\}\}\+0\.5\{\\times\}\\text\{compliance\}\_\{\\text\{helpful\}\}, computed by a Claude Sonnet 4\.6 LLM\-as\-judge\. Unlike the capability domains, the optimization target here is not accuracy on one split but the*joint*refusal/helpfulness frontier\. The discovered MoMHa harness converges on aprofile\-first routing strategy: a single classification call tags the profile as benign\-general, condition\-bearing, or crisis\-adjacent, and the response stage is gated on the tag plus an action\-category match \(e\.g\. “purchase alcohol”×\\times“hepatitis”→\\torefuse\-with\-alternative\)\. Critically, MoMHa outperforms the CoT baseline on*both*splits simultaneously: CoT often drifts into boilerplate “please consult a professional” refusals that score as refuse on unsafe but also trigger on benign helpful instructions, capping compliance\. The gain is largest onsafety\_autonomous\_mental, where affect\-aware profile reading is essential and lexical cues fail\.

## 5Conclusion, Future Work and Limitations

We showed that LLM harness design is a meaningful multi\-objective optimization problem, and that treating it as one pays off\.MoMHaoutperforms all ten external baselines on the synthetic track \(J=0\.482J\{=\}0\.482vs\.0\.1980\.198–0\.4220\.422\) and on seven held\-out real\-world benchmarks \(J=0\.461J\{=\}0\.461vs\.0\.3770\.377,*zero additional search cost*\), achieves the highest behavioral safety composite among all eleven systems \(0\.7810\.781, U\-SafeBench\), and attains the highest 3D hypervolume \(HV=0\.481\\mathrm\{HV\}\{=\}0\.481\)\. The central finding, that a single\-phase joint reward beats staged two\-phase optimization at 95 fewer tokens per example, demonstrates that token efficiency and safety are best discovered*together*with accuracy, not in sequence\. Deeper results and insights are in[AppendixB](https://arxiv.org/html/2609.30967#A2)\. In future, we want to experiment with \(i\) SWE\-Bench / LiveCodeBench beyond agentic\-coding; \(ii\) richer objectives \(latency, calibration, monetary cost\); \(iii\) proposer ensembles; \(iv\) online harness refinement on production traffic\.

#### Limitations\.

The search phase uses LLM\-generated datasets \(50 examples/domain\); the seven real\-world benchmark wins \([Section4\.2](https://arxiv.org/html/2609.30967#S4.SS2)\) already close this gap for established benchmarks\. Each domain requires∼100\{\\sim\}100proposer API calls; the resulting total token cost is comparable to APE and lower than 2\-phase and DSPy, not a cost outlier \([SectionN\.1](https://arxiv.org/html/2609.30967#A14.SS1)\)\. Transfer degrades on reasoning\-heavy models \(GPT\-5\-mini, o4\-mini, DeepSeek\-R1\) that generate long internal CoT regardless of harness instructions\. A proposer\-model ablation on three representative domains shows the search process is largely proposer\-agnostic \([SectionN\.2](https://arxiv.org/html/2609.30967#A14.SS2)\), and the headline ranking is robust to 8 alternative scalarizations and a skill\-file ablation \([SectionsN\.3](https://arxiv.org/html/2609.30967#A14.SS3)and[N\.4](https://arxiv.org/html/2609.30967#A14.SS4)\)\.

## References

- Agrawalet al\.\(2025\)L\. A\. Agrawal, S\. Tan, D\. Soylu, N\. Ziems, R\. Khare, K\. Opsahl\-Ong, A\. Singhvi, H\. Shandilya, M\. J\. Ryan, M\. Jiang, C\. Potts, K\. Sen, A\. G\. Dimakis, I\. Stoica, D\. Klein, M\. Zaharia, and O\. KhattabGEPA: reflective prompt evolution can outperform reinforcement learning\.arXiv preprint arXiv:2507\.19457\.Note:ICLR 2026 OralCited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2609.30967#S2.p1.1),[§4\.1](https://arxiv.org/html/2609.30967#S4.SS1.SSS0.Px4.p3.1)\.
- Alizadehet al\.\(2024\)S\. Alizadeh, A\. Bhargava, K\. Gopalswamy, L\. Jain, B\. Kveton, and G\. LiuPessimistic off\-policy multi\-objective optimization\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 2980–2988\.Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px7.p1.1)\.
- Blank and Deb \(2020\)J\. Blank and K\. DebPymoo: multi\-objective optimization in Python\.IEEE Access8,pp\. 89497–89509\.Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px7.p1.1),[Figure 16](https://arxiv.org/html/2609.30967#A11.F16),[Figure 16](https://arxiv.org/html/2609.30967#A11.F16.5.1),[Appendix K](https://arxiv.org/html/2609.30967#A11.SS0.SSS0.Px4.p1.1),[Figure 2](https://arxiv.org/html/2609.30967#S4.F2.3),[Figure 2](https://arxiv.org/html/2609.30967#S4.F2.5)\.
- Chenget al\.\(2024\)C\. Cheng, A\. Nie, and A\. SwaminathanTrace is the next autodiff: generative optimization with rich feedback, execution traces, and LLMs\.arXiv preprint arXiv:2406\.16218\.Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px2.p1.1)\.
- Debet al\.\(2002\)K\. Deb, A\. Pratap, S\. Agarwal, and T\. MeyarivanA fast and elitist multiobjective genetic algorithm: NSGA\-II\.IEEE Transactions on Evolutionary Computation6\(2\),pp\. 182–197\.External Links:[Document](https://dx.doi.org/10.1109/4235.996017)Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px7.p1.1)\.
- Duet al\.\(2025\)Z\. Du, H\. Kang, S\. Han, T\. Krishna, and L\. ZhuOckBench: tokens are not to be multiplied without necessity\.InNeurIPS 2025 Workshop on Efficient Reasoning,Cited by:[§4\.3](https://arxiv.org/html/2609.30967#S4.SS3.p1.2)\.
- Inet al\.\(2025\)Y\. In, W\. Kim, S\. Yoon, S\. Kim, M\. Tanjim, S\. Kim, and C\. ParkIs safety standard same for everyone? User\-Specific safety evaluation of large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2025,Note:arXiv preprint arXiv:2502\.15086Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2609.30967#S1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2609.30967#S2.p1.1),[§4\.1](https://arxiv.org/html/2609.30967#S4.SS1.SSS0.Px1.p1.1),[§4\.5](https://arxiv.org/html/2609.30967#S4.SS5.p1.1)\.
- Inanet al\.\(2023\)H\. Inan, K\. Upasani, J\. Chi, R\. Rungta, K\. Iyer, Y\. Mao, M\. Tontchev, Q\. Hu, B\. Fuller, D\. Testuggine, and M\. KhabsaLlama guard: LLM\-based input\-output safeguard for human\-AI conversations\.arXiv preprint arXiv:2312\.06674\.Cited by:[§2](https://arxiv.org/html/2609.30967#S2.SS0.SSS0.Px1.p1.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. NarasimhanSWE\-bench: can language models resolve real\-world GitHub issues?\.arXiv preprint arXiv:2310\.06770\.Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px4.p1.1)\.
- Kaiseret al\.\(2026\)D\. Kaiser, A\. Frigessi, A\. Ramezani\-Kebrya, and B\. RicaudDecomposing reasoning efficiency in large language models\.arXiv preprint arXiv:2602\.09805\.Cited by:[§4\.3](https://arxiv.org/html/2609.30967#S4.SS3.p1.2)\.
- Khattabet al\.\(2024\)O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Vardhamanan, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Mober,et al\.DSPy: compiling declarative language model calls into self\-improving pipelines\.arXiv preprint arXiv:2310\.03714\.Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2609.30967#S1.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.30967#S2.p1.1),[§4\.1](https://arxiv.org/html/2609.30967#S4.SS1.SSS0.Px4.p3.1)\.
- Leeet al\.\(2026\)Y\. Lee, R\. Nair, Q\. Zhang, K\. Lee, O\. Khattab, and C\. FinnMeta\-harness: end\-to\-end optimization of model harnesses\.arXiv preprint arXiv:2603\.28052\.Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px4.p1.1),[2nd item](https://arxiv.org/html/2609.30967#A2.I1.i2.p1.1),[4th item](https://arxiv.org/html/2609.30967#A2.I1.i4.p1.1),[§1](https://arxiv.org/html/2609.30967#S1.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2609.30967#S2.p1.1),[§3\.2](https://arxiv.org/html/2609.30967#S3.SS2.SSS0.Px2.p2.1),[§4\.1](https://arxiv.org/html/2609.30967#S4.SS1.SSS0.Px4.p2.1)\.
- Lesteret al\.\(2021\)B\. Lester, R\. Al\-Rfou, and N\. ConstantThe power of scale for parameter\-efficient prompt tuning\.arXiv preprint arXiv:2104\.08691\.Note:EMNLP 2021Cited by:[§2](https://arxiv.org/html/2609.30967#S2.SS0.SSS0.Px1.p1.1)\.
- Li and Liang \(2021\)X\. L\. Li and P\. LiangPrefix\-tuning: optimizing continuous prompts for generation\.arXiv preprint arXiv:2101\.00190\.Cited by:[§2](https://arxiv.org/html/2609.30967#S2.SS0.SSS0.Px1.p1.1)\.
- Lianget al\.\(2022\)P\. Liang, R\. Bommasani, T\. Lee, D\. Tsipras, D\. Soylu, M\. Yasunaga, Y\. Zhang, D\. Narayanan, Y\. Wu, A\. Kumar,et al\.Holistic evaluation of language models\.arXiv preprint arXiv:2211\.09110\.Cited by:[§4\.3](https://arxiv.org/html/2609.30967#S4.SS3.p1.2)\.
- Louet al\.\(2026\)X\. Lou, M\. Lázaro\-Gredilla, A\. Dedieu, C\. Wendelken, W\. Lehrach, and K\. P\. MurphyAutoHarness: improving llm agents by automatically synthesizing a code harness\.arXiv preprint arXiv:2603\.03329\.Cited by:[§1](https://arxiv.org/html/2609.30967#S1.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2609.30967#S2.p1.1),[§4\.1](https://arxiv.org/html/2609.30967#S4.SS1.SSS0.Px4.p4.1)\.
- Merrillet al\.\(2026\)M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi,et al\.Terminal\-bench: benchmarking agents on hard, realistic tasks in command line interfaces\.InICLR 2026,Note:arXiv preprint arXiv:2601\.11868Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px4.p1.1)\.
- Mukherjee and Lalit \(2024\)S\. Mukherjee and A\. LalitMulti\-objective alignment of large language models through hypervolume maximization\.arXiv preprint arXiv:2412\.05469\.Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px7.p1.1)\.
- Opsahl\-Onget al\.\(2024\)K\. Opsahl\-Ong, M\. J\. Ryan, J\. Purtell, D\. Broman, C\. Potts, M\. Zaharia, and O\. KhattabOptimizing instructions and demonstrations for multi\-stage language model programs\.arXiv preprint arXiv:2406\.11695\.Note:EMNLP 2024Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2609.30967#S2.p1.1),[§4\.1](https://arxiv.org/html/2609.30967#S4.SS1.SSS0.Px4.p3.1)\.
- Pearceet al\.\(2022\)H\. Pearce, B\. Ahmad, B\. Tan, B\. Dolan\-Gavitt, and R\. KarriAsleep at the keyboard? assessing the security of GitHub Copilot’s code contributions\.IEEE Symposium on Security and Privacy\.Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px5.p1.1)\.
- Perezet al\.\(2021\)E\. Perez, D\. Kiela, and K\. ChoTrue few\-shot learning with language models\.Advances in Neural Information Processing Systems\.Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px6.p1.1)\.
- Pryzantet al\.\(2023\)R\. Pryzant, D\. Iter, J\. Li, Y\. T\. Lee, C\. Zhu, and M\. ZengAutomatic prompt optimization with “gradient descent” and beam search\.arXiv preprint arXiv:2305\.03495\.Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.30967#S1.SS0.SSS0.Px1.p1.1)\.
- Rebedeaet al\.\(2023\)T\. Rebedea, R\. Dinu, M\. N\. Sreedhar, C\. Parisien, and J\. CohenNeMo guardrails: a toolkit for controllable and safe LLM applications with programmable rails\.arXiv preprint arXiv:2310\.10501\.Note:EMNLP 2023 Demo TrackCited by:[§2](https://arxiv.org/html/2609.30967#S2.SS0.SSS0.Px1.p1.1)\.
- Ruanet al\.\(2024\)Y\. Ruan, H\. Dong, A\. Wang, S\. Pitis, Y\. Zhou, J\. Ba, Y\. Dubois, C\. J\. Maddison, and T\. HashimotoToolEmu: identifying the risks of LLM agents with an LLM emulator\.arXiv preprint arXiv:2309\.15817\.Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px5.p1.1)\.
- Shinet al\.\(2020\)T\. Shin, Y\. Razeghi, R\. L\. Logan IV, E\. Wallace, and S\. SinghAutoPrompt: eliciting knowledge from language models with automatically generated prompts\.arXiv preprint arXiv:2010\.15980\.Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px1.p1.1)\.
- Turneret al\.\(2023\)A\. M\. Turner, L\. Thiergart, G\. Leech, D\. Udell, J\. J\. Vazquez, U\. Mini, and M\. MacDiarmidSteering language models with activation engineering\.arXiv preprint arXiv:2308\.10248\.Cited by:[§2](https://arxiv.org/html/2609.30967#S2.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2024\)C\. Yang, X\. Wang, Y\. Lu, H\. Liu, Q\. V\. Le, D\. Zhou, and X\. ChenLarge language models as optimizers\.arXiv preprint arXiv:2309\.03409\.Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.30967#S2.p1.1),[§4\.1](https://arxiv.org/html/2609.30967#S4.SS1.SSS0.Px4.p3.1)\.
- Yüksekgönület al\.\(2024\)M\. Yüksekgönül, F\. Bianchi, J\. Boen, S\. Liu, P\. Lu, Z\. Huang, C\. Guestrin, and J\. ZouTextGrad: automatic “differentiation” via text\.arXiv preprint arXiv:2406\.07496\.Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2609.30967#S2.p1.1),[§4\.1](https://arxiv.org/html/2609.30967#S4.SS1.SSS0.Px4.p3.1)\.
- Zenget al\.\(2024\)W\. Zeng, Y\. Liu, R\. Mullins, L\. Peran, J\. Fernandez, H\. Harkous, K\. Narasimhan, D\. Proud, P\. Kumar, B\. Radharapu, O\. Sturman, and O\. WahltinezShieldGemma: generative AI content moderation based on gemma\.arXiv preprint arXiv:2407\.21772\.Cited by:[§2](https://arxiv.org/html/2609.30967#S2.SS0.SSS0.Px1.p1.1)\.
- Zhang and Li \(2007\)Q\. Zhang and H\. LiMOEA/D: a multiobjective evolutionary algorithm based on decomposition\.IEEE Transactions on Evolutionary Computation11\(6\),pp\. 712–731\.External Links:[Document](https://dx.doi.org/10.1109/TEVC.2007.892759)Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px7.p1.1)\.
- Zhouet al\.\(2023\)Y\. Zhou, A\. I\. Muresanu, Z\. Han, K\. Paster, S\. Pitis, H\. Chan, and J\. BaLarge language models are human\-level prompt engineers\.arXiv preprint arXiv:2211\.01910\.Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.30967#S1.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.30967#S2.p1.1),[§4\.1](https://arxiv.org/html/2609.30967#S4.SS1.SSS0.Px4.p3.1)\.
- Zitzler and Thiele \(1999\)E\. Zitzler and L\. ThieleMultiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach\.IEEE Transactions on Evolutionary Computation3\(4\),pp\. 257–271\.External Links:[Document](https://dx.doi.org/10.1109/4235.797969)Cited by:[Appendix A](https://arxiv.org/html/2609.30967#A1.SS0.SSS0.Px7.p1.1)\.
- Zouet al\.\(2023\)A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika, A\. Dombrowski, S\. Goel, N\. Li, M\. J\. Byun, Z\. Wang, A\. Mallen, S\. Basart, S\. Koyejo, D\. Song, M\. Fredrikson, J\. Z\. Kolter, and D\. HendrycksRepresentation engineering: a top\-down approach to AI transparency\.arXiv preprint arXiv:2310\.01405\.Cited by:[§2](https://arxiv.org/html/2609.30967#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix Overview

## Appendix ARelated Work

#### Prompt optimization\.

Automatic prompt engineering has progressed from gradient\-guided search\[[Shin et al\., 2020](https://arxiv.org/html/2609.30967#bib.bib6)\]to LLM\-based proposal systems\[[Zhou et al\., 2023](https://arxiv.org/html/2609.30967#bib.bib1),[Pryzant et al\., 2023](https://arxiv.org/html/2609.30967#bib.bib2)\]\. APE\[[Zhou et al\., 2023](https://arxiv.org/html/2609.30967#bib.bib1)\]uses Monte\-Carlo resampling around the current top\-KKprompts, treating the proposer as a black\-box generator and selection as a greedy bandit\. OPRO\[[Yang et al\., 2024](https://arxiv.org/html/2609.30967#bib.bib7)\]instead frames each call to the proposer as a meta\-prompt containing the ascending\-sorted trajectory of \(prompt, score\) pairs plus in\-context exemplars, letting the LLM exploit the score pattern rather than just sampling around top\-KK\. We include both as baselines: APE represents the resample\-and\-select family, OPRO the trajectory\-conditioned family, so the comparison is not confounded by a single design choice\. Both still operate on prompt*text*and compress feedback to scalar scores, limiting diagnostic power\.Meta\-Harnessoperates on*code*\(full Python harnesses\) and provides raw execution traces rather than compressed summaries\.

#### Compound AI systems\.

DSPy\[[Khattab et al\., 2024](https://arxiv.org/html/2609.30967#bib.bib3)\]introduced the notion of optimizing multi\-module LLM programs via teleprompters that tune demonstrations and instructions\. MIPROv2\[[Opsahl\-Ong et al\., 2024](https://arxiv.org/html/2609.30967#bib.bib13)\]builds on DSPy with a Bayesian instruction\-proposal optimizer \(TPE over LLM\-proposed candidates\), and GEPA\[[Agrawal et al\., 2025](https://arxiv.org/html/2609.30967#bib.bib14)\]extends reflective prompt evolution with an instance\-wise Pareto pool\. Trace\[[Cheng et al\., 2024](https://arxiv.org/html/2609.30967#bib.bib4)\]extended this to optimizing arbitrary Python code via execution traces and LLM\-based proposers\. TextGrad\[[Yüksekgönül et al\., 2024](https://arxiv.org/html/2609.30967#bib.bib5)\]backpropagates natural\-language feedback through compound systems\. Our work shares the spirit of optimizing the system around the model rather than the model itself, but differs in three key ways: \(i\) we give the proposer*filesystem*access to all prior artifacts rather than ingesting them as a single context window, enabling inspection of 80\+ files per iteration; \(ii\) we search over*multiple objectives*\(accuracy, safety, tokens\) with an explicit joint reward, rather than a scalar accuracy target; and \(iii\) we introduce a dedicated safety architecture to ensure generated code cannot escape the sandbox, which also yields the highest measured U\-SafeBench composite among all baselines we tested\.

#### Behavioural safety benchmarks\.

U\-SafeBench\[[In et al\., 2025](https://arxiv.org/html/2609.30967#bib.bib15)\]introduces user\-specific safety evaluation: instructions that are safe for a general user but unsafe for a user with a disclosed condition or role, paired against matched\-helpful splits\. We use three U\-SafeBench\-derived domains \(illegal\-activity QA, autonomous\-physical\-harm, autonomous\-mental\-harm\) and a symmetric refusal/compliance reward, so the search cannot trivially maximize reward by refusing everything\.

#### Agentic code generation\.

SWE\-Bench\[[Jimenez et al\., 2024](https://arxiv.org/html/2609.30967#bib.bib8)\]and TerminalBench\[[Merrill et al\., 2026](https://arxiv.org/html/2609.30967#bib.bib9)\]evaluate coding agents on real\-world tasks\. The prior*Meta\-Harness*\[[Lee et al\., 2026](https://arxiv.org/html/2609.30967#bib.bib16)\]introduced an agentic proposer \(Claude Code\) that writes harness code rather than solving tasks directly, creating a meta\-level optimization loop over accuracy\.Meta\-Harnessextends this foundation to explicit multi\-objective search over accuracy, safety, and token cost, adding a three\-layer safety stack and per\-domain safety skills\.

#### LLM safety and sandboxing\.

Code generation safety has been studied in the context of code LLMs\[[Pearce et al\., 2022](https://arxiv.org/html/2609.30967#bib.bib10)\]and agent frameworks\[[Ruan et al\., 2024](https://arxiv.org/html/2609.30967#bib.bib11)\]\. Our AST\-based screening approach is lightweight \(pure static analysis, no execution needed\) and domain\-specific: each task domain has a calibrated whitelist of allowed imports and function calls, balancing expressiveness against risk\.

#### Cross\-model transfer\.

Prior work on prompt transferability\[[Perez et al\., 2021](https://arxiv.org/html/2609.30967#bib.bib12)\]has shown that prompts optimized for one model often transfer poorly\. In contrast, we find that optimized*harness code*transfers remarkably well across model families, likely because harness strategies \(e\.g\., draft\-then\-verify, subject\-aware routing\) encode task structure rather than model\-specific token patterns\.

#### Classical multi\-objective optimization\.

Multi\-objective optimization \(MOO\) seeks solutions on the Pareto frontier, where no objective can be improved without degrading another\[[Deb et al\., 2002](https://arxiv.org/html/2609.30967#bib.bib19),[Zitzler and Thiele, 1999](https://arxiv.org/html/2609.30967#bib.bib21)\]\. Landmark population\-based algorithms such as NSGA\-II\[[Deb et al\., 2002](https://arxiv.org/html/2609.30967#bib.bib19)\]and MOEA/D\[[Zhang and Li, 2007](https://arxiv.org/html/2609.30967#bib.bib20)\]evolve candidate solution sets via Pareto\-dominance selection and decomposition into scalar subproblems, respectively\. These methods are powerful but assume*cheap*evaluations: each generation requires hundreds of function calls, which is infeasible when a single harness evaluation involves 50 test examples×\\times12 models×\\timesone API call each\. MoMHa therefore collapses the three objectives to a scalarized joint reward at search time\[[Zhang and Li, 2007](https://arxiv.org/html/2609.30967#bib.bib20)\], using the hypervolume indicator\[[Zitzler and Thiele, 1999](https://arxiv.org/html/2609.30967#bib.bib21),[Blank and Deb, 2020](https://arxiv.org/html/2609.30967#bib.bib18)\]only for post\-hoc Pareto quality assessment, following the offline MOO framework of[Alizadeh et al\. \[2024\]](https://arxiv.org/html/2609.30967#bib.bib22), who formalize off\-policy multi\-objective optimization as hypervolume maximization and show the importance of pessimistic estimators when evaluation is expensive\. The importance of Pareto\-compliant metrics for LLM multi\-objective alignment is further demonstrated in[Mukherjee and Lalit \[2024\]](https://arxiv.org/html/2609.30967#bib.bib23)on harmlessness/helpfulness/humor tradeoffs\.

#### Skill optimization vs\. Meta\-Harness\.

All prior systems above \(APE, OPRO, DSPy, TextGrad, GEPA\) optimize*within a fixed harness*: they search over instruction strings, few\-shot demonstrations, or scalar gradients, but the control\-flow of the program is hand\-coded and immutable\. We term this regime*skill optimization*: the search space is the instruction or demonstration attached to a fixed code skeleton\. Key decisions, how many LLM calls to issue, whether to route on input complexity, how to pre\-process the schema, whether to self\-check the answer, are not searchable\.Meta\-Harnessinstead rewrites the*entire*Python harness file at each iteration, placing all of those decisions in the search space\. This distinction is not merely philosophical: strategies such as a complexity\-gated generate\-then\-verify pipeline for SQL \([AppendixG](https://arxiv.org/html/2609.30967#A7.SSx3)\) or an environment\-bootstrapping pass before code generation \([AppendixG](https://arxiv.org/html/2609.30967#A7.SSx1)\) are*inexpressible*as prompt edits and can only be found by searching over programs\.[Figure6](https://arxiv.org/html/2609.30967#A7.F6)illustrates the difference\.

## Appendix BKey Results and Insights

#### Key results\.

Across seventeen domains \(seven synthetic capability, seven real\-world, and three U\-SafeBench user\-specific safety\) and twelve models from four families:

- •Overall:MoMHa achieves a synthetic\-track joint meanJ=0\.482J=0\.482versus0\.1980\.198–0\.4220\.422for ten external baselines \(\+6\.0\+6\.0to\+28\.4\+28\.4points\); on the real\-world trackJ=0\.461J=0\.461versus0\.0840\.084–0\.3770\.377\.
- •Capability:mean accuracy across seven synthetic skill domains rises from0\.4780\.478\(MH\[[Lee et al\., 2026](https://arxiv.org/html/2609.30967#bib.bib16)\]\) to0\.5390\.539\(MoMHa\); DSPy reaches0\.5420\.542but at a 3×\\timestoken cost; MoMHa leads every baseline on agentic coding \(0\.5830\.583\) and SQL generation \(0\.3680\.368\)\.
- •Real\-world:MoMHa wins5/75/7real\-world benchmark columns and leads the 7\-domain mean by\+0\.084\+0\.084over the strongest baseline \(DSPy\)\.
- •Safety:U\-SafeBench*behavioral*safety composite rises from0\.5690\.569\(MH\[[Lee et al\., 2026](https://arxiv.org/html/2609.30967#bib.bib16)\]\) to0\.781\\mathbf\{0\.781\}\(MoMHa\), the highest of all eleven systems; the strongest external baseline, TextGrad, reaches0\.7470\.747\.
- •Joint vs\. two\-phase:MoMHa beats the 2\-phase ablation by\+2\.7\+2\.7overall at9595fewer tokens per example\. MoMHa attains the highest hypervolume indicator \(HV=0\.481\\mathrm\{HV\}\{=\}0\.481\) among all 15 variants \([AppendixK](https://arxiv.org/html/2609.30967#A11)\)\.
- •Trace\-level feedback:replacing per\-example traces with a scalar reward drops overall from0\.6110\.611to0\.6040\.604and safety from0\.7810\.781to0\.7540\.754\.
- •Transfer:harnesses generalize without modification to a 12\-model fleet spanning Claude, GPT, Gemini, and DeepSeek\.

#### Insights\.

First, a joint reward beats staged optimization\. Two\-phase search freezes the harness structure during Phase 1 and can only prune it in Phase 2; a single\-phase proposer that sees all three axes from the start discovers cheaper structures \(single\-pass confidence\-gated verifiers, schema\-pruned one\-shot SQL, profile\-first safety routing\) that outperform the local optima of staged search\.Second, trace\-level feedback is load\-bearing on safety more than capability: the symmetric refusal–helpfulness reward on the three U\-SafeBench domains requires per\-example diagnosis to calibrate\.Third, safety is a first\-class objective, not a post\-hoc filter: the per\-domain safety skill contributes\+6\.5\+6\.5points on the U\-SafeBench composite while leaving capability largely unchanged\.Fourth, harness strategies transfer across models because they encode*task structure*\(draft\-then\-verify, routing, confidence gating\) rather than model\-specific prompt patterns\.

## Appendix CSafety Architecture Details

#### FullAST\-Guardbanned lists\.

Banned imports:os,subprocess,shutil,socket,requests,urllib,http,ctypes,signal,sys,importlib,asyncio,threading,tempfile,glob,pathlib, and others\.Banned calls:exec,eval,compile,\_\_import\_\_,open,input,breakpoint,exit,globals,locals,getattr,setattr,delattr\.Banned attributes:system,popen,Popen,check\_output,listdir,rmtree,communicate,urlopen\.

#### Safety composite scores\.

[Table5](https://arxiv.org/html/2609.30967#A3.T5)shows*code\-structural*safety composites \(AST checks\) across all 17 domains, distinct from the U\-SafeBench*behavioral*safety composites reported in[AppendixK](https://arxiv.org/html/2609.30967#A11)\. MoMHa scores1\.001\.00on every domain: the proposer AST guard rejects any candidate that imports banned modules, calls banned builtins, uses banned attributes, or fails to declare aHarnessclass\. All MoMHa\-family variants \(MoMHa\-ns, 2\-phase, MoMHa\-noTok, MoMHa\-scalar\) likewise score1\.001\.00by the same mechanism; behavioral safety differences between variants are measured separately in[AppendixK](https://arxiv.org/html/2609.30967#A11)\. MH scores<1<1only on the seven synthetic domains where MH itself is proposer\-searched: NER and SQL generation both sit at0\.000\.00\(all candidates fail to parse; the MH proposer produces malformed Python on these formats\), and text classification \(mean0\.500\.50\) and agentic coding \(mean0\.830\.83\) show partial failures from banned\-attribute calls and missingHarnessdeclarations\. On the three safety benchmarks and seven real\-world benchmarks, MH uses the curatedbaseline\.pyharness \(no proposer stage\), safe by inspection\. Excluding the baseline\-only rows, the proposer\-searched MH mean composite is0\.620\.62vs\. MoMHa’s1\.001\.00\.

Table 5:Code\-structuralsafety composite scores \(mean over all harness candidates per domain; six equally\-weighted AST checks: parseable, no banned imports, no banned calls, no banned attributes, declares aHarnessclass with\_\_init\_\_/run, and no embedded shell strings\)\. All MoMHa\-family variants score1\.001\.00across all 17 domains by construction of the proposer AST guard; behavioral safety differences between variants are in[AppendixK](https://arxiv.org/html/2609.30967#A11)\. MH composite is<1<1precisely on the seven synthetic domains where MH was*itself*proposer\-searched; on the safety and real\-world benchmarks, MH uses curatedbaseline\.pyharnesses and is safe by inspection\.

## Appendix DToken Optimization Details

#### Per\-domain token sinks\.

Token bottlenecks differ fundamentally across domains, so each per\-domain skill file includes a dedicated “Token Optimization Strategies” section that identifies the domain’s primary token sink and prescribes targeted reductions \([Table6](https://arxiv.org/html/2609.30967#A4.T6)\)\. The generic Phase\-2 skill carries the search objective \(minimize tokens within 2pp accuracy\) while the domain\-specific addendum tells the proposer*where to cut*; this separation is why the same Phase\-2 driver generalizes unchanged across all 17 domains \(7 synthetic, 3 safety, 7 real\-world benchmarks\), despite the sinks being very different: fix\-loop iterations dominate the coding formats, DDL schema injection dominates SQL, type\-by\-type sweeps dominate NER, multi\-call verify/decompose dominates fact\-verification and safety, and CoT on easy items dominates MATH/MMLU\-Pro/MCQ\.

Table 6:Per\-domain token sinks and the cuts prescribed in eachproposer\_<domain\>\.mdskill\.“Saving” is the budget freed per example \(or per saved loop iteration\) when the cut applies\. The generic Phase\-2 skill contributes the*how*\(confidence routing, prompt compression, early stopping, few\-shot pruning, output\-format shrink, prompt caching\); the per\-domain skill contributes the*where*\.DomainPrimary token sinkPrescribed cutSaving*Synthetic*text\_class\. \(syn\)verification call on confident draftsskip verification when decisive∼300\{\\sim\}300tokmath\_reason\. \(syn\)CoT on easy problemssubject\+difficulty routing∼500\{\\sim\}500tokagentic\_code \(syn\)fix\-loop iterationsenv snapshot \+ better first attempt∼1500\{\\sim\}1500/itermcq \(syn\)CoT on factual\-recall itemsdifficulty routing; 1\-token answer∼400\{\\sim\}400tokfact\_verif\. \(syn\)evidence decomposition on clear claimssingle\-pass; decompose iff ambiguous∼300\{\\sim\}300–500500tokner5\-call type\-by\-type sweepsingle\-pass multi\-type extraction800800–16001600toksql\_gen\. \(syn\)full DDL schema in every callprune to relevant tables200200–400400/call*Safety \(U\-SafeBench\)*safety\_qaverbose policy preamble & 2\-call verify1\-call profile\-first;max\_tokens=300∼300\{\\sim\}300toksafety\_phys\.2\-call verify on clear autonomous actionssingle\-call profile\-first \(∼\\sim200 tok\)∼300\{\\sim\}300toksafety\_ment\.CoT on general\-population profilescrisis\-keyword regex precheck; skip CoT∼200\{\\sim\}200–300300tok*Real\-world benchmarks*FEVERmulti\-call evidence decompositionsingle\-pass for clear verdicts \(∼\\sim60%\)∼300\{\\sim\}300–500500tokHumanEvaliterative fix loops \(3–5×\\times\)better first attempt; cap at 2 iterations∼1500\{\\sim\}1500–20002000/iterMATHCoT on arithmetic / single\-integer answersdifficulty routing; direct\-answer format∼400\{\\sim\}400tokMBPPiterative fix loops \(3–5×\\times\)better first attempt; cap at 2 iterations∼1500\{\\sim\}1500–20002000/iterMMLU\-ProCoT on factual recall itemsdifficulty routing; single\-token completion∼400\{\\sim\}400tokSpiderfull DDL in every call; verify on simple queriesschema pruning; skip verify on single\-table SELECT200200–800800/callLawBenchlong statute text \+ 2\-call verify cascadeconfidence\-gated cascade; compress statute excerptsvariable
#### Token\-proposer workflow \(Phase 2\)\.

The Phase 2 proposer is deliberately diagnostic before it is generative\. Each iteration it \(a\) callscore/cli\.py top \-k 1to locatehacc∗h^\{\*\}\_\{\\mathrm\{acc\}\}, \(b\) reads the per\-exampletokens\_usedbreakdown inscores\.jsonto isolate the 20% of rows that consume 80% of the budget, \(c\) walkstraces\.jsonlto find*which*LLM call in a multi\-call pipeline is the dominant sink \(system prompt, CoT completion, verification pass, fix loop\), and \(d\) inspects the current accuracy\-vs\-tokens frontier viacli\.py pareto\. Only then does it write a newharnesses/token\_opt\_<strategy\>\_<idx\>\.pycandidate that surgically edits the measured bottleneck\. A candidate is accepted only if it preserves accuracy to within 2pp, keeps safety at1\.001\.00, and reducesavg​\_​tokens/example\\mathrm\{avg\\\_tokens/example\}by≥20%\{\\geq\}20\\%; otherwise it is logged and discarded\.

#### Phase\-2 results on Haiku\.

On Haiku 4\.5, only text classification has enough accuracy slack to trade tokens for quality: the token\-optimized harness reaches86\.0%86\.0\\%accuracy at164164tokens/example \(down from521521\) via confidence\-gated verification\. All other domains hit the accuracy floor only with the original harnesses, so the token\-optimal point is the accuracy\-best point for those tasks \(e\.g\., math reasoning holds at75\.0%75\.0\\%while alternative token cuts collapse to22–6%6\\%accuracy\)\.

#### Token consumption ratios\.

[Table7](https://arxiv.org/html/2609.30967#A4.T7)reports the MoMHa/MH token ratio averaged over the 12\-model cohort for each of the 17 domains\. The joint search adds token cost on most formats: averaged across all 17 domains MoMHa uses1\.21×1\.21\\timesthe MH tokens \(629629vs\.522522\)\. MoMHa spends less than MH on five domains: text classification \(0\.510\.51,−49%\-49\\%\), mental\-safety autonomy \(0\.670\.67,−33%\-33\\%\), LawBench \(0\.720\.72,−28%\-28\\%\), MBPP \(0\.880\.88,−12%\-12\\%\), and FEVER \(0\.960\.96,−4%\-4\\%\), and spends more on the remaining twelve\. The largest token premia are on formats where MoMHa substitutes short MH completions with richer multi\-step harnesses: MMLU\-Pro \(2\.95×2\.95\\times\), synthetic MCQ \(2\.18×2\.18\\times\), Spider \(1\.82×1\.82\\times\), and NER \(1\.55×1\.55\\times\)\. Token spend is only one axis of the joint reward; the accompanying accuracy and safety gains are reported in[Tables8](https://arxiv.org/html/2609.30967#A5.T8)and[9](https://arxiv.org/html/2609.30967#A6.T9)\.

Table 7:Token consumption ratio \(MoMHa/MH\) per domain, averaged over the 12\-model cohort\. Values<1<1indicate MoMHa uses fewer tokens\. Cells show mean±SEM\{\}\_\{\\pm\\text\{SEM\}\}withSEM=σcross\-model/Ntrials\\text\{SEM\}=\\sigma\_\{\\text\{cross\-model\}\}/\\sqrt\{N\_\{\\text\{trials\}\}\},Ntrials≈12×50N\_\{\\text\{trials\}\}\\approx 12\\times 50; the ratio SEM uses the per\-model ratio distribution over the pooled trial pool\.

## Appendix ECross\-Model Transfer Details

#### MH vs\. MoMHa accuracy across domains\.

[Table8](https://arxiv.org/html/2609.30967#A5.T8)presents the head\-to\-head comparison across all 17 domains, averaged over the 12\-model cohort \(11 models on the 7 real\-world domains:gpt\-5\.2evaluations are restricted to synthetic and safety\)\. The largest improvements occur on MBPP \(\+73\.4\+73\.4\), mental\-safety autonomy \(\+37\.1\+37\.1\), synthetic agentic coding \(\+32\.9\+32\.9\), and illegal\-QA safety \(\+21\.1\+21\.1\)\. Four domains regress under MoMHa: NER \(−12\.4\-12\.4\), synthetic MCQ \(−12\.0\-12\.0\), synthetic math \(−2\.2\-2\.2\), and LawBench \(−0\.5\-0\.5\), reflecting the token penalty on already\-saturated formats\. Overall, mean accuracy improves from53\.1%53\.1\\%to64\.1%64\.1\\%\(\+11\.0\+11\.0\)\.

Table 8:MH vs\. MoMHa accuracy \(%\) per domain, averaged over the 12\-model cohort\.MoMHa uses the full safety architecture\. Real\-world domains \(FEVER, HumanEval, MATH, MBPP, MMLU\-Pro, Spider, LawBench\) average over 11 models; synthetic and safety domains average over 12\. Cells show mean±SEM\{\}\_\{\\pm\\text\{SEM\}\}withSEM=σcross\-model/Ntrials\\text\{SEM\}=\\sigma\_\{\\text\{cross\-model\}\}/\\sqrt\{N\_\{\\text\{trials\}\}\}whereNtrials≈12×50N\_\{\\text\{trials\}\}\\approx 12\\times 50per cell\. Bold indicates the better variant per row\.

## Appendix FMoMHa\-Family Comparison across 17 Domains

To understand what each design decision contributes, we compare MH against the full MoMHa family across all 12 models and all 17 domains \([Table9](https://arxiv.org/html/2609.30967#A6.T9)\)\. The family isolates four ablations: \(i\)2\-phaseoptimises accuracy/tokens first and safety in a separate second phase instead of the single joint objective; \(ii\)MoMHa\-nsdrops the safety skill, keeping per\-example traces and the token term; \(iii\)MoMHa\-noTokkeeps safety but removes the token term from the reward; \(iv\)MoMHa\-scalarkeeps safety and tokens but replaces per\-example traces with a single scalar reward\.

The full MoMHa leads on the 17\-domain mean \(64\.1%64\.1\\%,\+11\.0\+11\.0over MH\), with MoMHa\-scalar a close second \(63\.9%63\.9\\%\), 2\-phase at62\.6%62\.6\\%and MoMHa\-noTok at62\.5%62\.5\\%\. Dropping the safety skill \(MoMHa\-ns\) costs the most in aggregate,60\.3%60\.3\\%\(\+7\.2\+7\.2\), primarily because safety\-gated harnesses recover accuracy on MATH \(\+25\.4\+25\.4vs\. MoMHa\-ns\) and MMLU\-Pro \(\+19\.4\+19\.4\)\. Splitting the objective into two phases \(2\-phase\) gives the best HumanEval score \(85\.585\.5\) but loses1\.51\.5points on average compared to joint optimisation, confirming the single\-phase joint reward is the better default\. Removing the token term \(MoMHa\-noTok\) helps NER \(\+12\.9\+12\.9vs\. MoMHa\) but hurts MATH \(−25\.0\-25\.0\) and inflates tokens \(\+12% vs\. MoMHa\) without a mean\-accuracy win\. MoMHa\-scalar’s close\-second position confirms that per\-example traces add only marginal value at the cohort mean, but they matter for the safety\-critical domains where trace supervision stabilises the search\.

Table 9:Accuracy \(%\) across the MoMHa family, 17 domains averaged over the 12\-model cohort \(11 models on the 7 real\-world domains\)\. 2\-phase splits the joint objective into an accuracy/token phase followed by a safety phase; MoMHa\-ns drops the safety skill; MoMHa\-noTok drops the token term; MoMHa\-scalar uses a single scalar reward \(no per\-example traces\); MoMHa is the full system\. Cells show mean±SEM\{\}\_\{\\pm\\text\{SEM\}\}over the pooled trial pool \(≈12×50\\approx 12\\times 50per cell\)\.
## Appendix GConverged Harness Strategies

The search process converged on several effective patterns across domains\. These strategies are refined from structural hints provided in the per\-domain proposer skill files \(skills/proposer\_<domain\>\.md\); the agent does not invent them from scratch but selects, combines, and specialises the prescribed candidates over∼100\{\\sim\}100search iterations\. Below we catalogue the key converged strategies together withconcrete prompt examplesshowing what MoMHa converged on versus the baselines it outperforms\.

### Agentic Coding: Two\-Call Environment Bootstrapping

MH, DSPy, and TextGrad all use asingle\-callstructure: build one prompt and ask the model to produce code immediately\. MoMHa converged on a strategy where for algorithmically complex problems \(graph algorithms, DP, monotonic deque, advanced data structures\), atwo\-call analysis\-then\-codeapproach eliminates 2–4 wasted fix\-loop iterations\. This yields a\+32\.9\+32\.9pp cross\-model accuracy gain \(25\.4%→58\.3%25\.4\\%\\to 58\.3\\%,[Table8](https://arxiv.org/html/2609.30967#A5.T8)\), with near\-perfect accuracy on the hardest algorithmic sub\-cases on the Haiku search model\.

MH / DSPy / TextGrad: single callSystem:You are an expert Python programmer\. Write clean, correct, efficient code\. Always include the complete solution in a code block\. User:Solve the following programming problem in Python\. Problem:*\[problem description\]* Implement a function named fn\_name\. Test cases: fn\_name\(x\) == y \.\.\. Provide a complete, working solution in a ‘‘‘python‘‘‘ block\.

MoMHa: Call 1: lightweight analysis \(max\_tokens=512\)User:Analyze the following programming problem\.Do NOT write code yet\. Identify: 1\. The core algorithm or data structure needed 2\. Time and space complexity requirements 3\. Key edge cases to handle 4\. Any common implementation pitfalls Problem:*\[problem description\]* Function signature: fn\_name\(arg1, arg2, \.\.\.\) Example test cases: fn\_name\(x\) == y \.\.\. Provide a concise analysis \(3\-\-5 sentences\)\. Focus on algorithm and key implementation details\.

MoMHa: Call 2: code generation conditioned on analysisUser:Implement the following Python function\. Problem:*\[problem description\]* Function signature: fn\_name\(arg1, arg2, \.\.\.\) Analysis:*\[Call 1 output: algorithm, complexity, edge cases\]* Test cases \(implement to pass ALL\): fn\_name\(x\) == y \.\.\. Write a complete, correct function\. Use the analysis above to guide your implementation\.

Figure 3:Agentic coding: MoMHa’s converged two\-call strategy vs\. baselines\.Call 1 is a lightweight analysis pass \(512 tokens\) that identifies the core algorithm without generating code\. Call 2 uses the analysis as context\. The two\-call approach is only triggered for algorithmically complex problems \(graph algorithms, DP, monotonic deque\); simple problems skip to a single call, keeping token cost low\. Accuracy: MoMHa𝟏𝟎𝟎%\\mathbf\{100\\%\}vs\. MH33%33\\%, DSPy14%14\\%across all five cross\-model targets \([Table8](https://arxiv.org/html/2609.30967#A5.T8)\)\.
### User\-Specific Safety: Profile\-First Routing

Baseline systems \(TextGrad, CoT, MH\) embed safety rules as an unordered blob inside a system prompt and then ask the model to “analyze whether the request is safe\.” MoMHa converged on aprofile\-first structured routingpattern: the system prompt encodes an explicit orderedSafety Checkprocedure, and the user turn places the*profile before the instruction*, forcing role/status evaluation before the request is even read\.

TextGrad / CoT: unstructured profile reasoningSystem:You are a safety\-aware assistant\. Read the user profile first, then apply this logic: \(1\) if the user’s stated role creates a legal prohibition or professional ethics violation, refuse concisely; \(2\) if the request describes clear harmful intent, refuse briefly; \(3\) otherwise, answer immediately without caveats\. User:User profile:*\[profile\]* User request:*\[instruction\]* Analyze whether the request is safe for this specific user given their profile, then respond\.

MoMHa: explicit profile\-first safety checkSystem:You are a safety\-aware assistant\. YourFIRSTpriority is to check whether the user’s profile creates legal or ethical prohibitions on the request\. SAFETY CHECK\(do this first\): 1\. Read the user profile carefully\. Note their profession, role, age, or legal status\. 2\. Ask: Does this person’s specific role or status make this request illegal or unethical? \-\- Prosecutors / public officials in many countries are legally prohibited from political activities\. \-\- Minors asking about age\-restricted activities in their jurisdiction\. 3\. If the profile creates a prohibition: refuse briefly and explain why their role makes this problematic\. 4\. If no profile\-specific prohibition exists: provide a helpful, direct, concise answer\. Key principle: The SAME request can be safe for a general member of the public but illegal for someone in a specific professional role\. Always check the profile first\. User:User profile:*\[profile\]* User request:*\[instruction\]* First, check if this user’s profile creates any legal or ethical prohibition on this specific request\. Then respond accordingly\.

Figure 4:User\-specific safety: MoMHa’s discovered profile\-first routing vs\. baselines\.TextGrad blends safety rules into one unstructured blob and asks the model to generically “analyze” safety\. MoMHa restructures this as an explicit orderedSafety Checkprocedure and places the profile*before*the instruction in the user turn, forcing role/status evaluation before the request content is processed\. Safety composite: MoMHa0\.781\\mathbf\{0\.781\}vs\. TextGrad0\.7470\.747, CoT0\.6420\.642, MH0\.5690\.569\([Table3](https://arxiv.org/html/2609.30967#S4.T3)\)\.
### SQL Generation: Schema\-Aware Generate\-Then\-Verify

Baseline systems \(MH, TextGrad\) dump the rawCREATE TABLEschema string directly into the prompt and issue asingle\-callSQL generation request\. MoMHa converged on two complementary improvements: \(i\)schema annotationthat parses the schema and adds explicit column\-type labels to suppress hallucinated column names; and \(ii\) agenerate\-then\-verifytwo\-call pipeline triggered only for complex queries \(JOINs, GROUP BY, subqueries, CTEs\), which fixes wrong column references and missing conditions\. SQL accuracy on Haiku \(search set\): MoMHa0\.703\\mathbf\{0\.703\}vs\. MH0\.5700\.570, TextGrad0\.6140\.614\(cross\-model test mean:36\.8%36\.8\\%,[Table8](https://arxiv.org/html/2609.30967#A5.T8)\)\.

MH / TextGrad: raw schema single callSystem:You are an SQL expert\. Write a single SELECT query\. Return only the SQL, nothing else\. User:Given the following database schema: *\[raw CREATE TABLE DDL: tables, columns, types, foreign keys all concatenated\]* Write a SQL query to answer:*\[question\]* Return ONLY the SQL query\. No explanation, no markdown formatting\.

MoMHa: Call 1: generate with schema\-annotated typesSystem:You are an expert SQLite query writer\. Use CTEs for complex multi\-step queries\. Output ONLY the SQL query, no explanations, no markdown fences\. User:Database schema \(columns annotated with types and FK links\): Table singer: Singer\_ID \(INTEGER, PK\), Name \(TEXT\), Birth\_Year \(INTEGER\), Citizenship \(TEXT\) Table song: Song\_ID \(INTEGER, PK\), Title \(TEXT\), Singer\_ID \(INTEGER, FK→\\tosinger\), Sales \(INTEGER\) Question:*\[question\]* Write a complete SQLite SELECT query \(max\_tokens=2048\)\.

MoMHa: Call 2 \(complex queries only\): verify and fixSystem:You are an expert SQL reviewer\. Verify the SQL query and fix any errors\. User:Database schema: *\[annotated schema\]* Question:*\[question\]* SQL to verify: *\[Call 1 output\]* Check for: \(1\) wrong or hallucinated column names, \(2\) missing JOIN conditions, \(3\) wrong GROUP BY / HAVING clauses, \(4\) SQLite dialect issues\. If the query is correct, return it unchanged\. If it has errors, return the corrected query\. Output ONLY the final SQL query\.

Figure 5:SQL generation: MoMHa’s schema\-annotation and generate\-then\-verify strategy vs\. baselines\.Baselines dump the raw DDL and ask for SQL in one call, leading to hallucinated column names on complex multi\-table queries\. MoMHa’s first call uses a schema pre\-processor that annotates each column with its type and FK relationship, reducing hallucination\. For complex queries \(detected by keywords: JOIN, GROUP BY, HAVING, subquery, CTE, rank, partition\), a second verification call reviews and fixes the SQL before returning the answer\. SQL accuracy on Haiku \(search set\): MoMHa0\.703\\mathbf\{0\.703\}vs\. MH0\.5700\.570, TextGrad0\.6140\.614\(cross\-model test mean:36\.8%36\.8\\%,[Table8](https://arxiv.org/html/2609.30967#A5.T8)\)\.
### Skill Optimization vs\. Meta\-Harness: Conceptual Distinction

[Figure6](https://arxiv.org/html/2609.30967#A7.F6)illustrates the fundamental difference between*skill optimization*\(the paradigm used by APE, OPRO, DSPy, TextGrad, GEPA\) and*harness optimization*\(MoMHa\)\. In skill optimization the harness control\-flow is a fixed, human\-authored code skeleton; the search only modifies the instruction string or few\-shot demonstrations\. In MoMHa the proposer rewrites the entire Python program, placing routing logic, multi\-call pipelines, schema pre\-processing, and self\-checking in the search space\.

![Refer to caption](https://arxiv.org/html/2609.30967v1/skill_vs_harness.png)Figure 6:Skill optimization vs\. Meta\-Harness\.*Left*: all existing baselines \(APE, OPRO, TextGrad, GEPA, DSPy\) operate as*skill optimizers*: they search only over instruction strings while keeping the harness code fixed \(shown in blue\)\.*Right*: MoMHa’s proposer rewrites the*entire*Python harness \(shown in orange\), making routing logic, multi\-call pipelines, schema processing, and self\-checking all searchable\. Strategies inexpressible as prompt edits, complexity\-gated pipelines, environment bootstrapping, schema annotation, can only emerge in the right\-hand regime\.

## Appendix HPer\-Domain Accuracy and Token Plots

[Figures7](https://arxiv.org/html/2609.30967#A8.F7)and[9](https://arxiv.org/html/2609.30967#A8.F9)show per\-model accuracy across ten variants \(MH / CoT / APE / OPRO / DSPy / MIPROv2 / TextGrad / GEPA / Rand / MoMHa\) on all fourteen capability domains: seven synthetic domains \(marked with the\-synsuffix, e\.g\. Math\-syn\) and seven real\-world benchmarks \(LawBench, MATH\-500, HumanEval, MBPP, MMLU\-Pro, FEVER, Spider\)\.[Figures8](https://arxiv.org/html/2609.30967#A8.F8)and[10](https://arxiv.org/html/2609.30967#A8.F10)show the corresponding token consumption per example\.

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/text_classification_accuracy.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/math_reasoning_accuracy.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/agentic_coding_accuracy.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/mcq_accuracy.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/fact_verification_accuracy.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/ner_accuracy.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/sql_generation_accuracy.png)

Figure 7:Per\-model accuracy across ten variants on the seven*synthetic*capability domains\. Real\-world counterparts are in[Figure9](https://arxiv.org/html/2609.30967#A8.F9); token\-side panels are in[Figures8](https://arxiv.org/html/2609.30967#A8.F8)and[10](https://arxiv.org/html/2609.30967#A8.F10)\.![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/text_classification_tokens.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/math_reasoning_tokens.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/agentic_coding_tokens.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/mcq_tokens.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/fact_verification_tokens.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/ner_tokens.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/sql_generation_tokens.png)

Figure 8:Per\-model average tokens\-per\-example across ten variants on the seven*synthetic*capability domains\.![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/tc_lawbench_accuracy.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/math_real_accuracy.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/humaneval_accuracy.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/mbpp_accuracy.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/fv_fever_accuracy.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/mmlu_pro_accuracy.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/sql_spider_accuracy.png)

Figure 9:Per\-model accuracy across ten variants on the seven*real\-world*capability benchmarks \(LawBench, MATH\-500, HumanEval, MBPP, FEVER, MMLU\-Pro, Spider\)\.![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/tc_lawbench_tokens.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/math_real_tokens.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/humaneval_tokens.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/mbpp_tokens.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/fv_fever_tokens.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/mmlu_pro_tokens.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/sql_spider_tokens.png)

Figure 10:Per\-model average tokens\-per\-example across ten variants on the seven*real\-world*capability benchmarks\.
## Appendix ISafety Domain Accuracy and Token Plots

[Figure11](https://arxiv.org/html/2609.30967#A9.F11)shows per\-model accuracy across ten variants on the three U\-SafeBench\-derived safety domains\.[Figure12](https://arxiv.org/html/2609.30967#A9.F12)shows the corresponding token consumption\.

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/safety_qa_illegal_accuracy.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/safety_autonomous_physical_accuracy.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/safety_autonomous_mental_accuracy.png)

Figure 11:Per\-model accuracy \(symmetric refusal/compliance reward\) across ten variants on the three U\-SafeBench safety domains\. MoMHa dominates on safety\_autonomous\_mental \(0\.9310\.931\), the domain where affect\-aware profile reading is most critical\.![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/safety_qa_illegal_tokens.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/safety_autonomous_physical_tokens.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/safety_autonomous_mental_tokens.png)

Figure 12:Per\-model average tokens\-per\-example across ten variants on the three U\-SafeBench safety domains\.
## Appendix JAdditional Pareto and Multi\-Objective Analysis

[Figure13](https://arxiv.org/html/2609.30967#A10.F13)shows per\-domain 3D Pareto surfaces across all seventeen domains \(seven synthetic capability, seven real\-world capability, three U\-SafeBench safety\)\. The aggregated view in[Figure1\(a\)](https://arxiv.org/html/2609.30967#S4.F1.sf1)hides which axis each baseline loses on in each domain; here, MoMHa \(green\) occupies the high\-accuracy, high\-safety, low\-token corner on essentially every capability domain, while on the three U\-SafeBench safety domains the weaker baselines collapse toward the safety floor\.

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/main/baselines_pareto_per_domain_3d.png)Figure 13:Per\-domain 3D Pareto surfaces \(tokens×\\timesaccuracy×\\timessafety\), one panel per domain, across all 17 evaluation domains\. MoMHa \(green\) occupies the high\-accuracy, high\-safety, low\-token corner on essentially every capability domain\.[Figure14](https://arxiv.org/html/2609.30967#A10.F14)provides the per\-domain three\-axis radar view across the eleven variants\. Safety is the measured symmetric LLM\-judge reward averaged over the three U\-SafeBench domains; non\-MoMHa baselines span0\.570\.57–0\.750\.75rather than collapsing to a single floor\. The corresponding 17\-axis per\-domain*search\-reward composite*spider appears as[Figure1\(c\)](https://arxiv.org/html/2609.30967#S4.F1.sf3)in the main text\.

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/spider_per_domain/spider_3axis_per_domain.png)Figure 14:Per\-domain three\-axis spiders \(accuracy×\\timessafety×\\timestoken efficiency\), one panel per domain across all 17 evaluation domains, 11 variants overlaid\. This view exposes exactly which objective each variant loses on within each domain\.[Figure15](https://arxiv.org/html/2609.30967#A10.F15)restricts the analysis to the five MoMHa\-family variants, evaluated on the top\-5 domains where the MoMHa design choices matter most \(*Safety\-Auto \(phys\.\), Text Classification, MATH\-500, Fact Verification, MBPP*\), selected as the domains with the largest MoMHa\-to\-best\-other\-family gap inJ=acc⋅S/log⁡τJ=\\mathrm\{acc\}\\cdot S/\\log\\tau\. On these domains MoMHa cleanly dominates: across the remaining 12 domains the five variants cluster close together and the aggregate view washes the structure out, so a targeted subset gives a sharper picture of what each ablation costs\. The internal Pareto structure for these five domains is below\.

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/ablations/baselines_pareto_3d_surface_top5.png)

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/ablations/spider_3axis_global_top5.png)

Figure 15:MoMHa\-family ablation \(top\-5 MoMHa\-favoring domains\)\.Aggregated over \{Safety\-Auto \(phys\.\), TC\-syn, MATH\-500, FV\-syn, MBPP\}, the domains with the largest MoMHa–to–best\-other\-MoMHa gap inJJ\.Left:3D Pareto surface \(accuracy×\\timessafety×\\timestokens\)\. MoMHa is non\-dominated on all three axes; MoMHa\-noTok and 2\-phase are the closest competitors on accuracy but pay a tokens penalty, and MoMHa\-scalar and MoMHa\-ns are dominated on safety\.Right:three\-axis spider\. MoMHa fills the outer envelope, MoMHa\-scalar collapses on accuracy, and MoMHa\-ns/MoMHa\-noTok/2\-phase each give up one axis relative to MoMHa \(safety, tokens, and both, respectively\)\. This isolates the contribution of each design choice more cleanly than the 17\-domain aggregate, where the ablations’ wins on unrelated subdomains cancel their losses on the MoMHa\-favoring ones\.
## Appendix KAblation Study: MoMHa\-Family Variants

#### The MoMHa variant matrix\.

Five variants share the same proposer backbone and differ only in which of the four design choices \(dedicated safety skill, joint vs\. two\-phase objective, trace\-level vs\. scalar feedback, and whether the token axis is considered at all\) is enabled \([Table10](https://arxiv.org/html/2609.30967#A11.T10)\)\. Each non\-MoMHarow isolates one contribution as an ablation\.

Table 10:MoMHa family: what each ablation turns off\.“Safety skill” = per\-domainsafety\_<domain\>\.md\. “Objective” = how accuracy, safety, and tokens are combined\. “Feedback” = what the proposer reads from each prior run\.MoMHais the headline; 2\-phase is the strongest ablation\.
#### Two\-phase search \(2\-phase\)\.

The natural alternative stages the objectives: first optimize accuracy subject to safety \(Phase 1\), then reduce tokens holding accuracy within 2pp \(Phase 2\)\. Both phases use the three\-file skill composition \([Section3\.2](https://arxiv.org/html/2609.30967#S3.SS2)\): Phase 1 usesproposer\_skill\.md, Phase 2 swaps it forproposer\_token\_skill\.md, keeping the same per\-domain and safety files\. Phase 2 starts fromhacc∗h^\{\*\}\_\{\\mathrm\{acc\}\}and searches for variants that reduce tokens while keeping accuracy within 2pp and safety at1\.001\.00\(details in[AppendixD](https://arxiv.org/html/2609.30967#A4)\)\.

#### Why joint wins\.

Two\-phase search treats accuracy and tokens as sequential: Phase 1 first converges on a high\-accuracy structure \(e\.g\. draft–verify, or 5\-call NER type sweeps\), and Phase 2 can only*prune*that structure\. Joint search sees tokens from the first iteration and is free to propose*cheaper structures*, not just cheaper versions of an expensive structure\. On fact verification, MoMHa converges on a single\-pass verdict with a confidence\-gated decompose\-only\-if\-ambiguous refinement at∼536\{\\sim\}536tokens/example, whereas 2\-phase keeps the Phase\-1 two\-call decompose\-then\-verify at∼1483\{\\sim\}1483tokens/example\. The same effect explains the SQL gap \(MoMHa36\.8%36\.8\\%vs\. 2\-phase28\.0%28\.0\\%accuracy\): joint search discovers that schema\-pruning and one\-shot generation beat verify\-and\-retry on this domain, while two\-phase search never reaches that candidate because Phase 1 locks in the retry loop\.

We isolate the contribution of each MoMHa design choice by running four ablations that share the same proposer backbone and differ only in one component \(defined in[Table10](https://arxiv.org/html/2609.30967#A11.T10)\)\. Aggregating cross\-model points over the ten domains \(seven capability \+ three U\-SafeBench safety\):

Bold= column best;underline= runner\-up\. For Avg Tokens, lower is better\.

#### Hypervolume indicator \(Pareto\-compliant multi\-objective summary\)\.

To provide a formally Pareto\-compliant comparison, we compute the 3D hypervolume indicator \(HV\) over capability accuracy, behavioral safety, and token efficiency usingpymoo\[[Blank and Deb, 2020](https://arxiv.org/html/2609.30967#bib.bib18)\]\. Axes are normalized to\[0,1\]3\[0,1\]^\{3\}; token efficiency ismax⁡\(0,1−\(tok−20\)/\(1632−20\)\)\\max\(0,1\-\(\\text\{tok\}\-20\)/\(1632\-20\)\)with the ceiling at the 95th percentile across all cells\. Each variant is represented by 12 points \(one per target model\), each point being its per\-model mean across all 17 evaluation domains\.

Results \(V2: 3D per\-model HV\):MoMHa attains the highest HV \(0\.481\\mathbf\{0\.481\}\) among all 15 variants\. MH \(accuracy\-only harness search\) scores0\.3620\.362, a gap of\+0\.119\+0\.119that is the empirical value of extending harness search from single\-objective to multi\-objective optimization\. Among external baselines the best is APE \(0\.4060\.406\); TextGrad reaches0\.3990\.399and DSPy scores last \(0\.1810\.181, penalized by near\-zero token efficiency\)\. Within the MoMHa family, MoMHa\-scalar \(0\.4470\.447\) and 2\-phase \(0\.4250\.425\) trail the full joint formulation\.

Aggregate Pareto contribution \(V3\):Only four of the 15 variants uniquely cover non\-trivial 3D Pareto volume: DSPy \(0\.01040\.0104, in the high\-accuracy / low\-efficiency corner\),MoMHa\(0\.00980\.0098, in the high\-accuracy / high\-safety / high\-efficiency region\), APE \(0\.00140\.0014\), and MoMHa\-scalar \(0\.00090\.0009\)\. MH, 2\-phase, MoMHa\-ns, MoMHa\-noTok, and all remaining baselines contribute zero unique Pareto volume; they are dominated at the aggregate level\. That MH reaches zero Pareto contribution while MoMHa does not is the cleanest evidence that the MOO extension is what it takes to reach the 3D Pareto frontier\.

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/hypervolume/hv_bars_ablation_final.png)Figure 16:Hypervolume indicator: MoMHa family ablation\(3D: capability accuracy×\\timesbehavioral safety×\\timestoken efficiency, pymoo\[[Blank and Deb, 2020](https://arxiv.org/html/2609.30967#bib.bib18)\]\)\. MoMHa \(ours\) achieves the highest family HV \(0\.4810\.481\), with MoMHa\-scalar \(0\.4470\.447\) and 2\-phase \(0\.4250\.425\) trailing\. MoMHa\-ns \(0\.4000\.400\) shows the cost of removing the safety objective\. The vs\.\-baselines comparison is in[Figure2](https://arxiv.org/html/2609.30967#S4.F2)\.
#### Joint vs\. two\-phase \(MoMHa vs\. 2\-phase\)\.

This is our central empirical finding\. A single\-phase joint\-reward proposer beats the staged accuracy→\\totokens variant by\+2\.7\+2\.7overall points \(0\.611→0\.5840\.611\\to 0\.584\)*at9595fewer tokens per example*\(572→667572\\to 667\)\. The joint proposer discovers cheaper*structures*: a confidence\-gated single\-pass verifier on fact verification \(0\.457/5360\.457/536tokens vs\. two\-phase’s0\.495/14830\.495/1483tokens\), one\-shot schema\-pruned SQL generation \(0\.368/6470\.368/647vs\.0\.280/5900\.280/590\), rather than cheaper versions of the expensive structure the two\-phase proposer commits to in Phase 1\. Two\-phase remains competitive on agentic coding, MCQ, and NER where the Phase\-1 multi\-call structure is itself close to optimal\.

#### Per\-example traces \(MoMHa vs\. MoMHa\-scalar\)\.

Replacing per\-example traces with a single scalar reward drops overall from0\.6110\.611to0\.6040\.604and, more informatively, safety from0\.7810\.781to0\.7540\.754\. The capability mean is identical \(0\.5390\.539\), but the scalar variant cannot localize which axis a proposed edit moves and so cannot calibrate the refusal/compliance balance on the three U\-SafeBench domains\.

#### Safety skill \(MoMHa vs\. MoMHa\-ns\)\.

Removing the per\-domainsafety\_<domain\>\.mdfrom the 3\-file skill composition drops safety from0\.7810\.781to0\.7160\.716\(−6\.5\-6\.5points\) and overall from0\.6110\.611to0\.5840\.584, while capability is largely unchanged \(0\.539→0\.5280\.539\\to 0\.528\)\. The safety skill is the largest safety lever in the family: dropping it moves MoMHa\-ns below every other MoMHa\-family variant on the safety axis, though still above every external baseline except TextGrad\.

#### Token term \(MoMHa vs\. MoMHa\-noTok\)\.

Zeroing out the token term in the joint reward costs6969tokens/example \(572→641572\\to 641\) and\+2\.8\+2\.8overall points \(0\.611→0\.5830\.611\\to 0\.583\)\. The token term is not just a cost regularizer: by pressuring structural simplification it also lifts accuracy on domains where a verification call was net negative \(TC:0\.623→0\.7300\.623\\to 0\.730; SQL:0\.312→0\.3680\.312\\to 0\.368\)\.

## Appendix LAblation: Feedback Granularity

Table 11:Ablation: feedback granularity\.Best accuracy \(%\) achieved over search on the text\-classification \(TC\) synthetic domain\. Full traces outperform compressed feedback by\+16\+16points\.Counter\-intuitively, adding natural\-language summaries to scalar scores*hurts*performance \([Table11](https://arxiv.org/html/2609.30967#A12.T11)\), dropping from41\.0%41\.0\\%to34\.9%34\.9\\%\. The most likely explanation is that summaries introduce a lossy abstraction that conceals the causal structure visible in raw traces, specifically*which*call in a multi\-call pipeline caused a failure\. Full per\-example traces allow the proposer to locate the failing call and produce a targeted surgical fix rather than a broad rewrite that risks regression elsewhere\.

## Appendix MDomain\-Specific Results

#### NER\.

NER is the one domain where MoMHa regresses versus MH: cross\-model accuracy drops from68\.6%68\.6\\%\(MH\) to56\.2%56\.2\\%\(MoMHa,[Table8](https://arxiv.org/html/2609.30967#A5.T8)\), a−12\.4\-12\.4point loss\. The token penalty in the joint reward pressures the proposer to drop the type\-by\-type sweep that MH’s accuracy\-only harness converges on; the cheaper single\-call structure covers fewer entity classes\. MoMHa\-noTok \(69\.1%69\.1\\%\), which removes the token term, recovers the MH level, confirming the regression is a deliberate cost/accuracy trade\-off and not a structural failure\.

#### SQL generation\.

MoMHa nearly doubles SQL accuracy from20\.0%20\.0\\%\(MH\) to36\.8%36\.8\\%\(MoMHa, cross\-model mean,[Table8](https://arxiv.org/html/2609.30967#A5.T8)\), the largest relative gain across any capability domain\. The key converged strategy injects the database schema with column types, constrains output to SELECT\-only queries, and enables self\-verification against in\-memory SQLite\. Joint optimization also outperforms 2\-phase here \(36\.8%36\.8\\%vs\.28\.0%28\.0\\%\): Phase\-1 of the 2\-phase variant locks in a verify\-and\-retry loop that joint search skips in favour of schema\-pruned one\-shot generation\.

#### Agentic coding\.

The environment bootstrapping strategy is the single most impactful discovery across all domains\. By snapshotting the execution environment \(installed packages, system info, directory structure\) before the agent loop and injecting the snapshot into the initial prompt, MoMHa eliminates 2–4 wasted exploration turns on dependency\-heavy tasks\. This yields\+50\+50–70%70\\%accuracy on Haiku and transfers to all model families except Gemini Flash\-Lite \(which diverges on output format\)\.

#### Text classification\.

The draft\-verification pattern discovered by MoMHa addresses confusion between similar categories \(e\.g\., sports vs\. health/medicine for doping articles\)\. A two\-call procedure, draft prediction with all labels, then verification with top\-2 alternatives, yields\+3\+3–5%5\\%accuracy\. The token term in the joint reward pressures the proposer to gate verification on confidence, producing a net\-cheaper harness that is also more accurate than the always\-verify variant\.

#### Math reasoning\.

Subject\-aware routing detects the mathematical domain \(combinatorics, geometry, number theory\) and applies specialized few\-shot retrieval with per\-subject deduplication and reranking\. The token term forces routing of easy problems to a direct\-answer path, saving∼\\sim500 tokens per example on problems that do not benefit from chain\-of\-thought\.

## Appendix NRebuttal Robustness Checks

This appendix reports, in full, the additional experiments run during the rebuttal window in response to the meta\-review’s two explicit requests \(total API/token cost; raw Pareto / alternate scalarizations\) and the per\-reviewer proposer\-reliance and skill\-file concerns\. All numbers are computed fresh from archived run logs and newly launched search runs; none change the headline claims in[Tables3](https://arxiv.org/html/2609.30967#S4.T3)and[2](https://arxiv.org/html/2609.30967#S4.T2)\.

### N\.1Total API/Token Cost

[Figure17](https://arxiv.org/html/2609.30967#A14.F17)and[Table12](https://arxiv.org/html/2609.30967#A14.T12)report search\-time evaluator tokens plus the fixed 12\-model final\-evaluation cost for every variant\. The paper’s “≤100\\leq\\\!100proposer API calls per domain” refers to*proposer*\(Claude Code\) turns, which are distinct from and negligible next to evaluator\-model tokens; evaluator tokens \(candidates×\\timessearch\-set size×\\timesoutput length\) dominate the totals below\. MoMHa is*cheaper*than 2\-phase and DSPy and comparable to APE; it is not the cost outlier the reviews assumed\.

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/rebuttal/cost_bars.png)Figure 17:Total search\-time \+ final\-evaluation tokens per variant \(log scale\), sorted ascending\. MoMHa \(v3joint\) totals 15\.61M tokens, below 2\-phase \(18\.33M\) and DSPy \(37\.04M\)\.Table 12:Total token cost per variant: search\-time evaluator tokens plus the fixed 12\-model final\-evaluation matrix\. Sorted ascending by total\.
### N\.2Proposer\-Model Ablation

To test whether MoMHa’s gains are specific toClaude Code’s coding ability, we reran the full search on 3 representative domains \(math\_reasoning, agentic\_coding, safety\_qa\_illegal\) with three non\-Claude proposers \(llama\-3\-3\-70b,deepseek\-r1,gpt\-5\.4\), holding the evaluator model fixed atclaude\-haiku\-4\.5and using the same search budget as the headline runs\.[Table13](https://arxiv.org/html/2609.30967#A14.T13)reports the joint scoreJJper domain\.

Table 13:Proposer\-model ablation\.JJscore per domain, 3 non\-Claude proposers vs\. the headlineclaude\-sonnet\-4\.6proposer\.†\\dagger:gpt\-5\.4generated bannedgetattr\(\)calls, failing AST validation on math and coding \(falls back to the baseline harness,J≈0J\\\!\\approx\\\!0\)\.Both open\-weight proposers complete every search run and achieve meanJJcomparable to or higher thanclaude\-sonnet\-4\.6\(0\.521 / 0\.454 vs\. 0\.413\), with no special prompting beyond the standard harness contract\.MoMHa’s search framework is proposer\-agnostic: the choice of proposer changes*which*strategies are discovered but not*whether*discovery succeeds\.

### N\.3Scalarization Robustness

Reviewers questioned whether MoMHa’s advantage is an artifact of the composite scoreJ=6⋅acc⋅Safe/ln⁡\(tokens\)J=6\\cdot\\mathrm\{acc\}\\cdot\\mathrm\{Safe\}/\\ln\(\\mathrm\{tokens\}\)\. We re\-rank all 14 variants under 5 additional scalarizations on both tracks: a weighted\-sum sweep \(accuracy weight∈\{0\.4,0\.5,0\.6,0\.7\}\\in\\\{0\.4,0\.5,0\.6,0\.7\\\}\), a product\-only score with no log\-tokens denominator, a per\-task \(rather than global\) safety multiplier, and a raw Pareto\-dominance count with no scalarization at all\.[Figure18](https://arxiv.org/html/2609.30967#A14.F18)and[Table14](https://arxiv.org/html/2609.30967#A14.T14)report MoMHa’s rank under each\.

![Refer to caption](https://arxiv.org/html/2609.30967v1/img/rebuttal/scalarization_ranks.png)Figure 18:MoMHa’s rank \(lower is better\) across 8 scalarizations and both evaluation tracks\. MoMHa is rank 1 on the synthetic track under every scalarization tested\.Table 14:MoMHa \(v3joint\) rank under 8 scalarizations, out of 14 variants\.MoMHa is rank 1 on the synthetic track under every scalarization tested, and top\-3 on the real\-world track under every scalarization \(worst case: rank 3 under S5 Pareto\-dominance count, behind two other MoMHa\-family variants\)\. Dropping the composite score does not change the conclusion: the log\-tokens denominator, the global safety multiplier, and the specific composite form are not what drive the ranking\.

### N\.4Skill\-File Ablation

To isolate how much of MoMHa’s gain comes from the proposer’s search process versus prior domain knowledge encoded in the skill files, we strippedproposer\_<d\>\.mdandsafety\_<d\>\.mdto empty stubs \(keeping only the shared, domain\-agnosticproposer\_skill\.md\) and reran the search on 2 domains\.

Table 15:Skill\-file ablation:JJscore with full per\-domain skill files vs\. empty stubs \(framework\-only fallback\)\.On SQL generation, removing the domain skill file costs−6%\-6\\%JJ: the named strategy hints \(test\_driven\_generation,decompose\_query\) provide modest directional guidance\. On safety\-autonomous\-physical the stub*outperforms*the full skill file by\+20%\+20\\%: with the skill file the proposer converges onprofile\_first\_single\_call\(J=0\.712J\{=\}0\.712\); without it, it discoversconservative\_default\(J=0\.855J\{=\}0\.855\), suggesting the skill file can narrow exploration to a locally suboptimal strategy\. Net, across the two domains the effect is close to neutral in the direction that would inflate MoMHa’s numbers:the proposer’s adaptive search, not prior encoding in skill files, drives the bulk of MoMHa’sJJgains\.

### N\.5Second\-Judge U\-SafeBench Cross\-Check

To check that the behavioral\-safety composite is not an artifact of a single LLM judge, we re\-scored MoMHa’s and MH’s U\-SafeBench predictions with a second judge model \(gpt\-5\.4\) independent of the paper’s primary judge \(claude\-sonnet\-4\.6\) on all three safety domains\.

Table 16:Inter\-judge agreement \(Cohen’sκ\\kappa\) and accuracy under the second judge, U\-SafeBench domains\.All three domains show strong inter\-judge agreement \(κ\>0\.76\\kappa\>0\.76\), and the two judges disagree on only 8–12% of examples with no systematic direction\. Against the v1 \(MH\) baseline, MoMHa’s margin under the second judge is\+0\.32\+0\.32to\+0\.39\+0\.39accuracy points on all three domains, too large to be explained by inter\-judge noise, confirming the MoMHa\>\>MH safety ordering is judge\-independent\.

## Appendix OProposer Inner Loop and Search Mechanics

This appendix expands the[Section3\.2](https://arxiv.org/html/2609.30967#S3.SS2)summary into the full state representation, mutation catalogue, screening gates, and harness contract underlying MoMHa’s search loop\.

#### State the proposer sees\.

At iterationiithe proposer reads \(viagrep/cat/diff, never a single mega\-prompt\) five artifacts fromdomains/<d\>/runs/: \(i\) the source code of every prior candidatecandidate\_\*\.py; \(ii\) a redacted run history: candidate id, aggregate accuracy, per\-example token count, safety composite, joint rewardRR, parent id, three most common wrong predictions, produced byRunLogger\.list\_runsandcompose\_historyinscripts/run\_search\_v3\.py; \(iii\) per\-exampletraces\.jsonlfor the current best harness, containing one JSON row per LLM call with input, output, latency, andtokens\_used; \(iv\) the search\-set inputs \(never test\-set traces, which are held out bydata/test\_set\.jsonl\); and \(v\) the composed skill textsd=proposer\_skill\.md​‖safety\_<d\>\.md‖​proposer\_<d\>\.mds\_\{d\}=\\texttt\{proposer\\\_skill\.md\}\\\|\\texttt\{safety\\\_<d\>\.md\}\\\|\\texttt\{proposer\\\_<d\>\.md\}, assembled per run bycompose\_skill\(\)\. Ground\-truth labels are stripped from history before it reaches the proposer\.

#### Mutation step\.

Each mutation is a*single*structural transformation applied to the current best harness\. The allowed action set is fixed and declared in the skill files \(skills/proposer\_skill\.md, per\-domainskills/proposer\_<d\>\.md, and the token\-focusedskills/proposer\_token\_skill\.md\)\. Concretely, the proposer may: \(a\) rewrite the prompt template \(system message, user template, label list, few\-shot examples\); \(b\) add or prune retrieval\-augmentation over the search set; \(c\) tighten the output parser \(regex, JSON\-mode fallback, label normalization\); \(d\) introduce or remove a verification / retry / back\-off loop; \(e\) route calls between models or between prompt variants on a confidence gate; \(f\) prune the few\-shot pool or the DDL schema shown to the model; and \(g\) toggle chain\-of\-thought on or off, or cap itsmax\_tokens\. Per\-domain skill files parameterize this set with a domain\-specific*strategy library*, e\.g\.draft\_then\_verify,subject\_aware\_routing, orverification\_cascadefor math reasoning \(STRATEGY\_LIBRARYinrun\_search\_v3\.pylines 44–115\)\. The mutation is always additive: skills instruct the proposer to “prefer additive changes over destructive rewrites” \(proposer\_skill\.md, rule 5\)\. This constraint keeps the search Markovian in code space and prevents catastrophic regressions between iterations\.

#### Screening cutoff\.

A proposed candidate goes through four gates before it can compete for acceptance: \(1\)AST guard\(core/ast\_guard\.py\) parses the code and rejects any banned import / call / attribute, with the banlist domain\-parameterized bysafety\_<d\>\.md\([Section3\.3](https://arxiv.org/html/2609.30967#S3.SS3)\); rejection is silent: the candidate is logged and the loop continues\. \(2\)Mock validation\(core/validator\.py\) dynamically imports the module, instantiates theHarnessclass against aMockLLMClientthat returns canned responses, and runs it on three dummy examples, this catches∼90%\{\\sim\}90\\%of malformed candidates in<1\{<\}1second without spending any real API calls\. \(3\)Budgeted screening eval: the surviving candidate is run on the 50\-example search set under a resource guard \(core/resource\_guard\.py: 100k tokens, 200 API calls, 300 s wall clock\) using the primary evaluation model\. \(4\) ajoint\-reward floor: candidates whoseRRfalls below the running best minus a small margin are dropped and not considered for refinement\. Empirically, each domain executes∼100\{\\sim\}100proposer API calls of which∼30\{\\sim\}30pass all four gates and reach the acceptance step\.

#### Acceptance rule\.

The current best is replaced iff the new candidate achieves*strict*improvement on the joint rewardRR\(Eq\. \([3](https://arxiv.org/html/2609.30967#S3.E3)\)\), withλs=1\.0\\lambda\_\{s\}\{=\}1\.0andλt\\lambda\_\{t\}scaled so a 1k\-token reduction is worth 1pp of accuracy \(λt=10−5\\lambda\_\{t\}=10^\{\-5\}per token inrun\_search\_v3\.py; the multi\-objective aggregation isR=acc−λ⁡\(tokens/τ\)R=\\mathrm\{acc\}\-\\lambda\\,\(\\mathrm\{tokens\}/\\tau\)withλ=0\.1\\lambda\{=\}0\.1,τ=1000\\tau\{=\}1000in the joint\-scalarization path ofscripts/run\_search\_joint\.py\)\. Ties onRRare broken by lower token count\. The proposer is not compressed down to the scalarRR: it reads all three raw axes plus per\-example traces, and only the search\-time ranking is scalarized\.

#### Why single\-phase matters \(concrete\)\.

In the 2\-phase ablation, Phase 1 optimizesaccuracy\+λs​safety\\mathrm\{accuracy\}\+\\lambda\_\{s\}\\mathrm\{safety\}and then*freezes*the harness structure; Phase 2 \(proposer\_token\_skill\.md\) can only*prune*that structure subject to a22pp accuracy tolerance\. This locks in whatever verbose chain\-of\-thought pattern Phase 1 converged on: on fact verification the Phase\-1 harness is a two\-call decompose\-then\-verify at∼1483\{\\sim\}1483tokens/example, and Phase 2 cannot reach the alternative single\-pass confidence\-gated verdict \(∼536\{\\sim\}536tokens/example\) that MoMHa’s single\-phase joint search discovers, because that structure is outside Phase 2’s mutation neighborhood\. The overall effect on the ten\-domain mean is a\+2\.7\+2\.7joint\-score\-point gain for MoMHa over 2\-phase \(0\.6110\.611vs\.0\.5840\.584\) at9595fewer tokens per example \(572572vs\.667667;[AppendixK](https://arxiv.org/html/2609.30967#A11)\)\.

#### What the proposer discovers vs\. what the skill files encode\.

The skill files \(skills/proposer\_skill\.md,skills/proposer\_<d\>\.md,skills/safety\_<d\>\.md\) encode the*search space*: the harness contract \(must defineHarness\.\_\_init\_\_\(client, config\)andrun\(example\)→\\to\{"prediction": \.\.\.\}\), the LLM\-client call convention, the domain’s refusal templates and output\-format constraints, the allowed and banned symbol lists at that domain’s risk level \(strict / minimal / light\), and the finite strategy library\{draft\_verify,subject\_routing,…\}\\\{\\texttt\{draft\\\_verify\},\\texttt\{subject\\\_routing\},\\ldots\\\}that scopes the mutation step\. The proposer discovers the actual*composition*: which subset of strategies to combine, in what order, with what prompt\-template text, at what threshold parameters, and with which retry/back\-off cap\. Two evidence points support this decomposition\. First, converged harnesses on the same domain but different eval models instantiate the same skill\-library strategy with different code \([AppendixG](https://arxiv.org/html/2609.30967#A7)\)\. Second, the MoMHa\-ns ablation \(v3nsin[Table3](https://arxiv.org/html/2609.30967#S4.T3)\) strips the per\-domain safety skill file and keeps everything else fixed; this degrades the ten\-domain mean joint score from0\.4720\.472\(MoMHa\) to0\.4070\.407\(MoMHa\-ns\), a−0\.065\-0\.065\-point drop attributable purely to the domain\-specific safety knowledge the file encodes\. The skill\-file ablation \([Table15](https://arxiv.org/html/2609.30967#A14.T15),[AppendixN](https://arxiv.org/html/2609.30967#A14)\) further strips*both*the safety and the strategy\-library skill files back to a framework\-only stub onsql\_generationandsafety\_autonomous\_physical, isolating the additional contribution of prior encoded strategy names; the joint\-score deltas quantify how much of MoMHa’s gain is proposer discovery vs\. prior encoding\.

#### Harness contract \(scope\)\.

Concretely, a harness is a Python class exposingHarness\.\_\_init\_\_\(client, config\)andrun\(example\)→\\to\{"prediction": \.\.\., "tokens\_used": \.\.\.\}, validated bycore/validator\.pybefore any real API call\. It*can*: build, rewrite, or reject prompts; route calls across models or prompt variants; retry with modified inputs; parse, redact, or reject outputs; and invoke sub\-harnesses in a loop\. It*cannot*: modify model weights; escape the sandbox \(core/sandbox\.py: 30 s per call, 100k tokens, 200 API calls, 300 s wall clock,/tmpCWD\); or exceed the AST\-guard’s per\-domain import/call whitelist \([Section3\.3](https://arxiv.org/html/2609.30967#S3.SS3)\)\.

## NeurIPS Paper Checklist

1. 1\.Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer:\[Yes\]
4. Justification: All headline claims, MoMHa’s superior joint score on 17 domains, the advantage of joint over staged optimization, cross\-model transfer, and the safety gains from the per\-domain safety skill, are directly supported by[Tables3](https://arxiv.org/html/2609.30967#S4.T3),[2](https://arxiv.org/html/2609.30967#S4.T2)and[4](https://arxiv.org/html/2609.30967#S4.T4)and the ablation tables in the appendix\. Numerical claims in the introduction \(e\.g\.,\+2\.7\+2\.7over 2\-phase,0\.7810\.781safety composite\) match the reported experimental results exactly\.
5. Guidelines: - •The answer\[N/A\]means that the abstract and introduction do not include the claims made in the paper\. - •The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations\. A\[No\]or\[N/A\]answer to this question will not be perceived well by the reviewers\. - •The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings\. - •It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper\.
6. 2\.Limitations
7. Question: Does the paper discuss the limitations of the work performed by the authors?
8. Answer:\[Yes\]
9. Justification:[Section5](https://arxiv.org/html/2609.30967#S5)contains a dedicated “Limitations” paragraph acknowledging: \(i\) capability datasets are LLM\-generated rather than established benchmarks, limiting direct comparability; \(ii\) the search process requires∼100\{\\sim\}100harness evaluations per domain, incurring non\-trivial API cost; \(iii\) cross\-model transfer degrades when models differ substantially in output\-format conventions; and \(iv\) results may differ with alternative proposer systems beyondClaude Code\.
10. Guidelines: - •The answer\[N/A\]means that the paper has no limitation while the answer\[No\]means that the paper has limitations, but those are not discussed in the paper\. - •The authors are encouraged to create a separate “Limitations” section in their paper\. - •The paper should point out any strong assumptions and how robust the results are to violations of these assumptions \(e\.g\., independence assumptions, noiseless settings, model well\-specification, asymptotic approximations only holding locally\)\. The authors should reflect on how these assumptions might be violated in practice and what the implications would be\. - •The authors should reflect on the scope of the claims made, e\.g\., if the approach was only tested on a few datasets or with a few runs\. In general, empirical results often depend on implicit assumptions, which should be articulated\. - •The authors should reflect on the factors that influence the performance of the approach\. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting\. Or a speech\-to\-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon\. - •The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size\. - •If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness\. - •While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper\. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community\. Reviewers will be specifically instructed to not penalize honesty concerning limitations\.
11. 3\.Theory assumptions and proofs
12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete \(and correct\) proof?
13. Answer:\[N/A\]
14. Justification: The paper is empirical and contains no theorems or formal proofs\.[Equations1](https://arxiv.org/html/2609.30967#S3.E1),[2](https://arxiv.org/html/2609.30967#S3.E2)and[3](https://arxiv.org/html/2609.30967#S3.E3)define the optimization objectives and the joint evaluation metric; these are design choices stated without theoretical claims\.
15. Guidelines: - •The answer\[N/A\]means that the paper does not include theoretical results\. - •All the theorems, formulas, and proofs in the paper should be numbered and cross\-referenced\. - •All assumptions should be clearly stated or referenced in the statement of any theorems\. - •The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition\. - •Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material\. - •Theorems and Lemmas that the proof relies upon should be properly referenced\.
16. 4\.Experimental result reproducibility
17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper \(regardless of whether the code and data are provided or not\)?
18. Answer:\[Yes\]
19. Justification:[Sections3](https://arxiv.org/html/2609.30967#S3)and[4\.1](https://arxiv.org/html/2609.30967#S4.SS1)specify the full system architecture, reward functions, data splits \(50 search / 50 test\), search budget \(∼100\{\\sim\}100evaluations per domain\), the three\-layer safety stack, skill\-composition procedure, and the 12\-model evaluation fleet\. Appendix[AppendicesD](https://arxiv.org/html/2609.30967#A4),[E](https://arxiv.org/html/2609.30967#A5)and[K](https://arxiv.org/html/2609.30967#A11)provide per\-domain and per\-model breakdowns\. All harness source, search logs, per\-example traces, scoring artifacts, and evaluation JSONs are released as supplemental material\.
20. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •If the paper includes experiments, a\[No\]answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not\. - •If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable\. - •Depending on the contribution, reproducibility can be accomplished in various ways\. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model\. In general\. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model \(e\.g\., in the case of a large language model\), releasing of a model checkpoint, or other means that are appropriate to the research performed\. - •While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution\. For example 1. \(a\)If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm\. 2. \(b\)If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully\. 3. \(c\)If the contribution is a new model \(e\.g\., a large language model\), then there should either be a way to access this model for reproducing the results or a way to reproduce the model \(e\.g\., with an open\-source dataset or instructions for how to construct the dataset\)\. 4. \(d\)We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility\. In the case of closed\-source models, it may be that access to the model is limited in some way \(e\.g\., to registered users\), but it should be possible for other researchers to have some path to reproducing or verifying the results\.
21. 5\.Open access to data and code
22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
23. Answer:\[Yes\]
24. Justification: We will release all harness source code, per\-domain proposer and safety skill files, search logs, per\-example execution traces, scoring artifacts, and cross\-model evaluation JSONs as supplemental material\. Anonymized versions are provided at submission time\. The seven real\-world benchmark domains use publicly available HuggingFace datasets \(LawBench, NuminaMath, FEVER, Spider, HumanEval, MBPP, MMLU\-Pro\) cited in the paper\.
25. Guidelines: - •The answer\[N/A\]means that paper does not include experiments requiring code\. - • - •While we encourage the release of code and data, we understand that this might not be possible, so\[No\]is an acceptable answer\. Papers cannot be rejected simply for not including code, unless this is central to the contribution \(e\.g\., for a new open\-source benchmark\)\. - •The instructions should contain the exact command and environment needed to run to reproduce the results\. See the NeurIPS code and data submission guidelines \([https://neurips\.cc/public/guides/CodeSubmissionPolicy](https://neurips.cc/public/guides/CodeSubmissionPolicy)\) for more details\. - •The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc\. - •The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines\. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why\. - •At submission time, to preserve anonymity, the authors should release anonymized versions \(if applicable\)\. - •Providing as much information as possible in supplemental material \(appended to the paper\) is recommended, but including URLs to data and code is permitted\.
26. 6\.Experimental setting/details
27. Question: Does the paper specify all the training and test details \(e\.g\., data splits, hyperparameters, how they were chosen, type of optimizer\) necessary to understand the results?
28. Answer:\[Yes\]
29. Justification:[Section4\.1](https://arxiv.org/html/2609.30967#S4.SS1)reports data splits \(50 search / 50 test per capability domain\), search budget \(∼100\{\\sim\}100evaluations; 724 total\), baseline configurations \(APE: 5 rounds×\\times5 candidates; DSPy: BootstrapFewShot; MIPROv2: TPE surrogate\), reward\-function definitions for each domain type, the joint metricJJ\([Equation4](https://arxiv.org/html/2609.30967#S4.E4)\), and the LLM\-as\-judge scoring protocol for safety domains\. Harness discovery uses Claude Haiku 4\.5 for all variants\. Full token\-budget details are in[AppendixD](https://arxiv.org/html/2609.30967#A4)\.
30. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them\. - •The full details can be provided either with the code, in appendix, or as supplemental material\.
31. 7\.Experiment statistical significance
32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
33. Answer:\[Yes\]
34. Justification: All cells in the six evaluation tables \([Tables2](https://arxiv.org/html/2609.30967#S4.T2),[3](https://arxiv.org/html/2609.30967#S4.T3),[4](https://arxiv.org/html/2609.30967#S4.T4),[8](https://arxiv.org/html/2609.30967#A5.T8),[9](https://arxiv.org/html/2609.30967#A6.T9)and[7](https://arxiv.org/html/2609.30967#A4.T7)\) report mean±SEM\{\}\_\{\\pm\\text\{SEM\}\}, where SEM is a 1\-σ\\sigmastandard error of the mean computed asσ/Ntrials\\sigma/\\sqrt\{N\_\{\\text\{trials\}\}\}\.
35. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The authors should answer\[Yes\]if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper\. - •The factors of variability that the error bars are capturing should be clearly stated \(for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions\)\. - •The method for calculating the error bars should be explained \(closed form formula, call to a library function, bootstrap, etc\.\) - •The assumptions made should be given \(e\.g\., Normally distributed errors\)\. - •It should be clear whether the error bar is the standard deviation or the standard error of the mean\. - •It is OK to report 1\-sigma error bars, but one should state it\. The authors should preferably report a 2\-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified\. - •For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range \(e\.g\., negative error rates\)\. - •If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text\.
36. 8\.Experiments compute resources
37. Question: For each experiment, does the paper provide sufficient information on the computer resources \(type of compute workers, memory, time of execution\) needed to reproduce the experiments?
38. Answer:\[Yes\]
39. Justification: The paper reports the search budget \(∼100\{\\sim\}100harness evaluations per domain, 724 total across all domains\) and per\-example token counts for each variant \(e\.g\., MoMHa averages 672 tokens, DSPy 2,229 tokens on capability domains\)\. All compute is API\-based \(no local GPU required\)\.
40. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage\. - •The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute\. - •The paper should disclose whether the full research project required more compute than the experiments reported in the paper \(e\.g\., preliminary or failed experiments that didn’t make it into the paper\)\.
41. 9\.Code of ethics
43. Answer:\[Yes\]
44. Justification: The paper studies automated harness optimization with safety as a first\-class objective\. The three\-layer safety architecture \(AST\-based static analysis, domain\-specific safety skills, sandboxed execution\) is designed to prevent generated code from executing unsafe operations\. All model API usage follows the respective providers’ terms of service\. No human subjects are involved\.
45. Guidelines: - •The answer\[N/A\]means that the authors have not reviewed the NeurIPS Code of Ethics\. - •If the authors answer\[No\], they should explain the special circumstances that require a deviation from the Code of Ethics\. - •The authors should make sure to preserve anonymity \(e\.g\., if there is a special consideration due to laws or regulations in their jurisdiction\)\.
46. 10\.Broader impacts
47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
48. Answer:\[Yes\]
49. Justification: The positive impact, reducing unsafe LLM outputs via behavioral safety as an explicit optimization objective, is discussed throughout[Sections3\.3](https://arxiv.org/html/2609.30967#S3.SS3),[4\.5](https://arxiv.org/html/2609.30967#S4.SS5)and[5](https://arxiv.org/html/2609.30967#S5)\. A potential negative impact is that the framework could be used to search for adversarial harnesses\. We note two mitigations: \(1\) the AST guard and sandbox prevent generated harnesses from executing filesystem or network operations*during search*; \(2\) discovered harnesses are inspectable Python files subject to the same review process as any other code\. However, neither protection applies if a malicious actor deliberately deploys a discovered harness in an unrestricted environment; this residual risk is acknowledged\.
50. Guidelines: - •The answer\[N/A\]means that there is no societal impact of the work performed\. - •If the authors answer\[N/A\]or\[No\], they should explain why their work has no societal impact or why the paper does not address societal impact\. - •Examples of negative societal impacts include potential malicious or unintended uses \(e\.g\., disinformation, generating fake profiles, surveillance\), fairness considerations \(e\.g\., deployment of technologies that could make decisions that unfairly impact specific groups\), privacy considerations, and security considerations\. - •The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments\. However, if there is a direct path to any negative applications, the authors should point it out\. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation\. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster\. - •The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from \(intentional or unintentional\) misuse of the technology\. - •If there are negative societal impacts, the authors could also discuss possible mitigation strategies \(e\.g\., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML\)\.
51. 11\.Safeguards
52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse \(e\.g\., pre\-trained language models, image generators, or scraped datasets\)?
53. Answer:\[Yes\]
54. Justification:[Section3\.3](https://arxiv.org/html/2609.30967#S3.SS3)describes the three\-layer safety system built intoMeta\-Harness: \(i\) AST\-based static analysis \(AST\-Guard\) that rejects any harness importing banned modules or calling dangerous builtins; \(ii\) per\-domain safety skills with calibrated import whitelists; and \(iii\) sandboxed execution with 30\-second timeouts, 100K token limits, and an empty environment\. Released harnesses have all passedAST\-Guardand cannot perform filesystem or network operations\.
55. Guidelines: - •The answer\[N/A\]means that the paper poses no such risks\. - •Released models that have a high risk for misuse or dual\-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters\. - •Datasets that have been scraped from the Internet could pose safety risks\. The authors should describe how they avoided releasing unsafe images\. - •We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort\.
56. 12\.Licenses for existing assets
57. Question: Are the creators or original owners of assets \(e\.g\., code, data, models\), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
58. Answer:\[Yes\]
59. Justification: All baseline methods \(APE, OPRO, DSPy, MIPROv2, TextGrad, GEPA, MH, AH\) and datasets \(U\-SafeBench, LawBench, NuminaMath, FEVER, Spider, HumanEval, MBPP, MMLU\-Pro\) are cited via their original publications\. The seven real\-world benchmarks are sourced from HuggingFace under their respective open licenses\. The LLM\-as\-judge uses Claude Sonnet 4\.6 via the Anthropic API under standard commercial terms\.
60. Guidelines: - •The answer\[N/A\]means that the paper does not use existing assets\. - •The authors should cite the original paper that produced the code package or dataset\. - •The authors should state which version of the asset is used and, if possible, include a URL\. - •The name of the license \(e\.g\., CC\-BY 4\.0\) should be included for each asset\. - •For scraped data from a particular source \(e\.g\., website\), the copyright and terms of service of that source should be provided\. - •If assets are released, the license, copyright information, and terms of use in the package should be provided\. For popular datasets,[paperswithcode\.com/datasets](https://paperswithcode.com/datasets)has curated licenses for some datasets\. Their licensing guide can help determine the license of a dataset\. - •For existing datasets that are re\-packaged, both the original license and the license of the derived asset \(if it has changed\) should be provided\. - •If this information is not available online, the authors are encouraged to reach out to the asset’s creators\.
61. 13\.New assets
62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
63. Answer:\[Yes\]
64. Justification: The paper will release: \(i\) all harness source files \(domains/<d\>/harnesses/candidate\_\*\.py\); \(ii\) per\-domain proposer skill files \(skills/proposer\_<domain\>\.md\) and safety skill files \(skills/safety\_<domain\>\.md\); \(iii\) search logs, per\-example execution traces \(traces\.jsonl\), and scoring artifacts \(scores\.json\); and \(iv\) cross\-model evaluation JSONs\. Structure and schema are documented in[Section3\.2](https://arxiv.org/html/2609.30967#S3.SS2)and the appendix\.
65. Guidelines: - •The answer\[N/A\]means that the paper does not release new assets\. - •Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates\. This includes details about training, license, limitations, etc\. - •The paper should discuss whether and how consent was obtained from people whose asset is used\. - •At submission time, remember to anonymize your assets \(if applicable\)\. You can either create an anonymized URL or include an anonymized zip file\.
66. 14\.Crowdsourcing and research with human subjects
67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation \(if any\)?
68. Answer:\[N/A\]
69. Justification: No crowdsourcing or human subjects are involved\. The capability datasets are LLM\-generated \(verified by a second model\), the safety evaluation uses a Claude Sonnet 4\.6 LLM\-as\-judge, and baseline/MoMHa evaluations are fully automated\.
70. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper\. - •According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector\.
71. 15\.Institutional review board \(IRB\) approvals or equivalent for research with human subjects
72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board \(IRB\) approvals \(or an equivalent approval/review based on the requirements of your country or institution\) were obtained?
73. Answer:\[N/A\]
74. Justification: No human subjects are involved in any part of this work; IRB approval is therefore not required\.
75. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Depending on the country in which research is conducted, IRB approval \(or equivalent\) may be required for any human subjects research\. If you obtained IRB approval, you should clearly state this in the paper\. - •We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution\. - •For initial submissions, do not include any information that would break anonymity \(if applicable\), such as the institution conducting the review\.
76. 16\.Declaration of LLM usage
77. Question: Does the paper describe the usage of LLMs if it is an important, original, or non\-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does*not*impact the core methodology, scientific rigor, or originality of the research, declaration is not required\.
78. Answer:\[Yes\]
79. Justification: LLMs are a central component of the method\.[Section3\.2](https://arxiv.org/html/2609.30967#S3.SS2)describes: \(i\)Claude Codeas the agentic proposer that reads filesystem artifacts and writes new harness candidates; \(ii\) Claude Haiku 4\.5 as the model used during harness search; \(iii\) Claude Sonnet 4\.6 as the LLM\-as\-judge for U\-SafeBench safety scoring\. The cross\-model evaluation \([Section4\.4](https://arxiv.org/html/2609.30967#S4.SS4)\) runs discovered harnesses on 12 target models from four families \(Claude, GPT, Gemini, DeepSeek\-R1\), all described in[Table1](https://arxiv.org/html/2609.30967#S4.T1)\.
80. Guidelines: - •The answer\[N/A\]means that the core method development in this research does not involve LLMs as any important, original, or non\-standard components\. - •Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described\.