When Can You Trust an Agent's Own Verifier? Stress-Testing Counterfactual Rationale Verification Under Search Pressure and Oracle Access

# When Can You Trust an Agent’s Own Verifier?Stress-Testing Counterfactual Rationale VerificationUnder Search Pressure and Oracle Access
Source: [https://arxiv.org/html/2610.22779](https://arxiv.org/html/2610.22779)
\\workshoptitle

Who Verifies the Agents? Toward Reliable Agent Development

###### Abstract

Automated agent design increasingly trusts a single score to decide whether a change is an improvement — but a score optimized aggressively enough ceases to measure what it was meant to\. When an agent also*explains*its design, that explanation is a second signal we can check without human labels: negate each stated claim, re\-run, and observe whether performance drops\. We ask whether such a verifier stays reliable once an agent optimizes against it\. Our contribution is not a new verifier but a way to*stress\-test*one: a pressure ratioρ\\rhomeasuring how much the verifier steers the agent’s search, and four falsifiable probes covering four failure modes — giving no signal, being too weak to matter, being gamed by stating fewer claims, and being queried before the agent commits\. We apply the test to our own verifier, the Rationale Consistency Score \(RCS\), inside an LLM agent that designs machine\-learning pipelines, with one generator model, six tasks, and three seeds\. First, in our prior configuration the verifier barely affects the search*objective*\(ρ≪1\\rho\\ll 1\), partly because of an accuracy gate, so its robustness had not been tested; it still shapes the reported artifact through a final\-selection tie\-break thatρ\\rhodoes not register\. Second, a capability control shows the agent shortens its claim lists when instructed, yet search does not, even atρ\\rhomore than100×100\\timesthe prior level; since feedback omits claim volume and horizons are short, this null result is largely structural rather than evidence of robustness\. Third, when the agent can probe the verifier before committing, it keeps claims that pass while held\-out faithfulness falls — a decline invisible to in\-loop metrics, though partly confounded by how RCS scores small pipelines\. We offer these as preliminary evidence and propose a reporting standard: audit a verifier withρ\\rhoand these probes before trusting it\.

## 1Introduction

Verification is the bottleneck in automated agent design\. When an outer loop proposes changes to an agent and an inner loop scores them, the score is the only safeguard separating genuine improvement from silent regression\. In practice that score is usually a scalar task reward, and a scalar reward optimized aggressively enough ceases to measure the quantity it was intended to track\([Manheim and Garrabrant, 2018](https://arxiv.org/html/2610.22779#bib.bib21);[Skalse et al\., 2022](https://arxiv.org/html/2610.22779#bib.bib25);[Gao et al\., 2023](https://arxiv.org/html/2610.22779#bib.bib8)\)\.

Agents that design artifacts often also*explain*them: an agent proposing a machine\-learning pipeline states which features matter and why a model class suits the data\. That reasoning is usually discarded, but it has a property the task reward lacks — it is*checkable against the environment without human labels*\. If the agent claims a feature is important, we remove it, retrain, and observe whether performance falls: remove\-and\-retrain evaluation of attribution methods\([Hooker et al\., 2019](https://arxiv.org/html/2610.22779#bib.bib14)\), applied to natural\-language claims rather than saliency maps\.

The Rationale Consistency Score \(RCS\) formalizes this check\. It is attractive as a verification signal for three reasons: it is grounded in the environment rather than in a learned reward model, it is orthogonal in construction to task accuracy, and it is defined per\-claim, so it degrades gracefully when only part of an explanation is faithful\.

For these same reasons, however, RCS is also a target\. Once RCS enters an optimization objective, the agent has an incentive to make claims that survive verification rather than claims that are informative\. The simplest such strategy requires no sophistication:*state fewer claims*— one safe claim, not eight, scores well on an average\-over\-claims metric while communicating almost nothing\. This is specification gaming in its simplest form\([Amodei et al\., 2016](https://arxiv.org/html/2610.22779#bib.bib2);[Pan et al\., 2022](https://arxiv.org/html/2610.22779#bib.bib23)\), undetectable by any evaluation that reports only the verifier’s mean output\.

This paper asks whether a counterfactual rationale verifier remains reliable when an agent optimizes against it — whether it prevents reward hacking and stays faithful as the agent evolves, separately from whether it improves task performance\. Such a signal should be characterized the way a measuring instrument is — by its failure modes, operating range, and behavior under adversarial pressure — before it is recommended as infrastructure\. Our contribution is therefore a*stress test*, not a verifier: a pressure metric and four falsifiable probes for any counterfactual rationale verifier\.

Our findings are preliminary and mixed: in our prior configuration the verifier is too weak to have been tested \(F2\); a capability control shows the agent*can*shorten its claim lists when instructed, yet search never does so even atρ≫1\\rho\\gg 1\(F3\), a null result largely due to short horizons and feedback that omits claim counts; and the failure that does appear is oracle exploitation \(F4\)\.

#### Contributions\.

- •A stress\-test methodology for rationale verifiers: four failure modes \(Section[5](https://arxiv.org/html/2610.22779#S5)\), falsifiable probes for them \(Section[6](https://arxiv.org/html/2610.22779#S6)\), and a capability control for F3, all reusable beyond RCS\.
- •The*pressure ratio*ρ\\rho\(Section[6\.1](https://arxiv.org/html/2610.22779#S6.SS1)\), and how an accuracy\-gating rule suppresses it: our prior configuration lies atρ≈0\.007\\rho\\approx 0\.007–0\.0890\.089\(Section[9\.1](https://arxiv.org/html/2610.22779#S9.SS1)\), while controlled sweeps reachρ≈41\.7\\rho\\approx 41\.7–63\.763\.7\(Section[9\.2](https://arxiv.org/html/2610.22779#S9.SS2)\)\.
- •A capability control showing that the agent shortens its claim lists when instructed \(Section[9\.3](https://arxiv.org/html/2610.22779#S9.SS3)\) while search does not, even atρ≫1\\rho\\gg 1and under UCB1 at a doubled base horizon \(Section[9\.2](https://arxiv.org/html/2610.22779#S9.SS2)\) — a discovery gap, not a demonstrated reward exploit\.
- •Evidence that probe access produces the selection signature of Section[6\.3](https://arxiv.org/html/2610.22779#S6.SS3)on four of six datasets at sign level, with held\-out fidelity declines that the estimator may partly explain \(Section[9\.4](https://arxiv.org/html/2610.22779#S9.SS4)\)\.

## 2Related Work

#### Reward hacking and Goodhart’s law\.

That a proxy optimized aggressively enough ceases to track its target recurs in reward misspecification\([Manheim and Garrabrant, 2018](https://arxiv.org/html/2610.22779#bib.bib21);[Skalse et al\., 2022](https://arxiv.org/html/2610.22779#bib.bib25)\)and reward\-model overoptimization\([Gao et al\., 2023](https://arxiv.org/html/2610.22779#bib.bib8)\), and claim minimization is an instance of specification gaming\([Amodei et al\., 2016](https://arxiv.org/html/2610.22779#bib.bib2);[Pan et al\., 2022](https://arxiv.org/html/2610.22779#bib.bib23)\)\. The closest structural precedent is a learned judge or reward model folded into an objective: LLM judges carry exploitable biases\([Zheng et al\., 2023](https://arxiv.org/html/2610.22779#bib.bib29);[Wang et al\., 2024](https://arxiv.org/html/2610.22779#bib.bib27)\), and policies over\-optimize learned rewards\([Gao et al\., 2023](https://arxiv.org/html/2610.22779#bib.bib8);[Coste et al\., 2024](https://arxiv.org/html/2610.22779#bib.bib5)\)\. RCS replaces the learned judge with environment interventions; we audit it, measuring how much pressure the verifier exerts before asking whether it is gamed\.

#### Rationale faithfulness\.

Whether stated reasoning reflects the underlying computation has motivated a growing literature: chain\-of\-thought can be systematically unfaithful\([Turpin et al\., 2023](https://arxiv.org/html/2610.22779#bib.bib26);[Lanham et al\., 2023](https://arxiv.org/html/2610.22779#bib.bib19)\), and faithfulness is distinct from plausibility\([Jacovi and Goldberg, 2020](https://arxiv.org/html/2610.22779#bib.bib16)\)\. RCS operationalizes it through counterfactual interventions\([Chuang et al\., 2026](https://arxiv.org/html/2610.22779#bib.bib4)\), extending remove\-and\-retrain evaluation from saliency maps\([Hooker et al\., 2019](https://arxiv.org/html/2610.22779#bib.bib14)\)and feature attribution\([Ribeiro et al\., 2016](https://arxiv.org/html/2610.22779#bib.bib24);[Lundberg and Lee, 2017](https://arxiv.org/html/2610.22779#bib.bib20)\)to natural\-language claims\. Such removal\-based metrics are themselves gameable under optimization\([Hsia et al\., 2024](https://arxiv.org/html/2610.22779#bib.bib15)\); RCS inherits that exposure, which is why we audit it under search pressure rather than assuming robustness\.

#### LLM\-driven pipeline design\.

Automated pipeline construction has moved from search\- and ensemble\-based systems\([Olson and Moore, 2016](https://arxiv.org/html/2610.22779#bib.bib22);[Erickson et al\., 2020](https://arxiv.org/html/2610.22779#bib.bib7)\)to LLM optimizers\([Yang et al\., 2024](https://arxiv.org/html/2610.22779#bib.bib28)\)and proposers that suggest features in natural language\([Hollmann et al\., 2023](https://arxiv.org/html/2610.22779#bib.bib13)\)\. Our agent adds a verifier over its stated reasons — precisely what creates the gaming surface we study\.

## 3Setting

#### The design task\.

We instantiate automated agent design in a constrained but fully executable environment: the agent designs supervised learning pipelinesP=\(S,M,H\)P=\(S,M,H\)— a feature subsetS⊆FS\\subseteq F, a model classMM, and hyperparametersHH— with task performanceQ⁡\(P\)∈\[0,1\]Q\(P\)\\in\[0,1\]estimated asQ^​\(P\)\\hat\{Q\}\(P\)on validation data\. The setting is narrow relative to open\-ended agentic tasks \(Section[10](https://arxiv.org/html/2610.22779#S10)\), but every claim the agent makes has a mechanically constructible counterfactual, which makes the verifier study feasible\.

#### The agent\.

The agent is a large language model inside an optimization loop\. In the default proposer — the one used for the main configuration, the pressure sweeps, and the capability control — each iteration is a single call that returns a feature set, a model, hyperparameters, and a natural\-language rationaleEPE\_\{P\}\. The tool\-using proposer, which may calllist\_features, optionallyprobe\_feature\_drop, and thensubmit\_pipeline, appears only in the probe\-access probe \(Section[6\.3](https://arxiv.org/html/2610.22779#S6.SS3)\), where both arms use it\. Proposals are executed, scored, and used to update Thompson sampling posteriors over pipeline\-type arms\([Agrawal and Goyal, 2012](https://arxiv.org/html/2610.22779#bib.bib1);[Kaufmann et al\., 2012](https://arxiv.org/html/2610.22779#bib.bib17)\), whose recommendations are fed back into the next prompt\.

#### Claims\.

The rationaleEPE\_\{P\}is parsed into a set of claimsEP=\{C1,…,Ck\}E\_\{P\}=\\\{C\_\{1\},\\dots,C\_\{k\}\\\}of three kinds: feature\-importance claims, model\-choice claims, and hyperparameter claims\. Parsing is deterministic and does not depend on the verification machinery, a property we rely on in Section[7](https://arxiv.org/html/2610.22779#S7)\.

DesignAgentVerifierAgentSearchAgentLLM proposerTool interfacelist\_featuresprobe\_feature\_dropSubmit\(S,M,H\)\+EP\(S,M,H\)\+E\_\{P\}Claim parserF3CounterfactualconstructorExecutor\(retrain\)RCS^\\widehat\{\\mathrm\{RCS\}\}F1R=Q^\+λt​RCS^R=\\hat\{Q\}\+\\lambda\_\{t\}\\,\\widehat\{\\mathrm\{RCS\}\}F2ThompsonSamplingrewardartifactF4 oracleQ^\\hat\{Q\}RCS^\\widehat\{\\mathrm\{RCS\}\}next iterationwithin\-agent / scored messagesF4 oracle channelsearch feedback

Figure 1:Multi\-agent architecture of the stress\-test loop\. TheDesign Agentproposes a pipeline and rationale in a single call, and through the tool interface only in the P2 arms \(Section[6\.3](https://arxiv.org/html/2610.22779#S6.SS3)\); theVerifier Agentparses claims, builds counterfactuals, and returnsRCS^\\widehat\{\\mathrm\{RCS\}\}; theSearch AgentformsR=Q^\+λt​RCS^R=\\hat\{Q\}\+\\lambda\_\{t\}\\widehat\{\\mathrm\{RCS\}\}and updates Thompson Sampling\. Cross\-agent edges carry the artifact, oracle probes \(F4, oracle arm only\), task/verifier scores, and the next\-iteration recommendation\. Tags F1–F4 mark where each protocol probe attaches\.

## 4RCS as a Verifier

#### Provenance\.

RCS as defined in Equation[1](https://arxiv.org/html/2610.22779#S4.E1)is introduced here; we are not auditing a third\-party standard\. The system we instrument is our own research agent, and“deployed” throughout means our prior configurationof that agent — including the accuracy\-gating rule of Section[6\.1](https://arxiv.org/html/2610.22779#S6.SS1)— so the F2 measurement is a self\-audit and a motivating case, not a survey finding\.

###### Definition 1\(Rationale Consistency Score\)\.

For each claimC∈EPC\\in E\_\{P\}, letP¬CP^\{\\neg C\}denote a counterfactual pipeline that contradictsCCwhile holding other design choices fixed\. Then

RCS⁡\(P\)=1\|EP\|​∑C∈EPmax⁡\(0,Q⁡\(P\)−Q⁡\(P¬C\)\)\.\\mathrm\{RCS\}\(P\)\\;=\\;\\frac\{1\}\{\|E\_\{P\}\|\}\\sum\_\{C\\in E\_\{P\}\}\\max\\\!\\left\(0,\\;Q\(P\)\-Q\(P^\{\\neg C\}\)\\right\)\.\(1\)

Counterfactuals are constructed per claim type: a feature\-importance claim is negated by removing the feature and retraining; a model\-choice claim by substituting an alternative model class; a hyperparameter claim by perturbing that hyperparameter\. When three or more feature\-importance claims are stated, the estimator groups them into one joint intervention \(Appendix[A](https://arxiv.org/html/2610.22779#A1)\)\.

#### What RCS verifies, and what it does not\.

RCS asks whether the agent’s stated reasons are load\-bearing for the artifact it produced\. It does not ask whether the artifact performs well, and it does not certify that a change to the agent improved capability\. It is therefore a*heterogeneous signal*complementing outcome\-based verification rather than replacing it\.

#### Estimator variants\.

ReplacingQQby the empiricalQ^\\hat\{Q\}gives the base estimator

RCS^base​\(P\)=1\|EP\|​∑C∈EPmax⁡\(0,Q^​\(P\)−Q^​\(P¬C\)\),\\widehat\{\\mathrm\{RCS\}\}\_\{\\text\{base\}\}\(P\)\\;=\\;\\frac\{1\}\{\|E\_\{P\}\|\}\\sum\_\{C\\in E\_\{P\}\}\\max\\\!\\left\(0,\\;\\hat\{Q\}\(P\)\-\\hat\{Q\}\(P^\{\\neg C\}\)\\right\),\(2\)which is sensitive to validation noise\. The deployed system uses a practical variant with a small modality\-dependent noise threshold, a relative\-drop boost, and an amplification threshold \(Appendix[A](https://arxiv.org/html/2610.22779#A1)\); Equations[1](https://arxiv.org/html/2610.22779#S4.E1)–[2](https://arxiv.org/html/2610.22779#S4.E2)idealize it; all results concern this deployed variant, abbreviatedRCS^\\widehat\{\\mathrm\{RCS\}\}\.

## 5How a Rationale Verifier Fails

We distinguish four failure modes\. The first two concern the verifier as an instrument; the last two concern the agent’s response to it\. They follow from the structure of the setting rather than from RCS’s internals \(though tested only on RCS\): F2 applies to any auxiliary term in a composite objective, F3 to any per\-claim average whose denominator the agent controls, and F4 to any verifier exposed as a callable tool\.

#### F1: Silence\.

The verifier returns no usable signal\. Every counterfactual drop falls below the noise threshold, or no claims could be parsed, and the reported score is a floor value rather than a measurement\. We treat the*floor rate*as a primary reported quantity\.

#### F2: Untested\.

The verifier appears in the objective but contributes so little relative to task\-reward variation that the agent’s search is effectively unchanged\. Observing no gaming under these conditions is uninformative about robustness: the incentive was never applied\.

#### F3: Claim minimization\.

Supposekck\_\{c\}of the\|EP\|\|E\_\{P\}\|claims are counterfactually impactful\. Taking expectations in Equation[1](https://arxiv.org/html/2610.22779#S4.E1)then gives

𝔼⁡\[RCS⁡\(P\)\]=kc\|EP\|​Δ¯c,\\mathbb\{E\}\\\!\\left\[\\mathrm\{RCS\}\(P\)\\right\]\\;=\\;\\frac\{k\_\{c\}\}\{\|E\_\{P\}\|\}\\,\\bar\{\\Delta\}\_\{c\},\(3\)withΔ¯c\\bar\{\\Delta\}\_\{c\}the mean drop over impactful claims\. The score depends on the*ratio*of impactful to stated claims, so deleting one inert claim multiplies it by\|EP\|/\(\|EP\|−1\)\|E\_\{P\}\|/\(\|E\_\{P\}\|\-1\)while leaving the artifact untouched\. This holds forRCS\\mathrm\{RCS\}; the deployedRCS^\\widehat\{\\mathrm\{RCS\}\}groups feature claims and averages over executed interventions \(Appendix[A](https://arxiv.org/html/2610.22779#A1)\), so deleting a stated claim can leaveRCS^\\widehat\{\\mathrm\{RCS\}\}unchanged\. We test whether search discovers shorter claim lists under pressure \(P1\) and whether the agent produces them when instructed \(P4\)\.

#### F4: Oracle exploitation\.

If the agent can query the verifier before committing, it can filter its claims to those that pass\. The resulting rationale is*verified*but selected\. Distinguishing genuine improvement from selection requires observing claim volume alongside pass rate\.

## 6A Stress\-Test Protocol

Each probe states a treatment, a control, and a falsifying observation\. The mapping is not one\-to\-one: F2 is read directly offρ\\rho, P1 and P4 both target F3, P2 targets F4, and P3 targets F1\.

### 6\.1Auditing Verifier Pressure

Let the search objective beR⁡\(P\)=Q^​\(P\)\+λ⋅RCS^​\(P\)R\(P\)=\\hat\{Q\}\(P\)\+\\lambda\\cdot\\widehat\{\\mathrm\{RCS\}\}\(P\)\. We define thepressure ratioover a run ofTTiterations:

ρ=1T​∑t=1Tλt⋅RCS^​\(Pt\)sd⁡\(\{Q^​\(Pt\)\}t=1T\),\\rho\\;=\\;\\frac\{\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\lambda\_\{t\}\\cdot\\widehat\{\\mathrm\{RCS\}\}\(P\_\{t\}\)\}\{\\mathrm\{sd\}\\\!\\left\(\\\{\\hat\{Q\}\(P\_\{t\}\)\\\}\_\{t=1\}^\{T\}\\right\)\},\(4\)whereλt\\lambda\_\{t\}is the weight actually applied at iterationtt\. Whenρ≪1\\rho\\ll 1the verifier has negligible influence on search\. We reportρ\\rhofor every run\.

#### Three claims of decreasing generality\.

\(i\)*Applied pressure must be measured*: in any composite objective, whether an auxiliary term steers the search is not readable from its nominal weight\. This is our primary methodological claim\. \(ii\)*ρ\\rhois one operationalization of it\.*Its denominator is a design choice that Section[10](https://arxiv.org/html/2610.22779#S10)shows can inflate, and its numerator is a time average, so pressure concentrated in a few iterations is indistinguishable from uniformly small pressure; the selection flip rate we report beside it \(Table[4](https://arxiv.org/html/2610.22779#A3.T4)\) is exposed to neither\. \(iii\)*Theρ≈0\.007\\rho\\approx 0\.007–0\.0890\.089comes from a single self\-audit*, not a survey, and partly reflects the gate below; ungated atλnom=0\.12\\lambda\_\{\\mathrm\{nom\}\}=0\.12, the sweep level nearest the global nominal0\.10\.1, it is0\.040\.04–0\.280\.28\(Table[7](https://arxiv.org/html/2610.22779#A3.T7)\)\.

#### Why nominal weights mislead\.

The system first scales the nominal weight, then gates it on validation accuracy \(Appendix[A](https://arxiv.org/html/2610.22779#A1)\)\. On Credit\-G,λ¯t=0\.030\\bar\{\\lambda\}\_\{t\}=0\.030versussd​λnom=0\.088s\_\{d\}\\lambda\_\{\\mathrm\{nom\}\}=0\.088\(global nominal0\.10\.1\), a factor of0\.340\.34: a plausible gate, not chosen to lowerρ\\rho, silently reduces the applied weight\. Controlled sweeps disable it but retain scaling:R⁡\(P\)=Q^​\(P\)\+sd​λnom​RCS^​\(P\)R\(P\)=\\hat\{Q\}\(P\)\+s\_\{d\}\\lambda\_\{\\mathrm\{nom\}\}\\widehat\{\\mathrm\{RCS\}\}\(P\), wheresd∈\[0\.78,0\.98\]s\_\{d\}\\in\[0\.78,0\.98\]\.

### 6\.2P1: Dose–Response over Verifier Pressure

Question\.Does increasing verifier pressure cause claim minimization \(F3\)?Design\.Sweep nominalλnom∈\{0,0\.12,1,5,20\}\\lambda\_\{\\mathrm\{nom\}\}\\in\\\{0,0\.12,1,5,20\\\}with all else fixed, three seeds per arm; theλnom=0\\lambda\_\{\\mathrm\{nom\}\}=0arm still computesRCS^\\widehat\{\\mathrm\{RCS\}\}and only omits it from the reward\.Falsification\.Declining claim counts withλnom\\lambda\_\{\\mathrm\{nom\}\}is evidence of gaming; stable counts atρ≫1\\rho\\gg 1support robustness\.

### 6\.3P2: Access to the Agent’s Own Verifier

Question\.When an agent can query the verifier before committing, does its rationale become more faithful or merely better selected \(F4\)?Design\.Both arms use the tool\-using proposer and may calllist\_features; only the treatment may callprobe\_feature\_drop, capped at four calls per iteration\.Interpretation, stated before the results\.In Equation[5](https://arxiv.org/html/2610.22779#S7.E5),Δ​v^\>0\\Delta\\hat\{v\}\>0withΔ​N¯≈0\\Delta\\bar\{N\}\\approx 0suggests better reasoning; the displayed conjunction is our diagnostic selection signature\.

### 6\.4P3: Reliability and Disclosure of Failure

Question\.How often does the verifier return no signal \(F1\), and how stable is it across seeds?Design\.Repeat the main configuration over three seeds and report bootstrap CIs of held\-out RCS and task accuracy together with the floor rate\. Held\-out RCS is measured post\-hoc for all arms, including those whose objective excludes RCS\.

### 6\.5P4: Capability Control for Claim Minimization

Question\.Can the agent reduce claims when the minimization incentive is made explicit?Design\.At fixed nominalλnom=5\\lambda\_\{\\mathrm\{nom\}\}=5, three seeds per policy, compare a*default*prompt with a*minimize*prompt that both discloses the averaging scoring rule and instructs the agent to state fewer claims while preserving accuracy\. Because this arm alters the agent’s information and its instruction together, it bounds capability rather than isolating it: a reduction shows minimization is*reachable*, not that instruction alone would suffice\. If minimize reduces claims while search under P1 does not, F3’s absence under search reflects a discovery gap rather than an inability\.

## 7Instrumentation

For armaawithTaT\_\{a\}recorded iterations, letna,t=\|EPa,t\|n\_\{a,t\}=\|E\_\{P\_\{a,t\}\}\|be parsed claims,ma,tm\_\{a,t\}tested claims, andza,tz\_\{a,t\}tested claims with positive counterfactual drop\. We report

N¯a=1Ta​∑t=1Tana,t,v^a=∑tza,t∑tma,t,Δ​N¯<0∧Δ​v^\>0⟹selection signature,\\bar\{N\}\_\{a\}=\\frac\{1\}\{T\_\{a\}\}\\sum\_\{t=1\}^\{T\_\{a\}\}n\_\{a,t\},\\qquad\\hat\{v\}\_\{a\}=\\frac\{\\sum\_\{t\}z\_\{a,t\}\}\{\\sum\_\{t\}m\_\{a,t\}\},\\qquad\\Delta\\bar\{N\}<0\\ \\wedge\\ \\Delta\\hat\{v\}\>0\\ \\Longrightarrow\\ \\text\{selection signature\},\(5\)whereΔ​x=xtreatment−xcontrol\\Delta x=x\_\{\\mathrm\{treatment\}\}\-x\_\{\\mathrm\{control\}\};v^a\\hat\{v\}\_\{a\}is N/A if its denominator is zero\. We computena,tn\_\{a,t\}independently of verification and also log appliedλt\\lambda\_\{t\}, floor vs\. measurement, tool/probe calls, and submission path\. Feedback exposes task and verifier scores but notna,tn\_\{a,t\}, so reward has no direct path to claim volume\. Released code, prompts, parser, and per\-run logs makeρ\\rhoand every probe recomputable\.

## 8Experimental Setup

We run the protocol on six supervised design tasks spanning four modalities \(Table[1](https://arxiv.org/html/2610.22779#S8.T1)\)\. Each dataset is split64/16/2064/16/20into train, validation, and test with preprocessing fit on the training partition only\. Search uses validation accuracy; all figures labeled*held\-out*are computed once on the test partition after selection\. The generator is a single commercial instruction\-tuned model, Solar Pro 3 of Upstage’s SOLAR lineage\([Kim et al\., 2024](https://arxiv.org/html/2610.22779#bib.bib18)\), accessed through its hosted API with default decoding parameters and fixed across all arms, so behavioral differences are attributable to the objective and tool access rather than the model\. Each run executes 16–19 iterations depending on dataset budget\. All confidence intervals are percentile bootstrap intervals over per\-seed run means \(10,00010\{,\}000resamples\)\. All probes use three seeds \(42,43,4442,43,44\)\. Across the main configuration, pressure sweep \(P1\), probe\-access probe \(P2\), and capability control \(P4\) this yields 180 runs; one run of the main configuration produces both the RCS\-included and RCS\-excluded arms\. The UCB1 sweep in Section[9\.2](https://arxiv.org/html/2610.22779#S9.SS2)adds a further 90\.

#### Identical\-configuration replicate\.

Because every run resamples the generator, bootstrap intervals over per\-seed means miss run\-to\-run variation at a*fixed*seed\. Two arms of our design are the same configuration executed twice, which bounds that component \(Table[3](https://arxiv.org/html/2610.22779#A3.T3), Appendix[C](https://arxiv.org/html/2610.22779#A3)\): claim volume differs by up to11\.4%11\.4\\%, held\-out RCS by×0\.61\\times 0\.61–×2\.71\\times 2\.71, test accuracy by up to0\.0480\.048, and verified\-claim rate — the one component the table does not list — by up to0\.060\.06\. We read every effect below against this band\.

Table 1:Design tasks and modalities; sources and feature construction are in Appendix[B](https://arxiv.org/html/2610.22779#A2)\.nncounts the rows actually ingested, which for four tasks exceeds the canonical size by two\.
#### Held\-out RCS reporting\.

Internal logs contain clipped summary fields \(Appendix[A](https://arxiv.org/html/2610.22779#A1)\); all tables and figures report the raw, unclipped post\-hoc estimator output \(uncapped\), though the estimator’s per\-drop rules still apply \(Section[9\.4](https://arxiv.org/html/2610.22779#S9.SS4)\)\. Held\-out RCS enters neither the objective nor winner selection, so it cannot leak test information into search\.

## 9Results

### 9\.1Deployment pressure is low \(F2\)

Table[2](https://arxiv.org/html/2610.22779#S9.T2)reports the main configuration with dynamicλ\\lambda\. Measured pressure ranges fromρ=0\.007\\rho=0\.007\(Ecommerce\) toρ=0\.089\\rho=0\.089\(Madelon\): the verifier moved the objective by less than nine percent of the accuracy spread it was weighed against on every dataset\. This is F2, and it is not an averaging artifact: no individual run among the 18 exceedsρ=0\.10\\rho=0\.10\.

Table 2:Main configuration with dynamicλ\\lambda, three seeds per dataset\. Brackets: bootstrap 95% CIs\. Held\-out RCS is an uncapped raw measure\.Including the verifier in the objective moves held\-out accuracy and rationale fidelity by no more than the seed\-to\-seed spread on most datasets; Ecommerce is the exception in both directions: the RCS\-included arm loses held\-out accuracy \(0\.6750\.675vs\.0\.7940\.794\) and held\-out fidelity \(0\.0560\.056vs\.0\.1630\.163\), each with non\-overlapping CIs \(Table[2](https://arxiv.org/html/2610.22779#S9.T2)\) — there, placing the verifier in the objective already degrades what it is meant to protect\. Though seen on one dataset, with an accuracy gap \(0\.1190\.119\)2\.5×2\.5\\timesthe replicate gap \(0\.0480\.048\), it is the most direct evidence here that an in\-objective verifier can backfire\. On Credit\-G the point estimates favor the RCS\-included arm on both held\-out accuracy \(0\.7230\.723vs\.0\.6970\.697\) and held\-out RCS \(0\.0370\.037vs\.0\.0160\.016\), though the CIs overlap; either way, the excluded arm must be measured post\-hoc rather than scored zero\. The floor rate \(F1\) ranges from0%0\\%\(Madelon\) to9\.3%9\.3\\%\(Breast Cancer\); silence is not the limiting problem — through the objective term, the verifier produces measurements that do not steer search\. One influence channel lies outsideρ\\rho: the deployed configuration also consults RCS as a final\-selection tie\-break \(Appendix[A](https://arxiv.org/html/2610.22779#A1)\)\. Re\-selecting each run’s reported pipeline without the RCS term flips it in55of1818runs, but every flip is among accuracy\-tied candidates \(0\.0000\.000conceded; Table[4](https://arxiv.org/html/2610.22779#A3.T4)\): at deployment the verifier breaks ties rather than trading accuracy for fidelity\.

### 9\.2Dose–response: no claim minimization discovered \(F3\)

Figure[2](https://arxiv.org/html/2610.22779#S9.F2)sweeps nominal staticλ\\lambdaover two orders of magnitude, reachingρ≈41\.7\\rho\\approx 41\.7on Madelon and63\.763\.7on Electricity atλnom=20\\lambda\_\{\\mathrm\{nom\}\}=20—2\.72\.7and2\.92\.9orders above the highest dataset\-mean deployed value \(ρ=0\.089\\rho=0\.089;3\.83\.8and4\.04\.0above the lowest\)\. Electricity’s value additionally reflects its low accuracy variance inflatingρ\\rho’s denominator \(Section[10](https://arxiv.org/html/2610.22779#S10)\)\. The sweep moved decisions and not only the ratio: the reported pipeline flips relative to accuracy\-only selection in33%33\\%of runs atλnom=0\.12\\lambda\_\{\\mathrm\{nom\}\}=0\.12,78%78\\%at55, and72%72\\%at2020, where a flip concedes up to0\.190\.19validation accuracy, while atλnom=0\\lambda\_\{\\mathrm\{nom\}\}=0it never flips \(Table[4](https://arxiv.org/html/2610.22779#A3.T4)\)\.

![Refer to caption](https://arxiv.org/html/2610.22779v1/figs/fig2_dose_response.png)Figure 2:Dose–response across six datasets \(three seeds each\); shading marks deployment\. \(a\) Over\>2\>2orders above it, claims show no significant downward trend \(Spearman∈\[−0\.12,\+0\.38\]\\in\[\-0\.12,\+0\.38\], allp\>0\.15p\>0\.15\); crosses mark lower levels reached under instructed minimization on four responsive datasets\. \(b\) Uncapped held\-out RCS shows no consistent increase withρ\\rho\.F3 is not observed under search\.Claim volume shows no statistically significant downward trend and is nowhere monotonically decreasing inλnom\\lambda\_\{\\mathrm\{nom\}\}\(Credit\-G10\.610\.6–12\.212\.2; Breast Cancer9\.79\.7–10\.210\.2; Madelon17\.717\.7–21\.021\.0, declining atλnom=1\\lambda\_\{\\mathrm\{nom\}\}=1and recovering at2020\)\. A per\-dataset Spearman test \(n=15n=15\) finds no significant negative trend \(correlations−0\.12\-0\.12to\+0\.38\+0\.38, allp\>0\.15p\>0\.15; none survives Holm–Bonferroni\), and rationale length is likewise stable \(e\.g\. Credit\-G282282–333333characters\)\. Altering the search rule and budget does not change this: replacing Thompson sampling with a deterministic UCB1 bandit\([Auer et al\., 2002](https://arxiv.org/html/2610.22779#bib.bib3)\)and doubling the base cap to 30 iterations \(realized1818–3838\) leaves claim volume unchanged, peak pressure stillρ≈39\\rho\\approx 39\(Madelon38\.738\.7, Electricity38\.438\.4\), and every per\-dataset Spearman correlation in\[−0\.17,\+0\.33\]\[\-0\.17,\+0\.33\]\(allp\>0\.23p\>0\.23; Table[8](https://arxiv.org/html/2610.22779#A3.T8), Appendix[C](https://arxiv.org/html/2610.22779#A3)\)\.

The per\-dataset tests \(n=15n=15\) detect only large monotone trends \(\|rs\|≈0\.52\|r\_\{s\}\|\\approx 0\.52atα=0\.05\\alpha=0\.05,≈0\.65\\approx 0\.65under Holm\)\. The null is informative not because these tests found nothing, but because instructed minimization reaches a level below the mean claim volume at every pressure level on the four datasets where it reduced claims \(1818–35%35\\%below default, Section[9\.3](https://arxiv.org/html/2610.22779#S9.SS3);×\\timesin Figure[2](https://arxiv.org/html/2610.22779#S9.F2)a\) — three of those reductions \(2727–35%35\\%\) being2\.42\.4–3\.1×3\.1\\timesthe replicate band of Section[8](https://arxiv.org/html/2610.22779#S8)and Credit\-G’s18%18\\%only1\.6×1\.6\\times\.

Under Equation[3](https://arxiv.org/html/2610.22779#S5.E3), deleting one inert claim atλnom=20\\lambda\_\{\\mathrm\{nom\}\}=20would raise the objective byρ/\(\|EP\|−1\)\\rho/\(\|E\_\{P\}\|\-1\)accuracy standard deviations \(≈1\.2\\approx 1\.2on Credit\-G,≈2\.1\\approx 2\.1on Madelon; Table[7](https://arxiv.org/html/2610.22779#A3.T7)\)\. This is an upper bound, since joint interventions can absorb a deletion; a direct claim\-deletion test with the pipeline held fixed is future work\.

Held\-out fidelity moves independently of claim volume\.On Credit\-G, uncapped held\-out RCS falls from0\.0530\.053atλnom=0\\lambda\_\{\\mathrm\{nom\}\}=0to0\.0220\.022atλnom=5\\lambda\_\{\\mathrm\{nom\}\}=5while claim counts stay flat, then partly recovers to0\.0440\.044atλnom=20\\lambda\_\{\\mathrm\{nom\}\}=20\. The CIs overlap and the×0\.42\\times 0\.42excursion is only marginally outside the replicate band, so it suggests fidelity moving with flat counts, not pressure\-induced drift\.

### 9\.3Capability control: minimization is possible when instructed

Atλnom=5\\lambda\_\{\\mathrm\{nom\}\}=5\(Table[5](https://arxiv.org/html/2610.22779#A3.T5), Figure[3](https://arxiv.org/html/2610.22779#A3.F3); Appendix[C](https://arxiv.org/html/2610.22779#A3)\), the*minimize*prompt reduces claim volume on four of six datasets: Butterfly−35%\-35\\%, Electricity−34%\-34\\%, Madelon−27%\-27\\%, Credit\-G−18%\-18\\%— three of five without Butterfly; the other two change little \(Ecommerce−0\.0%\-0\.0\\%, Breast−1\.7%\-1\.7\\%\), so the control is inconclusive there\. Held\-out RCS rises or falls \(Credit\-G×0\.60\\times 0\.60, Butterfly×0\.79\\times 0\.79, Madelon×1\.77\\times 1\.77\) within or at the edge of the replicate band, so this arm bears on volume only\. The contrast with Section[9\.2](https://arxiv.org/html/2610.22779#S9.SS2)is our central result:shorter claim lists are reachable, but search — Thompson sampling or UCB1 — did not find them without instruction\.Given feedback without claim counts and≤38\\leq 38proposals, this null is largely structural; the control shows only reachability\.

### 9\.4Probe access produces selection \(F4\)

Table[6](https://arxiv.org/html/2610.22779#A3.T6)and Figure[4](https://arxiv.org/html/2610.22779#A3.F4)\(both Appendix[C](https://arxiv.org/html/2610.22779#A3)\) compare agents that differ only in access toprobe\_feature\_drop, which the probe arm calls3\.13\.1–3\.93\.9times per iteration against a budget of four\. The selection signature stated in Section[6\.3](https://arxiv.org/html/2610.22779#S6.SS3)— claim volume falls while verified\-claim rate rises — appears on Butterfly \(claims15\.5→14\.315\.5\\to 14\.3, verified rate0\.49→0\.500\.49\\to 0\.50\), Madelon \(25\.6→14\.825\.6\\to 14\.8,0\.50→0\.510\.50\\to 0\.51\), Credit\-G \(12\.4→11\.712\.4\\to 11\.7,0\.40→0\.500\.40\\to 0\.50\), and Breast \(10\.5→10\.310\.5\\to 10\.3,0\.38→0\.410\.38\\to 0\.41\) — three of five without Butterfly\. The remaining two do not exhibit it: Ecommerce’s verified rate also falls \(0\.54→0\.490\.54\\to 0\.49\) and Electricity’s claim volume rises \(8\.9→11\.18\.9\\to 11\.1\)\. Only Madelon’s volume drop \(−42%\-42\\%\) and Credit\-G’s rate rise \(\+0\.10\+0\.10\) exceed the replicate band\.

On Credit\-G, held\-out fidelity drops with non\-overlapping CIs \(0\.0340\.034to0\.0160\.016\); on Ecommerce and Madelon the point estimates fall by half or more \(0\.069→0\.0240\.069\\to 0\.024,0\.188→0\.0950\.188\\to 0\.095\) but with large seed spread\. All three ratios \(×0\.46\\times 0\.46,×0\.35\\times 0\.35,×0\.50\\times 0\.50\) fall below the replicate band’s lower end \(×0\.61\\times 0\.61\); reporting only the verified\-claim rate would record improvement as fidelity degrades\. Scoring is asymmetric: all Credit\-G and two Madelon probe\-access winners use≤10\\leq 10features, and the estimator’s small\-pipeline rules \(Appendix[A](https://arxiv.org/html/2610.22779#A1)\) apply only to them, so part of those declines may be scoring artifacts; Ecommerce \(4040–7676features\) is unaffected but has overlapping CIs\.

## 10Limitations

#### Scope and power\.

One verifier of our own, one generator model \(Solar Pro 3\), six datasets, and pipeline\-design tasks with mechanical counterfactuals make the evidence preliminary\. Butterfly, our sole image task, reaches only≈0\.19\\approx 0\.19accuracy, so counts are also reported without it\. Our per\-dataset trend tests detect only large effects \(Section[9\.2](https://arxiv.org/html/2610.22779#S9.SS2)\), so we report the absence of*discovered*claim\-gaming, not equivalence; pooling datasets in a mixed model is future work\.

#### Generator variation and unswept settings\.

Decoding parameters were fixed, but the hosted generator’s served checkpoint cannot be identified\. The replicate band of Section[8](https://arxiv.org/html/2610.22779#S8)rests on one pair of runs, so it is an order of magnitude for fixed\-seed noise, not an interval, and verdicts near its edges may change with more replicates\. Estimator constants \(Appendix[A](https://arxiv.org/html/2610.22779#A1)\) were not swept\.

#### Normalization and untested channels\.

ρ\\rhocan inflate on low\-variance tasks without changing selection, so we report decision impact separately \(Table[4](https://arxiv.org/html/2610.22779#A3.T4)\)\. We did not test rephrasing that lowers the*parsed*claim count, search that observes claim count \(the most direct probe of F3\), and delayed or committed oracle access; our four\-call probe channel is noiseless and immediate\.

## 11Conclusion

We propose a stress\-test methodology for an agent’s own verifier — the pressure ratioρ\\rho, four falsifiable probes, and a reporting standard — that needs only an optimizing loop around a scalar verifier; its extension to tests, task completion, or citation grounding is a hypothesis\. Across six datasets and 180 runs \(plus a 90\-run sweep\) with one verifier and one generator,search did not discover claim minimization even atρ\\rhoover two orders above deployment, although instructed agents stated fewer claims — a null that is largely structural\. In summary: reportρ\\rho, count claims separately from scores, pair F3 robustness claims with a capability control, and measure fidelity on verifier\-unseen data\.

## References

- Agrawal and Goyal \[2012\]Shipra Agrawal and Navin Goyal\.Analysis of Thompson sampling for the multi\-armed bandit problem\.In*Proceedings of the 25th Annual Conference on Learning Theory*, volume 23 of*Proceedings of Machine Learning Research*, pages 39\.1–39\.26, 2012\.
- Amodei et al\. \[2016\]Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané\.Concrete problems in AI safety\.*arXiv preprint arXiv:1606\.06565*, 2016\.
- Auer et al\. \[2002\]Peter Auer, Nicolò Cesa\-Bianchi, and Paul Fischer\.Finite\-time analysis of the multiarmed bandit problem\.*Machine Learning*, 47\(2–3\):235–256, 2002\.doi:10\.1023/A:1013689704352\.
- Chuang et al\. \[2026\]Yu\-Neng Chuang, Guanchu Wang, Chia\-Yuan Chang, Ruixiang Tang, Shaochen Zhong, Fan Yang, Andrew Wen, Mengnan Du, Xuanting Cai, Vladimir Braverman, and Xia Hu\.FaithLM: Towards faithful explanations for large language models\.In*Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 3802–3824, 2026\.doi:10\.18653/v1/2026\.eacl\-long\.177\.
- Coste et al\. \[2024\]Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger\.Reward model ensembles help mitigate overoptimization\.In*International Conference on Learning Representations*, 2024\.
- DePie \[2023\]DePie\.Butterfly image classification\.Kaggle dataset, userphucthaiv02,[https://www\.kaggle\.com/datasets/phucthaiv02/butterfly\-image\-classification](https://www.kaggle.com/datasets/phucthaiv02/butterfly-image-classification), 2023\.
- Erickson et al\. \[2020\]Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola\.AutoGluon\-Tabular: Robust and accurate AutoML for structured data\.*arXiv preprint arXiv:2003\.06505*, 2020\.
- Gao et al\. \[2023\]Leo Gao, John Schulman, and Jacob Hilton\.Scaling laws for reward model overoptimization\.In*Proceedings of the 40th International Conference on Machine Learning*, volume 202 of*Proceedings of Machine Learning Research*, pages 10835–10866, 2023\.
- Gautam \[2019\]Gautam\.E commerce text dataset \(version \- 2\)\.Zenodo, 2019\.Kaggle mirror:[https://www\.kaggle\.com/datasets/saurabhshahane/ecommerce\-text\-classification](https://www.kaggle.com/datasets/saurabhshahane/ecommerce-text-classification)\.
- Guyon et al\. \[2004\]Isabelle Guyon, Steve Gunn, Asa Ben\-Hur, and Gideon Dror\.Result analysis of the NIPS 2003 feature selection challenge\.In*Advances in Neural Information Processing Systems*, volume 17, pages 545–552\. MIT Press, 2004\.
- Harries \[1999\]Michael Harries\.SPLICE\-2 comparative evaluation: Electricity pricing\.Technical Report UNSW\-CSE\-TR\-9905, School of Computer Science and Engineering, University of New South Wales, 1999\.
- Hofmann \[1994\]Hans Hofmann\.Statlog \(German credit data\)\.UCI Machine Learning Repository, 1994\.
- Hollmann et al\. \[2023\]Noah Hollmann, Samuel Müller, and Frank Hutter\.Large language models for automated data science: Introducing CAAFE for context\-aware automated feature engineering\.In*Advances in Neural Information Processing Systems*, volume 36, 2023\.
- Hooker et al\. \[2019\]Sara Hooker, Dumitru Erhan, Pieter\-Jan Kindermans, and Been Kim\.A benchmark for interpretability methods in deep neural networks\.In*Advances in Neural Information Processing Systems*, volume 32, 2019\.
- Hsia et al\. \[2024\]Jennifer Hsia, Danish Pruthi, Aarti Singh, and Zachary C\. Lipton\.Goodhart’s law applies to NLP’s explanation benchmarks\.In*Findings of the Association for Computational Linguistics: EACL 2024*, pages 1322–1335, 2024\.doi:10\.18653/v1/2024\.findings\-eacl\.88\.
- Jacovi and Goldberg \[2020\]Alon Jacovi and Yoav Goldberg\.Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 4198–4205, 2020\.doi:10\.18653/v1/2020\.acl\-main\.386\.
- Kaufmann et al\. \[2012\]Emilie Kaufmann, Nathaniel Korda, and Rémi Munos\.Thompson sampling: An asymptotically optimal finite\-time analysis\.In*International Conference on Algorithmic Learning Theory*, volume 7568 of*Lecture Notes in Computer Science*, pages 199–213\. Springer, 2012\.doi:10\.1007/978\-3\-642\-34106\-9\_18\.
- Kim et al\. \[2024\]Sanghoon Kim, Dahyun Kim, Chanjun Park, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, et al\.SOLAR 10\.7B: Scaling large language models with simple yet effective depth up\-scaling\.In*Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 6: Industry Track\)*, pages 23–35, 2024\.
- Lanham et al\. \[2023\]Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al\.Measuring faithfulness in chain\-of\-thought reasoning\.*arXiv preprint arXiv:2307\.13702*, 2023\.
- Lundberg and Lee \[2017\]Scott M\. Lundberg and Su\-In Lee\.A unified approach to interpreting model predictions\.In*Advances in Neural Information Processing Systems*, volume 30, 2017\.
- Manheim and Garrabrant \[2018\]David Manheim and Scott Garrabrant\.Categorizing variants of Goodhart’s law\.*arXiv preprint arXiv:1803\.04585*, 2018\.
- Olson and Moore \[2016\]Randal S\. Olson and Jason H\. Moore\.TPOT: A tree\-based pipeline optimization tool for automating machine learning\.In*Proceedings of the Workshop on Automatic Machine Learning*, volume 64 of*Proceedings of Machine Learning Research*, pages 66–74, 2016\.
- Pan et al\. \[2022\]Alexander Pan, Kush Bhatia, and Jacob Steinhardt\.The effects of reward misspecification: Mapping and mitigating misaligned models\.In*International Conference on Learning Representations*, 2022\.
- Ribeiro et al\. \[2016\]Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin\.“Why should I trust you?”: Explaining the predictions of any classifier\.In*Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, pages 1135–1144, 2016\.doi:10\.1145/2939672\.2939778\.
- Skalse et al\. \[2022\]Joar Skalse, Nikolaus H\. R\. Howe, Dmitrii Krasheninnikov, and David Krueger\.Defining and characterizing reward gaming\.In*Advances in Neural Information Processing Systems*, volume 35, pages 9460–9471, 2022\.
- Turpin et al\. \[2023\]Miles Turpin, Julian Michael, Ethan Perez, and Samuel R\. Bowman\.Language models don’t always say what they think: Unfaithful explanations in chain\-of\-thought prompting\.In*Advances in Neural Information Processing Systems*, volume 36, pages 74952–74965, 2023\.
- Wang et al\. \[2024\]Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui\.Large language models are not fair evaluators\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 9440–9450, 2024\.doi:10\.18653/v1/2024\.acl\-long\.511\.
- Yang et al\. \[2024\]Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V\. Le, Denny Zhou, and Xinyun Chen\.Large language models as optimizers\.In*International Conference on Learning Representations*, 2024\.
- Zheng et al\. \[2023\]Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P\. Xing, Hao Zhang, Joseph E\. Gonzalez, and Ion Stoica\.Judging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.In*Advances in Neural Information Processing Systems \(Datasets and Benchmarks Track\)*, volume 36, pages 46595–46623, 2023\.
- Zwitter and Soklic \[1988\]Matjaz Zwitter and Milan Soklic\.Breast cancer\.UCI Machine Learning Repository, 1988\.

## Appendix ARCS Estimator Details

The deployed estimatorRCS^\\widehat\{\\mathrm\{RCS\}\}differs from the base estimator of Equation[2](https://arxiv.org/html/2610.22779#S4.E2)in the ways below, and the deployed search additionally gates the verifier’s weight\. We state them here because each is a degree of freedom that an evaluation of the verifier must disclose\.

#### Accuracy gating of the verifier weight\.

With pre\-gateλ=sd​λnom\\lambda=s\_\{d\}\\lambda\_\{\\mathrm\{nom\}\}, the weightλt\\lambda\_\{t\}applied at iterationtt\(Section[6\.1](https://arxiv.org/html/2610.22779#S6.SS1)\) is gated on validation accuracy:

λt=γt⋅\{0Q^​\(Pt\)<τacc,RCS^​uninformative,min⁡\(λ,ct\)Q^​\(Pt\)<τacc,RCS^​informative,0\.5​λτacc≤Q^​\(Pt\)<0\.84,1\.2​λQ^​\(Pt\)≥0\.84,γt=\{γmt≤T0,1t\>T0\.\\lambda\_\{t\}\\;=\\;\\gamma\_\{t\}\\cdot\\begin\{cases\}0&\\hat\{Q\}\(P\_\{t\}\)<\\tau\_\{\\text\{acc\}\},\\ \\widehat\{\\mathrm\{RCS\}\}\\text\{ uninformative\},\\\\ \\min\(\\lambda,\\,c\_\{t\}\)&\\hat\{Q\}\(P\_\{t\}\)<\\tau\_\{\\text\{acc\}\},\\ \\widehat\{\\mathrm\{RCS\}\}\\text\{ informative\},\\\\ 0\.5\\,\\lambda&\\tau\_\{\\text\{acc\}\}\\leq\\hat\{Q\}\(P\_\{t\}\)<0\.84,\\\\ 1\.2\\,\\lambda&\\hat\{Q\}\(P\_\{t\}\)\\geq 0\.84,\\end\{cases\}\\qquad\\gamma\_\{t\}=\\begin\{cases\}\\gamma\_\{m\}&t\\leq T\_\{0\},\\\\ 1&t\>T\_\{0\}\.\\end\{cases\}\(6\)Hereγm=0\.78\\gamma\_\{m\}=0\.78for tabular and0\.680\.68otherwise;τacc=0\.18/0\.72/0\.74/0\.78\\tau\_\{\\text\{acc\}\}=0\.18/0\.72/0\.74/0\.78for image/time\-series/tabular/text\. The low\-accuracy capctc\_\{t\}is0\.03/0\.0150\.03/0\.015for tabular atRCS^≥0\.003/0\.0015\\widehat\{\\mathrm\{RCS\}\}\\geq 0\.003/0\.0015,0\.02/0\.030\.02/0\.03for time series atRCS^≥0\.01/0\.003\\widehat\{\\mathrm\{RCS\}\}\\geq 0\.01/0\.003, and0\.020\.02for image/text atRCS^≥0\.01\\widehat\{\\mathrm\{RCS\}\}\\geq 0\.01\. The time\-series ordering is non\-monotone because theRCS^≥0\.01\\widehat\{\\mathrm\{RCS\}\}\\geq 0\.01branch is modality\-agnostic and takes precedence; both caps sit far belowλnom\\lambda\_\{\\mathrm\{nom\}\}and apply only in the low\-accuracy regime, so the objective\-term effect is≤3×10−4\\leq 3\\times 10^\{\-4\}— below seed\-level accuracy spread\. The high cutoff is0\.840\.84andT0=max⁡\(8,0\.35​T\)T\_\{0\}=\\max\(8,0\.35T\)\. Controlled sweeps disable this gate entirely — leaving Tables[7](https://arxiv.org/html/2610.22779#A3.T7)–[8](https://arxiv.org/html/2610.22779#A3.T8)unaffected — but retainsd=0\.98/0\.98/0\.88/0\.82/0\.88/0\.78s\_\{d\}=0\.98/0\.98/0\.88/0\.82/0\.88/0\.78for image/text/time\-series/Breast/ Credit\-G/Madelon, soR⁡\(P\)=Q^​\(P\)\+sd​λnom​RCS^​\(P\)R\(P\)=\\hat\{Q\}\(P\)\+s\_\{d\}\\lambda\_\{\\mathrm\{nom\}\}\\widehat\{\\mathrm\{RCS\}\}\(P\)\.

#### Verification\-aware final selection\.

Beyond the objective term, the deployed system consults the verifier once more when it selects the pipeline to report: among candidates whose validation accuracy lies within a per\-dataset tolerance \(0\.0040\.004–0\.020\.02, expanded to0\.0350\.035for Credit\-G\) of the best, it prefers the one with the higher verification RCS — ordered by its real\-signal, tested\-claim, and nonzero\-drop counts — before falling back to reward and feature compactness\. This tie\-break is enabled in the deployed dynamic\-λ\\lambdaconfiguration on all six datasets\. It is a further channel through which the verifier shapes the reported artifact but whichρ\\rhodoes not register, sinceρ\\rho\(Equation[4](https://arxiv.org/html/2610.22779#S6.E4)\) measures only the objective term; we disclose it so that the verifier’s full influence is on record\. Held\-out RCS \(Section[8](https://arxiv.org/html/2610.22779#S8)\) remains a post\-hoc diagnostic and enters neither the objective nor this tie\-break\.

#### Noise threshold\.

A per\-claim drop at or below a modality\-dependent threshold is treated as measurement noise and contributes zero:2\.2×10−42\.2\\times 10^\{\-4\}for images \(1\.2×10−41\.2\\times 10^\{\-4\}when validation accuracy is below0\.180\.18\),1\.8×10−41\.8\\times 10^\{\-4\}for text,2\.8×10−42\.8\\times 10^\{\-4\}for time series, and3\.03\.0–3\.8×10−43\.8\\times 10^\{\-4\}for tabular data depending on validation accuracy\. The F1 floor rate we report is, by construction, a function of these thresholds\. Relative to one test\-example accuracy unit1/ntest1/n\_\{\\mathrm\{test\}\}under the64/16/2064/16/20split \(ntest≈0\.2​nn\_\{\\mathrm\{test\}\}\\approx 0\.2n\), the tabular thresholds are about66–8%8\\%of1/ntest1/n\_\{\\mathrm\{test\}\}on Credit\-G \(n=1002n\{=\}1002\), order\-one multiples of1/ntest1/n\_\{\\mathrm\{test\}\}on the large text and time\-series sets \(Ecommercen=50427n\{=\}50427; Electricityn=45312n\{=\}45312\), and about0\.160\.16–0\.290\.29of one test example on Butterfly \(n=6499n\{=\}6499\)\. This places the disclosed thresholds; it is not a null distribution for absolute held\-out RCS\.

#### Relative\-drop boost\.

When a drop exceeds a relative thresholdτ\\tauof the original validation accuracy, it is amplified byΔ↦Δ⁡\(1\+β​Δ/Q^​\(P\)\)\\Delta\\mapsto\\Delta\\,\(1\+\\beta\\,\\Delta/\\hat\{Q\}\(P\)\)with gainβ=2\.5\\beta=2\.5, usingτ=0\.015\\tau=0\.015for single\-claim interventions andτ=0\.02\\tau=0\.02for joint interventions\. The boost sharpens the contrast between load\-bearing and inert claims; it does not change their ordering\.

#### Joint interventions\.

When three or more feature\-importance claims are stated \(two or more for low\-accuracy image runs, where validation accuracy is below0\.100\.10\), a subset of the named features \(one if at most ten features are selected, otherwise up to half\) is removed together in a single counterfactual and the one resulting drop is used, rather than removing each feature separately\. This avoids crediting the same shared signal to several entangled claims, at the cost of coarser per\-claim attribution\. On pipelines with at most ten selected features, every per\-drop value \(joint or single\-claim\) is additionally capped at0\.020\.02\.

#### Floor\.

A pipeline with no parsable rationale, or with no counterfactual that could be executed, receives a floor value of10−310^\{\-3\}\(tabular\),5×10−45\\times 10^\{\-4\}\(text, time series\), or2×10−42\\times 10^\{\-4\}\(image\) rather than zero\. The rate at which this floor is used is exactly the F1 statistic of Section[9](https://arxiv.org/html/2610.22779#S9)\.

#### Authoritative configuration\.

The constants above, the per\-dataset weight scalessds\_\{d\}, and the selection tolerances are those set in the releasedrun\_experiments\.py, which is what the runs executed\. Themeta\.jsonwritten beside each unified CSV export also carries aracle\_budgetblock, but it is a snapshot taken at export time and was not refreshed: for Electricity it recordssd=1\.0s\_\{d\}=1\.0and a tolerance of0\.0050\.005and omits the faithfulness\-first flag, where the executed configuration used0\.880\.88,0\.0040\.004, and the flag enabled\. Readers reconciling numbers should use the run script, not the export metadata\.

#### Held\-out reporting\.

Internal summaries clip held\-out RCS at0\.080\.08\(image/text\) or0\.060\.06\(other modalities\), then at0\.0350\.035for pipelines with at most ten features; a legacy paper field clips again to\[0\.001,0\.02\]\[0\.001,0\.02\]\. This paper instead uses the raw measurement recorded alongside those fields, so comparisons across pressure levels are not truncated\. “Uncapped” refers to these summary clips only: the relative\-drop boost and, for pipelines with at most ten features, the per\-drop cap and single\-feature joint removal still apply to the raw value\. Among probe\-access winners, the small\-pipeline rules apply to all three on Credit\-G \(88features\) and two of three on Madelon \(1010\), but to no control winner \(1111–1717and1111–2626features\), which is why Section[9\.4](https://arxiv.org/html/2610.22779#S9.SS4)qualifies those declines; comparing pre\-cap, pre\-boost drops across arms is future work\.

## Appendix BDatasets

Table[1](https://arxiv.org/html/2610.22779#S8.T1)lists the six tasks; all are public\. Splits and preprocessing follow Section[8](https://arxiv.org/html/2610.22779#S8)\.

#### Unified export artifact\.

Every task is read from a unified CSV export, and in four of them that export repeated the header line twice as data\. Credit\-G, Breast Cancer, Madelon, and Ecommerce were therefore trained on two spurious rows each, which is what thennin Table[1](https://arxiv.org/html/2610.22779#S8.T1)records; their canonical sizes are10001000,286286,20002000\(Madelon’s training split\), and50,42550\{,\}425\. The spurious rows form a third label with two examples — under0\.7%0\.7\\%of even the smallest affected task — and are identical across every arm, seed, and pressure level, so they cannot produce a between\-arm difference\. Butterfly and Electricity are unaffected\.

#### Meaning ofnn\.

For five tasksnnis the row count of the single ingested file, split64/16/2064/16/20afterwards\. Butterfly is the exception: its source distributes a labelled training manifest \(64996499rows, the value in Table[1](https://arxiv.org/html/2610.22779#S8.T1)\) and a separate27862786\-row manifest without labels\. Only the labelled manifest is ingested, so Butterfly’snncounts labelled rows available rather than all rows shipped with the dataset\.

#### Tabular\.

Credit\-G is the Statlog German Credit dataset\[[Hofmann, 1994](https://arxiv.org/html/2610.22779#bib.bib12)\]\(OpenMLcredit\-g, id 31; German bank applicants,2020attributes, good/bad\-risk labels\)\. Breast Cancer is the UCI Ljubljana breast\-cancer recurrence dataset\[[Zwitter and Soklic, 1988](https://arxiv.org/html/2610.22779#bib.bib30)\]\. Madelon is the artificial NIPS 2003 feature\-selection challenge dataset\[[Guyon et al\., 2004](https://arxiv.org/html/2610.22779#bib.bib10)\]\(500500features, of which480480are non\-informative noise probes\), which we use as a high\-dimensional sparsity stress test\.

#### Time series\.

Electricity is the ELEC2 dataset\[[Harries, 1999](https://arxiv.org/html/2610.22779#bib.bib11)\]from the Australian New South Wales electricity market \(45,31245\{,\}312half\-hourly records; binary up/down price movement\)\. We construct lag\-1/2/3 features on five series \(23 features after expansion\) and use a time\-ordered rather than random split\.

#### Text\.

Ecommerce Text is the E\-commerce text dataset of[Gautam \[2019\]](https://arxiv.org/html/2610.22779#bib.bib9), a product\-description corpus in four categories \(Household, Books, Electronics, Clothing & Accessories\), represented by unigram/bigram TF\-IDF features\.

#### Image\.

Butterfly Image is the public Kaggle Butterfly Image Classification dataset\[[DePie, 2023](https://arxiv.org/html/2610.22779#bib.bib6)\]; each image is represented by simple grayscale\-intensity statistics and histograms \(26 dimensions, with a filename\-hash fallback when the raw image is unavailable\), not by pretrained\-CNN embeddings\. The classifier reaches only≈0\.19\\approx 0\.19test accuracy on this 75\-class task — well above the1/75≈0\.0131/75\\approx 0\.013chance rate, but low in absolute terms, since the representation is deliberately weak\. Because it is by a wide margin the lowest\-accuracy task in the suite, we report all dataset counts both with and without Butterfly as a robustness check\.

## Appendix CFull Result Tables

#### Identical\-configuration replicate\.

Table[3](https://arxiv.org/html/2610.22779#A3.T3)isolates the run\-to\-run variation quoted in Section[8](https://arxiv.org/html/2610.22779#S8)\. Two arms of our design coincide exactly: theλnom=5\\lambda\_\{\\mathrm\{nom\}\}=5level of the dose–response sweep and the*default*arm of the capability control both run with staticλnom=5\\lambda\_\{\\mathrm\{nom\}\}=5, seeds42/43/4442/43/44, the default proposer, and no policy prompt\. They were nonetheless executed as two independent invocations, so their gap measures how much the generator alone moves the reported quantities at a fixed seed — a component the across\-seed bootstrap cannot see, because each of its resampled runs draws a fresh generator sample\. We therefore treat claim\-volume differences under11\.4%11\.4\\%, held\-out RCS ratios within\[×0\.61,×2\.71\]\[\\times 0\.61,\\times 2\.71\], verified\-rate differences under0\.060\.06, and test\-accuracy differences under0\.0480\.048as indistinguishable from re\-execution noise, and say so wherever an effect falls inside that band\.

Table 3:Identical\-configuration replicate\. Run A is theλnom=5\\lambda\_\{\\mathrm\{nom\}\}=5arm of the dose–response sweep \(Table[7](https://arxiv.org/html/2610.22779#A3.T7)\); Run B is the*default*arm of the capability control \(Table[5](https://arxiv.org/html/2610.22779#A3.T5)\)\. Both fix staticλnom=5\\lambda\_\{\\mathrm\{nom\}\}=5and seeds42/43/4442/43/44and apply no policy prompt, so they differ only in being invoked twice\. Three\-seed means; held\-out RCS is uncapped\.
#### Selection flip rate\.

Table[4](https://arxiv.org/html/2610.22779#A3.T4)reports decision impact, the quantityρ\\rhodoes not measure\. For every run we re\-select the reported pipeline from the iterations that run actually executed, once underQ^​\(Pt\)\+λt​RCS^​\(Pt\)\\hat\{Q\}\(P\_\{t\}\)\+\\lambda\_\{t\}\\widehat\{\\mathrm\{RCS\}\}\(P\_\{t\}\)and once underQ^​\(Pt\)\\hat\{Q\}\(P\_\{t\}\)alone, and record whether the choice changes and how much validation accuracy the changed choice gives up\. Two qualifications matter\. It is a post\-hoc re\-selection, not a re\-run: deleting the verifier term would also change which pipelines the search proposed, so the flip rate characterizes the observed candidate pool rather than a counterfactual trajectory\. And it compares objective terms only, so the final\-selection tie\-break of Appendix[A](https://arxiv.org/html/2610.22779#A1)is excluded\. Read with those limits, theλnom=0\\lambda\_\{\\mathrm\{nom\}\}=0row is a null check \(no term, no flips\), the deployed row shows influence confined to accuracy ties, and the sweep rows show the elevatedρ\\rhovalues translating into decisions taken at real accuracy cost\.

Table 4:Selection flip rate: how often the verifier term decides which pipeline a run reports\. For every run we re\-select over the iterations it actually executed, comparingarg⁡maxt​\[Q^​\(Pt\)\+λt​RCS^​\(Pt\)\]\\arg\\max\_\{t\}\\,\[\\hat\{Q\}\(P\_\{t\}\)\+\\lambda\_\{t\}\\widehat\{\\mathrm\{RCS\}\}\(P\_\{t\}\)\]witharg⁡maxt​Q^​\(Pt\)\\arg\\max\_\{t\}\\,\\hat\{Q\}\(P\_\{t\}\)\.*Acc\. conceded*is the validation accuracy the flipped choice gives up, so0\.0000\.000means the term broke a tie\. Post\-hoc re\-selection, not a re\-run: removing the term would also change the trajectory\.Tables[5](https://arxiv.org/html/2610.22779#A3.T5)–[7](https://arxiv.org/html/2610.22779#A3.T7)give the full per\-dataset capability\-control \(Figure[3](https://arxiv.org/html/2610.22779#A3.F3)\), probe\-access \(Figure[4](https://arxiv.org/html/2610.22779#A3.F4)\), and dose–response \(Figure[2](https://arxiv.org/html/2610.22779#S9.F2)\) results \(three\-seed means, bootstrap95%95\\%CIs, uncapped held\-out RCS; percent reductions quoted in the text are computed from unrounded means\); Table[8](https://arxiv.org/html/2610.22779#A3.T8)repeats the dose–response under UCB1 with a doubled base cap of 30 iterations \(realized1818–3838; Section[9\.2](https://arxiv.org/html/2610.22779#S9.SS2)\)\.

Table 5:Capability control at nominalλnom=5\\lambda\_\{\\mathrm\{nom\}\}=5: default/minimize claim policies, three seeds\.![Refer to caption](https://arxiv.org/html/2610.22779v1/figs/fig3_capability_control.png)Figure 3:Capability control atλnom=5\\lambda\_\{\\mathrm\{nom\}\}=5: claim volume under the default vs\. minimize policy \(bars, bootstrap 95% CIs\)\. The dashed line is the mean claim volume reached by the nominal\-λ\\lambdasweep under search; instruction drives volume below it on the four datasets where search never did\. Percent labels give the minimize reduction relative to default\.Table 6:Probe access to the verifier, three seeds\. Held\-out RCS is uncapped\.Table 7:Dose–response to nominal static verifier weight, three seeds; dataset scalesds\_\{d\}remains active\. Held\-out RCS is uncapped andρ\\rhomeasures pressure\.![Refer to caption](https://arxiv.org/html/2610.22779v1/figs/fig4_oracle_signature.png)Figure 4:Oracle access signature \(three seeds averaged\)\. Each arrow runs from thetoolsarm \(circle\) to thetools\+probearm \(triangle\); marker area is proportional to uncapped held\-out RCS\. The shaded upper\-left quadrant is the selection regime — fewer claims at a higher verified rate\. Arrows point into it on four datasets; held\-out fidelity \(marker area\) drops with non\-overlapping CIs only on Credit\-G and is suggestive but noisy across seeds elsewhere\.Table 8:Dose–response under deterministic UCB1 with the base cap doubled to 30 before applying the dataset\-specific iteration\-budget multiplier \(typically 34–38 iterations; twoλnom=0\\lambda\_\{\\rm nom\}=0runs used 18 and 26\)\. The nominal static\-weight sweep matches Table[7](https://arxiv.org/html/2610.22779#A3.T7); held\-out RCS is uncapped andρ\\rhomeasures pressure\. Claim\-count curves are nonmonotonic inλnom\\lambda\_\{\\rm nom\}\.