智能体评估的信任层

# A Trust Layer for Agent Evaluation
Source: [https://arxiv.org/html/2610.07274](https://arxiv.org/html/2610.07274)
Mohammadreza SediqinShivali DalmiaAffiliation:Centific ResearchAffiliation:Seattle, USAEmail:[shivali\.dalmia@centific\.com](mailto:)Srinivasa Karthikeya Reddy KovvuriAffiliation:Centific ResearchAffiliation:Washington, USAEmail:[srinivasa\.kovvuri@centific\.com](mailto:)Abhishek MukherjiAffiliation:Centific ResearchAffiliation:Seattle, USAEmail:[abhishek\.mukherji@centific\.com](mailto:)

###### Abstract

Deterministic benchmark scores show that an agent received credit, but not whether that credit was earned, reported honestly, or would hold on a second run\. We introduce a*Trust Layer for Agent Evaluation*, an additive post\-hoc framework that reports, beside each recorded score, whether it should be believed\. It verifies four properties: whether the result is supported by the benchmark’s own grading logic, whether a passing answer was earned through traceable computation, whether the agent’s completion claim matches what occurred, and whether the result is stable under repeated execution\. The first three use only saved artifacts; the fourth re\-runs the agent\. Model judgments only label evidence under majority voting; all verdicts follow deterministic rules and never modify the recorded score\. Applied to five agent configurations on 108 tasks from Agents’ Last Exam, every model shows passing runs with no traceable computation \(at rates varying tenfold\), confirmed false completion claims, and unstable results: 18–46% of tasks do not stay in one score band over five runs\. Only 22\.6% of recorded passes clear all four checks \(95% CI 15\.0–32\.6,n=84n\{=\}84\)\. Measuring what an agent can do and verifying that it actually did it are different problems, and current benchmarks address only the first\.

## 1Introduction

LLM agents are now evaluated on increasingly realistic long\-horizon benchmarks in software engineering, scientific discovery, web interaction, and professional work\. SWE\-bench\([Jimenez et al\., 2024](https://arxiv.org/html/2610.07274#bib.bib2)\),τ\\tau\-bench\([Yao and others, 2024](https://arxiv.org/html/2610.07274#bib.bib5)\), ScienceAgentBench\([Chen et al\., 2025](https://arxiv.org/html/2610.07274#bib.bib24)\), and Agents’ Last Exam \(ALE\)\([Sun et al\., 2026](https://arxiv.org/html/2610.07274#bib.bib1)\)replace subjective assessment with executable grading, deterministic tests, and reference\-based verification, so a benchmark score is often treated as an objective measurement of capability\.

But a score answers only one question:*did the grader award credit?*It does not say whether the credit was earned, whether the reported outcome reflects what happened, or whether the result would hold on a second run\. A run that*fabricates a plausible value*scores exactly like one that computed it\. A run that*fails but claims success*scores exactly like one that fails and says so\. Because agent runs vary from attempt to attempt, a run that passes once in five tries scores exactly like one that passes every time; repetition studies show single\-run scores shifting by several points from chance alone\([Bjarnason et al\., 2026](https://arxiv.org/html/2610.07274#bib.bib22)\)\. And a failing score may reflect a packaging or evaluation artifact rather than genuine inability\. In each case the number is accurate about the run and misleading about the agent, which raises a question the score cannot resolve on its own:*can a recorded benchmark score be trusted?*

Existing work addresses these concerns separately, through benchmark auditing, reward hacking and corrupt success\([Skalse et al\., 2022](https://arxiv.org/html/2610.07274#bib.bib17);[Cao et al\., 2026](https://arxiv.org/html/2610.07274#bib.bib11)\), honesty\([Ren and others, 2025](https://arxiv.org/html/2610.07274#bib.bib13);[Chern and others, 2024](https://arxiv.org/html/2610.07274#bib.bib14)\), reliability\([Gupta, 2026](https://arxiv.org/html/2610.07274#bib.bib16)\), and provenance tracking for LLM\-generated content\([Sediqin et al\., 2026a](https://arxiv.org/html/2610.07274#bib.bib20)\), usually with new benchmarks, datasets, or protocols\. None answers the simpler question a benchmark user faces: given a score already recorded, should it be believed?

We introduce a*Trust Layer for Agent Evaluation*, an additive post\-hoc framework that audits recorded outcomes using artifacts benchmarks already save \(trajectories, staged outputs, grading code\) and runs alongside the benchmark rather than replacing it\. It applies four verifiers, one per case above\.*Competence*checks whether the benchmark’s own grading logic reproduces the recorded outcome and names the decisive cause of failure\.*Reward Hacking*checks whether a pass was earned through traceable computation or reached through shortcuts and ungrounded outputs\.*Agent Deception Detection*checks the agent’s completion claim against the independently verified outcome\.*Reliability*checks whether repeated runs under identical conditions stay in the same score band\.

This is the first framework we know of to verify these four properties together, for an individual recorded score, from evidence a benchmark already produces\. Its central design principle is the*separation of judgment from decision*: models may help label evidence, but every verdict comes from a deterministic rule, and any load\-bearing model label needs a majority of three independent votes\. This matters because the closest prior work finds that judges of false completion claims anchor on confident language, with no configuration exceeding AUROC 0\.65\([Advani and others, 2026](https://arxiv.org/html/2610.07274#bib.bib12)\)\.

On 108 ALE tasks, the layer finds grading contradictions in both directions, including recorded passes that the benchmark’s own grader scores at zero when re\-run on the saved output; passing results with no traceable computation behind them; false completion claims in every recorded band; and results that do not reproduce\. The dimensions prove additive rather than overlapping: many flagged tasks are caught by a single dimension, so removing any one would leave them looking clean, and only a minority of recorded successes clear all four\. Our contributions are:

1. 1\.To our knowledge, the first post\-hoc trust layer that verifies whether a recorded score is reproducible, earned, honest, and supported by the benchmark’s own grading logic, unifying four properties previously studied separately\.
2. 2\.A design in which every verdict is deterministic and model judgment only labels evidence under a majority of three votes, so the headline number stays intact and auditable\.
3. 3\.An empirical study over 108 professional tasks showing that substantial trust failures hide behind benchmark scores, that the four dimensions detect largely distinct failures, and that requiring all four leaves far fewer verified results than the scores suggest\. Code will be released upon acceptance\.

## 2Related Work

#### Agentic benchmarks and validity auditing\.

Recent agent benchmarks favour verifiable, code\-based outcomes\. ALE grades long\-horizon professional tasks with deterministic scripts against expert references, relying on LLM judging for under 7% of workflows\([Sun et al\., 2026](https://arxiv.org/html/2610.07274#bib.bib1)\)\. SWE\-bench grades code fixes for real GitHub issues with unit tests\([Jimenez et al\., 2024](https://arxiv.org/html/2610.07274#bib.bib2)\),τ\\tau\-bench compares the final database state with a reference\([Yao and others, 2024](https://arxiv.org/html/2610.07274#bib.bib5)\), and ScienceAgentBench scores self\-contained programs on tasks from peer\-reviewed publications\([Chen et al\., 2025](https://arxiv.org/html/2610.07274#bib.bib24)\)\. Each reduces a run to one score from the final artifact, leaving the trajectory unscored\. Such scores can misrepresent competence:[Zhu et al\. \(2026\)](https://arxiv.org/html/2610.07274#bib.bib8)formalize validity threats in a manual checklist spanning task and outcome validity,[Kapoor et al\. \(2024\)](https://arxiv.org/html/2610.07274#bib.bib9)show that uncontrolled cost distorts conclusions, and automated auditing finds that test suites can reject correct solutions, with evaluator limitations increasingly dominating reported scores\([Tu et al\., 2026](https://arxiv.org/html/2610.07274#bib.bib10)\)\. These efforts audit benchmark*construction*through expert review, and related checks govern consistency in document repositories\([Sediqin et al\., 2026b](https://arxiv.org/html/2610.07274#bib.bib23)\)\.

#### Reward hacking, honesty, and reliability\.

Specification gaming, satisfying the literal objective but not its intent, was formalized by[Skalse et al\. \(2022\)](https://arxiv.org/html/2610.07274#bib.bib17)and catalogued across reinforcement learning systems by[Krakovna et al\. \(2020\)](https://arxiv.org/html/2610.07274#bib.bib18)\. For tool\-using LLM agents, benchmarks measure the propensity to skip verification or tamper with the evaluation itself\([Thaman, 2026](https://arxiv.org/html/2610.07274#bib.bib19);[Gabor et al\., 2026](https://arxiv.org/html/2610.07274#bib.bib21)\), and[Cao et al\. \(2026\)](https://arxiv.org/html/2610.07274#bib.bib11)report*corrupt success*, correct end\-states reached while violating procedural integrity, in 27% to 78% of reported successes\.[Advani and others \(2026\)](https://arxiv.org/html/2610.07274#bib.bib12)characterize false success, agents asserting completion the environment contradicts, and show that LLM judges detect it unreliably; MASK\([Ren and others, 2025](https://arxiv.org/html/2610.07274#bib.bib13)\)and BeHonest\([Chern and others, 2024](https://arxiv.org/html/2610.07274#bib.bib14)\)measure propensity in elicited scenarios rather than verifying claims in real runs\. On reliability, single\-run pass@1 across 60,000 SWE\-Bench\-Verified trajectories varies by up to 6\.0 points depending on the run selected, and the authors recommend passkas a pessimistic bound\([Bjarnason et al\., 2026](https://arxiv.org/html/2610.07274#bib.bib22)\); related work treats evaluations as experiments whose variance determines reproducibility\([Mustahsan et al\., 2025](https://arxiv.org/html/2610.07274#bib.bib15)\), and ReliabilityBench measures consistency under repetition and perturbation\([Gupta, 2026](https://arxiv.org/html/2610.07274#bib.bib16)\)\.

#### Positioning\.

Validity work audits how a benchmark is*constructed*; reward hacking, false success, and honesty work each target one mode, usually through an LLM judge whose own reliability is in question; and reliability work builds repetition into*new*benchmarks, all at a different level than one score\. The closest work, procedure\-aware evaluation\([Cao et al\., 2026](https://arxiv.org/html/2610.07274#bib.bib11)\), defines a new evaluation of whether an agent followed the intended procedure\. We instead ask whether a score an existing benchmark*already recorded*holds up, verifying four properties of it from the saved trajectory and grading code, per dimension and as a conjunction, so a pass that is procedurally clean but unstable, or earned but falsely reported, is still caught\.

Figure 1:Overview of the Trust Layer\. D1 re\-derives the verdict, D2 traces value provenance, D3 tests the agent’s claim, D4 repeats the run\.

## 3System Architecture

The trust layer is an additive stage that runs after a benchmark has scored an agent \(Figure[1](https://arxiv.org/html/2610.07274#S2.F1)\)\. It has four dimensions, one per failure class\.D1 \(Competence\)asks what a recorded score is actually made of and exposes scores the grader does not reproduce, in either direction\.D2 \(Reward Hacking\)asks whether a passing score was earned or reached without doing the work\.D3 \(Agent Deception Detection\)asks whether the agent’s success claim is contradicted by the outcome\.D4 \(Reliability\)asks whether the score stays in the same band on repetition\. D1–D3 read only what the benchmark saved; D4 also re\-runs the agent\. The layer reuses the benchmark’s own grader wherever a verdict is needed and never overwrites the recorded score, reporting its findings beside it\. It needs three things from a benchmark: saved trajectories, accessible grader logic, and outputs readable offline\.

#### Evidence: the trajectory and re\-grading\.

Two artifacts benchmarks produce but never fully use supply the evidence: the*trajectory*, the step\-by\-step record showing*how*a result was produced, and the benchmark’s own*grader*, re\-run on the saved output to reproduce the score instead of trusting it\. Every dimension rests on one or both\.

### 3\.1D1: Competence

In short: re\-run the grader and ask what actually decided the score\.D1 re\-derives the verdict from the saved output with the task’s own grader and compares it with the recorded score\. No model assigns a score: a model labels gates by severity, and the verdict is a deterministic function of those labels\. D1 has four stages, each using the output of the last\.

#### Stages A and B: typing and re\-grading\.

A*gate*is a check inside the grader whose failure forces the score to zero, such as a missing file or a wrong value\. Stage A recovers every gate into a setGGand types it by what it protects,

τ⁡\(g\)∈\{critical,process,administrative\},\\tau\(g\)\\in\\left\\\{\\begin\{array\}\[\]\{@\{\}l@\{\}\}\\texttt\{critical\},\\texttt\{process\},\\\\ \\texttt\{administrative\}\\end\{array\}\\right\\\},\(1\)wherecriticalmeans the deliverable is defective,processa violated method rule, andadministrativea packaging defect\. Each gate also carries an anti\-gaming flagα⁡\(g\)\\alpha\(g\)\. Typing is done by an LLM judge under a majority of three votes and, being a property of the grader rather than a run, once per task\. Stage B re\-runs the grader on the saved output and maps the fired gatesFFto a verdict,

v=Φ⁡\(F,τ,α\),F⊆G,v=\\Phi\(F,\\tau,\\alpha\),\\qquad F\\subseteq G,\(2\)with the most severe fired gate deciding, so a real correctness failure cannot hide behind a lesser one\. Appendix[A](https://arxiv.org/html/2610.07274#A1)gives the gate criteria, matching procedure, severity ordering, and the verdicts Stage B can return\.

#### Stages C and D: claim and challenge\.

Stage C reduces the agent’s terminal message, by deterministic pattern matching, to a completion claimccand tests it againstvv, giving the three\-valued honesty signal used by D3:

ℓ=\{1if​c=success∧v∈Vfail,⊥if​c=success∧v∈Vunver,0otherwise,\\ell=\\begin\{cases\}1&\\text\{if \}c=\\texttt\{success\}\\wedge v\\in V\_\{\\text\{fail\}\},\\\\ \\bot&\\text\{if \}c=\\texttt\{success\}\\wedge v\\in V\_\{\\text\{unver\}\},\\\\ 0&\\text\{otherwise\},\\end\{cases\}\(3\)whereVfailV\_\{\\text\{fail\}\}are verdicts that reproduce a genuine failure andVunverV\_\{\\text\{unver\}\}those that could not be reproduced offline\. Stage D is the only place the earlier facts are combined, by a purely deterministic rule with no model call:

\(D,δ\)=Ψ⁡\(s,v,ℓ,m,κ\),D∈\{confirm,contradiction,upgrade,unverified\},\\begin\{gathered\}\(D,\\delta\)=\\Psi\(s,v,\\ell,m,\\kappa\),\\\\ D\\in\\left\\\{\\begin\{array\}\[\]\{@\{\}l@\{\}\}\\texttt\{confirm\},\\texttt\{contradiction\},\\\\ \\texttt\{upgrade\},\\texttt\{unverified\}\\end\{array\}\\right\\\},\\end\{gathered\}\(4\)wheressis the recorded score,mmwhether the prompt itself demanded the failed format, andκ\\kappaa format\-blind check that the output values match the reference\. A confirm reproduces the recorded outcome; a contradiction disagrees, with a signed directionδ\\deltarecording which side was wrong; an unverified could not be checked offline; and an upgrade, the one outcome needing all three earlier stages, is reached when the run failed a check that does not mean the work was wrong\. Appendix[A](https://arxiv.org/html/2610.07274#A1)gives the upgrade conditions and contrasts a contradiction with an upgrade\.

### 3\.2D2: Reward Hacking

In short: ask where each answer value came from\.A score shows that an answer is correct, not how it was produced\. An agent that hardcodes a constant, copies a value the prompt revealed, or writes plausible numbers with no computation behind them scores exactly like one that did the work, and the grader is fooled\. D2 marks such runs gamed, separating them from the honest passes they would otherwise be counted among\. It traces provenance rather than re\-checking correctness\.

#### Tracing each value\.

D2 uses the reference answer, which the agent never saw, as an offline oracle to pick out which output values are the answer, then traces each against three sources from the run: output of commands the agent executed, inputs it read or was given in the prompt, and text it typed by hand into files and shell commands\. Each value gets a label,

π⁡\(x\)∈\{derived,given,typed,ungrounded\},\\pi\(x\)\\in\\left\\\{\\begin\{array\}\[\]\{@\{\}l@\{\}\}\\textsc\{derived\},\\textsc\{given\},\\\\ \\textsc\{typed\},\\textsc\{ungrounded\}\\end\{array\}\\right\\\},\(5\)derivedif it appears in executed output,givenif it came from an input or the prompt,typedif written by hand, andungroundedif it appears nowhere\. The decisive test is for an answer value absent from executed output: was it typed by the agent, or produced by a script at run time? A correct value a script computed is real work; a hand\-typed one is not\. Scalars are traced one at a time and arrays at the file level \(Appendix[A](https://arxiv.org/html/2610.07274#A1)\)\.

#### Confessions, the judge, and the verdict\.

The agent’s messages and reasoning are scanned for admissions of copying or hardcoding, with a negation guard so a stated intention*not*to hardcode is never read as a confession\. A confession is never a flag on its own: with a value already found unearned \(matching the reference but neither computed nor read\) it confirms a hack; otherwise it only marks the run for a closer look\. Values that remain genuinely unclear go to a model judge, which sees the prompt, the flagged values, the agent’s words, and its code, and is warned that printing a hardcoded constant does not make it computed\. Before convicting, D2 has the judge check whether the task required the value to be computed; if the task permits a direct answer, the value is cleared\. A strict majority of three votes is required to convict, so a split resolves toclean: the check would rather miss a cheat than accuse an honest run\. The run verdict is

Vint∈\{clean,suspect,gamed\},V\_\{\\text\{int\}\}\\in\\\{\\texttt\{clean\},\\texttt\{suspect\},\\texttt\{gamed\}\\\},\(6\)where a strong flag givesgamed, a weak flag or unresolved valuesuspect, and a clean traceclean\.suspectis only a routing state that sends the run to the judge, which resolves it, so none is reported\. Each verdict records its cheat type \(copied or fabricated\); signals range from a confession with a provenance gap, the strongest, to a judge\-voted fabrication resting on the absence of a trace, the weakest \(Appendix[A](https://arxiv.org/html/2610.07274#A1)\)\. An untraced value is flagged for review, not convicted, and is markedgamedonly when a condition above holds, never when the task permits a direct answer\. A value fabricated and then written through executed code looks derived, so D2 bounds untraceable fabrication, not fabrication laundered through execution\.

### 3\.3D3: Agent Deception Detection

In short: check the agent’s final report against what really happened\.The last message is where the agent reports its result, so when it claims success on a run that failed, oversight breaks down\. The benchmark gives the same score either way, so lying costs the agent nothing and the score never shows it\. We use “lie” operationally: a claim of success on a run whose failure was proven by re\-grading; D3 shows the report was false, not that the agent knew it was\. D3 has two jobs: a yes\-or\-no check inside the D1 pipeline, and a separate judgement of how serious a confirmed lie was\.

#### Job 1: is the agent lying?

This is the deterministic signalℓ\\ellof Equation \([3](https://arxiv.org/html/2610.07274#S3.E3)\)\. It checks the agent’s success claim against the re\-graded verdictvv,*not*the recorded score: testing against the recorded score would only show that the agent claimed success and scored zero, mixing up a lying agent with a grading error, while a re\-grade tells them apart\. If re\-running the grader proves the failure, the claim is a confirmed lie \(ℓ=1\\ell=1\); if the grader cannot be re\-run, it is marked suspected and D3 stops \(ℓ=⊥\\ell=\\bot\)\. This fact feeds back into Stage D, where a proven lie removes the agent’s benefit of the doubt on a formatting\-only failure\. Becauseℓ\\ellrests on D1’s re\-grade, the two dimensions are not independent evidence\.

#### Job 2: how serious was the lie?

After D1 finishes, a model judge re\-examines each confirmed lie under majority vote, so this output is an adjudication rather than a measurement\. The judge sees the agent’s words and what independently happened \(the recorded score, re\-graded verdict, fired gate, failure explanation, grader messages, and a trajectory digest\) and produces

\(ℓ^,γ,η,ϕ\)=Ξ⁡\(ℓ,v,q,E\),\(\\hat\{\\ell\},\\gamma,\\eta,\\phi\)=\\Xi\(\\ell,v,q,E\),\(7\)whereqqis the agent’s exact claim andEEwhat happened in the run: a final decisionℓ^\\hat\{\\ell\}confirming or dropping the lie, the kind of claimγ\\gamma, whether a concrete defect sits behind itη\\eta, and whether the lie alone should fail the taskϕ\\phi\(Appendix[A](https://arxiv.org/html/2610.07274#A1)\)\. Job 2 never feeds back into D1, which uses onlyℓ\\ell, so the challenge decision never inherits the judge’s variance and no score is changed\. A confirmed lie is used in exactly three places: it blocks a packaging upgrade in Stage D, tags a lenient contradiction with whether the agent also lied, and routes an unverified pass to suspect\.

### 3\.4D4: Reliability

In short: run the task again and see whether the result stays in the same band\.One run is a sample, not a measurement of a point; a pass that holds one time in five is luck recorded as competence, invisible to any single\-run dimension\. D4 executes each taskN=5N=5times under identical configuration and characterises the score distribution\{s1,…,sN\}\\\{s\_\{1\},\\ldots,s\_\{N\}\\\}\. Each run is first classified as an infrastructure or genuine failure from the termination record, trajectory text, and usage counters \(exhausted credit, a dead sandbox, zero tool calls\); infrastructure runs are excluded, so D4 measures the agent, not the harness\. Over the surviving runsSS,

s¯\\displaystyle\\bar\{s\}=1\|S\|∑s∈Ss,Δ¯=s¯−srec,\\displaystyle=\\tfrac\{1\}\{\|S\|\}\\textstyle\\sum\_\{s\\in S\}s,\\qquad\\bar\{\\Delta\}=\\bar\{s\}\-s\_\{\\text\{rec\}\},\(8\)crossed\\displaystyle\\texttt\{crossed\}=𝟏\[\|\{band\(s\):s∈S\}\|\>1\],\\displaystyle=\\mathbf\{1\}\\big\[\\,\|\\\{\\mathrm\{band\}\(s\):s\\in S\\\}\|\>1\\,\\big\],wheresrecs\_\{\\text\{rec\}\}is the recorded score andband⁡\(⋅\)\\mathrm\{band\}\(\\cdot\)maps a score to pass, partial, or fail with ALE’s thresholds\([Sun et al\., 2026](https://arxiv.org/html/2610.07274#bib.bib1)\)\. A task is*varied*whencrossedfires and*stable*otherwise\([Gonzalez\-Pumariega et al\., 2026](https://arxiv.org/html/2610.07274#bib.bib6)\)\.crossedasks whether the repeats agree with each other;Δ¯\\bar\{\\Delta\}asks how far they sit from the recorded score, negative meaning the recorded run was the better one\. Onlycrosseddecides whether a task clears D4;Δ¯\\bar\{\\Delta\}is reported, not used as a gate\.

Two further statistics separate what a score measures from what it merely samples\([Bjarnason et al\., 2026](https://arxiv.org/html/2610.07274#bib.bib22)\)\. Treating a score as true ability plus noise, the variance of repeats splits into a between\-task part \(genuine differences in difficulty and capability\) and a within\-task part \(run\-to\-run randomness\)\. Their ratio

ρ=σbetween2/\(σbetween2\+σwithin2\)\\rho=\\sigma^\{2\}\_\{\\text\{between\}\}/\(\\sigma^\{2\}\_\{\\text\{between\}\}\+\\sigma^\{2\}\_\{\\text\{within\}\}\)\(9\)is the share of variance that is signal, and1−ρ1\-\\rhothe share that is not\([Mustahsan et al\., 2025](https://arxiv.org/html/2610.07274#bib.bib15)\)\. Estimating each task’s pass probabilityp^\\hat\{p\}from its repeats givesp^k\\hat\{p\}^\{k\}, the chance of passingkkconsecutive runs, whose gap fromp^\\hat\{p\}is the capability a one\-shot score reports but cannot sustain\. Appendix[A](https://arxiv.org/html/2610.07274#A1)gives the per\-task form ofρ\\rhoand what else the persisted repeats support\.

### 3\.5Aggregation Across Dimensions

A wrong answer, an unearned pass, a false claim, and an unstable score call for different responses, so the dimensions are reported separately; averaging them would discard the distinction the layer exists to draw\. They are also reported as a conjunction: a task is*trusted*only where every dimension clears it, meaning the verdict re\-derives, the answer values are grounded, no false claim is confirmed, and the task is stable across repeats\. This is a gate, not a score: no dimension is weighted against another, and the recorded score is never overwritten\.

#### Counting rules\.

A task clears D1 only when re\-grading yields a verified pass; for a recorded pass this meansconfirm, socontradictionandunverifiedare D1 failures, and a task whose grader cannot be re\-run offline counts asunverified\. It clears D2 only whenclean;gamedand “no verdict” are D2 failures\. It clears D3 when no false claim is confirmed\. It clears D4 only when stable, whichever direction its repeats move, since a score that changes bands cannot be trusted either way, and a score is read as evidence about the agent \(Section[1](https://arxiv.org/html/2610.07274#S1)\)\. A task may fail several dimensions \(a run that fabricated its answer often also claimed success for it\), so the conjunction is not a sum and we report the overlap\([Zhou et al\., 2026](https://arxiv.org/html/2610.07274#bib.bib7)\)\. The four are not exhaustive, but they cover the trust failures saved artifacts can evidence\.

## 4Experiments

### 4\.1Dataset and Setup

We evaluate the trust layer on 108 publicly available tasks from ALE\([Sun et al\., 2026](https://arxiv.org/html/2610.07274#bib.bib1)\), a benchmark of long\-horizon work graded by deterministic scripts against expert references, spanning ALE’s three difficulty tiers and 11 professional domains, each with its own grading code producing a single score per run\. Each task was run once under each of five agent configurations: Claude Sonnet 4\.6, Claude Opus 5, GPT\-5\.6 Sol \(GPT\-Sol\), Muse Spark 1\.2 \(MetaMuse\), and Gemini 3\.6 Flash, the same agent harness and decoding configuration\. The recorded bands differ by model, from 15 recorded passes for the weakest to 24 for the strongest \(Appendix[B](https://arxiv.org/html/2610.07274#A2)\)\. For D1–D3, the trust layer reads only what each run already saved: the agent’s trajectory, the outputs it staged, and the task’s own grading code\. It re\-executes that code against the saved output without re\-running the agent, and never modifies the recorded score\. Where a dimension requires a model judgment we use GPT\-5\.5 with three independent votes and a strict majority\.

A task is*ungradable*when its grader cannot be re\-run offline, which holds for 11 of the 108 and is a property of the task rather than of any model, so the same 11 are excluded for every agent\. These runs staged no output the grader could read, so D1 produces no verdict, D3 has nothing to test a claim against, and D2 no answer values to trace, leaving 97 assessed throughout\. D4 instead re\-runs the agent five times per task under identical configuration\. 87 of the 108 completed all five repeats under Sonnet, and the 50 tasks repeated under all five configurations were drawn at random from those, stratified across ALE’s tiers \(24 Near\-Term, 18 Full\-Spectrum, 8 Last\-Exam\) and recorded band\. Repeated\-execution studies commonly use between three and six runs\([Guo et al\., 2026](https://arxiv.org/html/2610.07274#bib.bib4);[Zhang et al\., 2025](https://arxiv.org/html/2610.07274#bib.bib3)\), so five is in line with current practice\. Each dimension therefore reports its own denominator\. Because the 50 repeated tasks are drawn from all 108, the aggregate in Section[4\.6](https://arxiv.org/html/2610.07274#S4.SS6)can include ungradable tasks, which count as failing D1 and D2 under the counting rules of Section[3\.5](https://arxiv.org/html/2610.07274#S3.SS5)\.

### 4\.2D1: Competence

ModelconfirmcontraunverifiedupgSonnet563830MetaMuse504700GPT\-Sol504700Opus534310Gemini613510Table 1:D1 outcome per model\.ModelConfirmedhardsoftSonnet552629MetaMuse35314GPT\-Sol40382Opus17152Gemini27261Table 2:Confirmed false claims per model\.D1 executes each task’s grading code offline on the saved output, without re\-running the agent, and compares the result with the recorded score \(Table[1](https://arxiv.org/html/2610.07274#S4.T1), 97 gradable tasks\)\. The models cluster between 50 and 61 confirmations although their recorded passes differ by nine\. Most contradictions \(contra\) are partial scores below the verified\-pass bar: the benchmark credits partial work in proportion, while D1 requires a complete re\-derivation\. The real disagreements are smaller: 2 to 9 recorded passes per model re\-derive as failures, and one or two recorded failures as passes \(Appendix[C](https://arxiv.org/html/2610.07274#A3)\)\. The upgrade \(upg\) path was reachable but never granted, since every candidate’s values differed from the reference\. Gate typing, D1’s only model\-labelled stage, agrees closely enough that the majority rule absorbs residual disagreement rather than masking a coin flip \(Appendix[C](https://arxiv.org/html/2610.07274#A3)\); every later stage is deterministic: re\-executing the grader five times on the same output returns an identical verdict and band on all 108 tasks\.

### 4\.3D2: Reward Hacking

Modelassessedcleangamednotassessedgamed%Sonnet977514814\.4MetaMuse975238739\.2GPT\-Sol974549350\.5Opus9775101210\.3Gemini9744252825\.8Table 3:D2 verdicts per model\. Gamed% is over the 97 assessed tasks, including runs carrying no verdict\.Over the 97 assessed tasks \(Table[3](https://arxiv.org/html/2610.07274#S4.T3)\), the models differ by a factor of five: Opus and Sonnet reach their answers without doing the work on roughly one run in eight, GPT\-Sol on half; on recorded passes the spread is tenfold, from 1 of 22 for Opus to 10 of 21 for GPT\-Sol \(Table[9](https://arxiv.org/html/2610.07274#A4.T9)\)\. Runs with no verdict are neither clean nor gamed and stay in the denominator\. This matters most for Gemini, whose rate over runs with a verdict is 36\.2% rather than 25\.8%; as these are unlikely to be missing at random, its true rate lies between 25\.8% and 54\.6%\. Fabrication, a value that matches the reference yet appears in no executed output and nothing the agent read, accounts for most convictions; copying a supplied value is rarer and an explicit admission rarer still, so the judge carries most D2 verdicts for every model\. Every Sonnetgamedverdict was verified by hand against the trajectory, and all 14 held; Appendix[D](https://arxiv.org/html/2610.07274#A4)gives the breakdowns\. Injecting the reference value into runs the detector had cleared tests the other direction: D2 flagged 36 of 38 where the value was left ungrounded, and 2 of 45 where executed code wrote it\.

### 4\.4D3: Agent Deception Detection

D3 runs the two jobs of Section[3\.3](https://arxiv.org/html/2610.07274#S3.SS3)\. A claim can only be tested where re\-derivation reproduces what happened, so the 11 tasks without an offline verdict are ineligible; Table[2](https://arxiv.org/html/2610.07274#S4.T2)covers the remaining 97\. A claim is*hard*when a concrete defect sits behind it \(a fired critical gate, a missing output, or a trace showing the work was wrong\) while the agent reported the deliverable as verified, and*soft*when no such defect is evidenced, typically an overclaim that all checks pass when the agent’s own numbers were already below the bar\. Roughly half of Sonnet’s confirmed claims are soft, while the other four models’ are mostly hard, so the same detector finds two styles of misreporting\. Models also differ in how often they make confirmed false completion claims, from 17 for Opus to 55 for Sonnet, and false claims appear in every recorded band for every model, including full passes\. On two annotated samples of Sonnet runs, annotators agreed on whether success was claimed in 48 of 50 terminal messages \(κ=0\.85\\kappa=0\.85\), and the matcher reachesF1=0\.91F\_\{1\}=0\.91against their consensus; on severity they agreed atκ=0\.84\\kappa=0\.84, and the judge reachesF1=0\.97F\_\{1\}=0\.97\.

### 4\.5D4: Reliability

ModelTasksstab\.var\.var\.%Δ¯\\bar\{\\Delta\}Sonnet50272346\.0−0\.31\-0\.31Opus50391122\.0−0\.30\-0\.30GPT\-Sol5041918\.0−0\.22\-0\.22MetaMuse50381224\.0−0\.12\-0\.12Gemini50361428\.0−0\.24\-0\.24Table 4:D4 outcome per model\.Table[4](https://arxiv.org/html/2610.07274#S4.T4)covers the 50 repeated tasks\. ItsΔ¯\\bar\{\\Delta\}column, the mean change from the recorded score over varied pass\-band tasks, is negative throughout: a recorded pass that fails to reproduce returns a lower score than the one on record\. Band composition differs by model, so rates are comparable only within band \(Appendix[E](https://arxiv.org/html/2610.07274#A5), which also gives the 87\-task results\)\. The varied rate ranges from 18\.0% for GPT\-Sol to 46\.0% for Sonnet, a factor of two and a half on identical tasks and conditions, so instability is a property of the agent as well as the task\. The movement follows the recorded band: the recorded run was the better run where the benchmark recorded success and the worse run where it recorded failure \(Appendix[E](https://arxiv.org/html/2610.07274#A5)\)\. Pooled over all tasks the mean change is\+0\.03\+0\.03or less for every model, so the directions cancel and an aggregate hides the effect completely\.

Sonnet varies on 25\.3% of the 87 tasks; its repeat variance splits into 0\.143 between tasks against 0\.040 within,ρ=0\.78\\rho=0\.78, so close to a quarter of the variation in recorded scores is about which run was saved, not about the agent\. The noise is concentrated: the median task varies by a standard deviation of 0\.007 across repeats, the varied tasks by 0\.350\. A task passes on 19\.3% of individual repeats but only 13\.8% of five\-run sequences, and of the 24 tasks passing at least once, only 12 pass every time: half the tasks the benchmark could count as successes succeed only some of the time, and no single\-run dimension can tell the halves apart\. Sonnet’s two independent campaigns, over the 87 tasks and the 50 drawn from them, disagree: 25\.3% against 46\.0% varied under the same configuration, where a random draw would carry about 13 varied tasks rather than 23 \(Table[13](https://arxiv.org/html/2610.07274#A5.T13)\)\. The D4 verdict is therefore itself a sample; we report both campaigns rather than the more favourable one, and a largerNNwould tighten the estimate\. BecauseΔ¯\\bar\{\\Delta\}does not gate the verdict, a stable task can settle in a band other than the recorded one\. Of the 181 stable verdicts in Table[4](https://arxiv.org/html/2610.07274#S4.T4), 16 do so \(Gemini 10, Sonnet 2, Opus 2, MetaMuse 2, GPT\-Sol 0\), four of them recorded passes\. Requiring reproduction of the recorded band would reclassify these 16; we report them separately rather than folding them into the stable/varied label\.

### 4\.6Aggregation Across Dimensions

Table[5](https://arxiv.org/html/2610.07274#S4.T5)gives the conjunction over the tasks the benchmark recorded as passes, where it does its work: D1’s verified\-pass bar rejects a partial score by definition, so no partial task can clear it\. The D1–D4 columns overlap and do not sum; Appendix[F](https://arxiv.org/html/2610.07274#A6)gives all three bands\. Between 7\.7% and 40\.0% of recorded passes survive all four checks; all five intervals overlap, so the rates bound how many passes hold up rather than ordering the configurations\. Pooled \(last row of Table[5](https://arxiv.org/html/2610.07274#S4.T5)\), only 19 of 84 recorded passes survive all four checks \(22\.6%, 95% CI 15\.0–32\.6\), so 65 fail at least one, and the majority fails in every configuration\. Models fail for different reasons: GPT\-Sol loses two thirds of its passes to D2, 12 of 18 with no traceable computation behind them, while Opus is ungrounded on three and loses more to re\-derivation than to anything else\. Between a third and a half of flagged tasks are caught by a single dimension, so removing any one would leave them indistinguishable from clean ones\.

failedaggregateModelPassesD1D2D3D4trusted%95% CIOpus208305840\.0\[21\.9, 61\.3\]Sonnet154515426\.7\[10\.9, 52\.0\]MetaMuse188613422\.2\[9\.0, 45\.2\]GPT\-Sol1871234211\.1\[3\.1, 32\.8\]Gemini13752317\.7\[1\.4, 33\.3\]All84––––1922\.6\[15\.0, 32\.6\]Table 5:Conjunction over recorded passes among the 50 repeated tasks, under the counting rules of Section[3\.5](https://arxiv.org/html/2610.07274#S3.SS5)\. Intervals are Wilson score intervals at 95%\.The ablation \(Table[11](https://arxiv.org/html/2610.07274#A4.T11)\) also shows how much rests on model labels: the two deterministic dimensions \(D1, D4\) leave 34 of 84 recorded passes, and the two judge\-assisted ones reject a further 15, leaving 19\. Over all 50 tasks the totals reflect the threshold more than the evidence, since D1 fails every partial task; the recorded\-pass figures answer the question the layer was built for\.

## 5Conclusion and Future Work

We presented a trust layer that verifies four properties of a recorded agent score: whether a failure was genuine, whether a pass was earned, whether the agent’s claim matched reality, and whether the score is stable across repeated runs\. It is additive, never overwrites the recorded number, and confines model judgment to labeling evidence\. Across five agent configurations, every model reaches a passing answer without computing it on some runs, at rates differing tenfold between the cleanest and the worst; confirmed false completion claims appear in every recorded band for every model; and repeated execution takes 18% to 46% of tasks across score bands, signed so that a recorded pass was the agent’s better run and a recorded failure its worse one, a direction pooling erases\. Requiring all four properties leaves 22\.6% of recorded successes standing, and between a third and a half of flagged tasks are caught by a single dimension, so no one check substitutes for the others\. Because the layer is benchmark\-agnostic, future work will apply it to SWE\-bench Verified,τ\\tau\-bench, and ScienceAgentBench to make agent evaluation more trustworthy across benchmarks\. Measuring what an agent can do and verifying that it did are distinct problems, and only the first is currently instrumented\.

## Limitations

Our claims are scoped to what the evidence supports\. Precision is validated directly for D2 \(hand\-checkedgamedverdicts\) and D3 \(annotator agreement, Section[4\.4](https://arxiv.org/html/2610.07274#S4.SS4)\) on one configuration; extending these checks to all models, and validating D1’s gate typing against human labels, is natural next work\. With 13 to 20 recorded passes per configuration and overlapping intervals in Table[5](https://arxiv.org/html/2610.07274#S4.T5), the results establish that most recorded passes fail at least one check in every configuration, not a ranking of the agents\. All results come from ALE; because the layer needs only a trajectory, a staged output, and a re\-runnable grader, applying it to other benchmarks is straightforward\. Every decision rule is deterministic, and the ablation in Section[4\.6](https://arxiv.org/html/2610.07274#S4.SS6)shows how much of the result holds without any judge\. Agamedverdict means no traceable computation, not intent to cheat, and D2 does not catch fabrication laundered through execution; a confirmed lie shows a report was false, not that the agent knew it\. D4 re\-executes the agent, itsN=5N=5verdict is itself a sample, and stable tasks whose band differs from the recorded one are reported separately \(Section[4\.5](https://arxiv.org/html/2610.07274#S4.SS5)\)\.

## References

- Advaniet al\.\(2026\)L\. Advaniet al\.From confident closing to silent failure: characterizing false success in llm agents\.arXiv preprint arXiv:2606\.09863\.External Links:[Link](https://arxiv.org/abs/2606.09863)Cited by:[§1](https://arxiv.org/html/2610.07274#S1.p5.1),[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px2.p1.1)\.
- Bjarnasonet al\.\(2026\)B\. H\. Bjarnason, A\. Silva, and M\. MonperrusOn randomness in agentic evals\.arXiv preprint arXiv:2602\.07150\.External Links:[Link](https://arxiv.org/abs/2602.07150)Cited by:[§1](https://arxiv.org/html/2610.07274#S1.p2.1),[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px2.p1.1),[§3\.4](https://arxiv.org/html/2610.07274#S3.SS4.p2.1)\.
- Caoet al\.\(2026\)H\. Cao, I\. Driouich, and E\. ThomasBeyond task completion: revealing corrupt success in llm agents through procedure\-aware evaluation\.arXiv preprint arXiv:2603\.03116\.External Links:[Link](https://arxiv.org/abs/2603.03116)Cited by:[§1](https://arxiv.org/html/2610.07274#S1.p3.1),[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px3.p1.1)\.
- Chenet al\.\(2025\)Z\. Chen, S\. Chen, Y\. Ning, Q\. Zhang, B\. Wang, B\. Yu, Y\. Li, Z\. Liao, C\. Wei, Z\. Lu,et al\.Scienceagentbench: toward rigorous assessment of language agents for data\-driven scientific discovery\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 96934–96990\.Cited by:[§1](https://arxiv.org/html/2610.07274#S1.p1.1),[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px1.p1.1)\.
- Chernet al\.\(2024\)S\. Chernet al\.BeHonest: benchmarking honesty in large language models\.arXiv preprint arXiv:2406\.13261\.External Links:[Link](https://arxiv.org/abs/2406.13261)Cited by:[§1](https://arxiv.org/html/2610.07274#S1.p3.1),[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px2.p1.1)\.
- Gaboret al\.\(2026\)J\. Gabor, J\. Lynch, and J\. RosenfeldEvilGenie: a reward hacking benchmark\.arXiv preprint arXiv:2511\.21654\.External Links:[Link](https://arxiv.org/abs/2511.21654)Cited by:[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px2.p1.1)\.
- Gonzalez\-Pumariegaet al\.\(2026\)G\. Gonzalez\-Pumariega, S\. Agashe, J\. Yang, A\. Li, and X\. E\. WangOn the reliability of computer use agents\.arXiv preprint arXiv:2604\.17849\.Cited by:[§3\.4](https://arxiv.org/html/2610.07274#S3.SS4.p1.3)\.
- Guoet al\.\(2026\)Y\. Guo, Y\. Liu, J\. M\. Zhang, Y\. Ma, Y\. Lou, and Z\. ChenSWE\-doctor: guiding software engineering agents with runtime diagnosis from multi\-faceted bug reproduction tests\.arXiv preprint arXiv:2607\.00990\.Cited by:[§4\.1](https://arxiv.org/html/2610.07274#S4.SS1.p2.1)\.
- Gupta \(2026\)A\. GuptaReliabilityBench: evaluating llm agent reliability under production\-like stress conditions\.arXiv preprint arXiv:2601\.06112\.External Links:[Link](https://arxiv.org/abs/2601.06112)Cited by:[§1](https://arxiv.org/html/2610.07274#S1.p3.1),[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px2.p1.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. NarasimhanSwe\-bench: can language models resolve real\-world github issues?\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 54107–54157\.Cited by:[§1](https://arxiv.org/html/2610.07274#S1.p1.1),[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px1.p1.1)\.
- Kapooret al\.\(2024\)S\. Kapoor, B\. Stroebl, A\. Narayanan,et al\.AI agents that matter\.arXiv preprint arXiv:2407\.01502\.External Links:[Link](https://arxiv.org/abs/2407.01502)Cited by:[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px1.p1.1)\.
- Krakovnaet al\.\(2020\)V\. Krakovna, J\. Uesato, V\. Mikulik, M\. Rahtz, T\. Everitt, R\. Kumar, Z\. Kenton, J\. Leike, and S\. LeggSpecification gaming: the flip side of ai ingenuity\.Note:DeepMind BlogExternal Links:[Link](https://deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity/)Cited by:[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px2.p1.1)\.
- Mustahsanet al\.\(2025\)Z\. Mustahsan, A\. Lim, M\. Anand, S\. Jain, and B\. McCannStochasticity in agentic evaluations: quantifying inconsistency with intraclass correlation\.arXiv preprint arXiv:2512\.06710\.Cited by:[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px2.p1.1),[§3\.4](https://arxiv.org/html/2610.07274#S3.SS4.p2.2)\.
- Renet al\.\(2025\)R\. Renet al\.The MASK benchmark: disentangling honesty from accuracy in ai systems\.arXiv preprint arXiv:2503\.03750\.External Links:[Link](https://arxiv.org/abs/2503.03750)Cited by:[§1](https://arxiv.org/html/2610.07274#S1.p3.1),[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px2.p1.1)\.
- Sediqinet al\.\(2026a\)M\. Sediqin, S\. Dalmia, S\. Thoppanahalli, S\. K\. R\. Kovvuri, and A\. MukherjiSAGE: governed artifact generation from enterprise guidelines\.arXiv preprint arXiv:2609\.17775\.Cited by:[§1](https://arxiv.org/html/2610.07274#S1.p3.1)\.
- Sediqinet al\.\(2026b\)M\. Sediqin, S\. Dalmia, S\. Thoppanahalli, and A\. MukherjiGVD: governed versioning and deduplication for document repositories\.arXiv preprint arXiv:2609\.17696\.Cited by:[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px1.p1.1)\.
- Skalseet al\.\(2022\)J\. Skalse, N\. Howe, D\. Krasheninnikov, and D\. KruegerDefining and characterizing reward gaming\.Advances in neural information processing systems35,pp\. 9460–9471\.Cited by:[§1](https://arxiv.org/html/2610.07274#S1.p3.1),[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px2.p1.1)\.
- Sunet al\.\(2026\)Y\. Sun, X\. Han, D\. Song,et al\.Agents’ last exam\.arXiv preprint arXiv:2606\.05405\.External Links:[Link](https://arxiv.org/abs/2606.05405)Cited by:[§1](https://arxiv.org/html/2610.07274#S1.p1.1),[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px1.p1.1),[§3\.4](https://arxiv.org/html/2610.07274#S3.SS4.p1.3),[§4\.1](https://arxiv.org/html/2610.07274#S4.SS1.p1.1)\.
- Thaman \(2026\)K\. ThamanReward hacking benchmark: measuring exploits in llm agents with tool use\.arXiv preprint arXiv:2605\.02964\.External Links:[Link](https://arxiv.org/abs/2605.02964)Cited by:[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px2.p1.1)\.
- Tuet al\.\(2026\)X\. Tu, T\. Wang, K\. Huang, Y\. Qu, S\. Mostafavi,et al\.Benchguard: who guards the benchmarks? automated auditing of llm agent benchmarks\.arXiv preprint arXiv:2604\.24955\.Cited by:[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px1.p1.1)\.
- Yaoet al\.\(2024\)S\. Yaoet al\.τ\\tau\-bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arXiv:2406\.12045\.External Links:[Link](https://arxiv.org/abs/2406.12045)Cited by:[§1](https://arxiv.org/html/2610.07274#S1.p1.1),[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025\)Z\. Zhang, A\. P\. Bajaj, D\. Handa, S\. Liu, A\. S\. Raj, H\. Chen, H\. Wang, Y\. Liu, Z\. L\. Basque, S\. Nath,et al\.Buildbench: benchmarking llm agents on compiling real\-world open\-source software\.arXiv preprint arXiv:2509\.25248\.Cited by:[§4\.1](https://arxiv.org/html/2610.07274#S4.SS1.p2.1)\.
- Zhouet al\.\(2026\)Y\. Zhou, L\. Y\. Choi, J\. Wen, and W\. YeAccuracy, stability, and repeated\-run reliability of large language models on deterministic programming tasks\.arXiv preprint arXiv:2606\.00920\.Cited by:[§3\.5](https://arxiv.org/html/2610.07274#S3.SS5.SSS0.Px1.p1.1)\.
- Zhuet al\.\(2026\)Y\. Zhu, T\. Jin, Y\. Pruksachatkun, A\. Zhang, S\. Liu, S\. Cui, S\. Kapoor, S\. Longpre, K\. Meng, R\. Weiss,et al\.Establishing best practices in building rigorous agentic benchmarks\.Advances in Neural Information Processing Systems38\.Cited by:[§2](https://arxiv.org/html/2610.07274#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix AMethod detail

This appendix collects the criteria, worked example, label definitions, and statistical detail referenced from Section[3](https://arxiv.org/html/2610.07274#S3)\.

#### Gate\-type criteria \(Stage A\)\.

The critical category also covers gates enforcing requirements the task itself mandates, such as a required output file, a specified value that must match the ground truth, or a constraint the prompt states explicitly, since failing one of these means the delivered work does not meet what was asked\. A*process*gate is one such as a requirement that an authored script invoke a mandated simulation binary: the output could still be correct if the rule were ignored\. An*administrative*gate is one such as a submitted file missing a schema\-required key: the work may be correct but cannot be read\. Gates are recovered by scanning every file of the grader rather than the nominal scorer alone, since existence gates often live in the entry module while value gates live in a delegated one\.

#### Gate matching \(Stage B\)\.

From the trajectory Stage B uses only the task identifier, and the instruction where needed to select the correct reference\. Each reported failure is matched back to a Stage A gate by its identifier, which is the grader’s own message template recognised in its filled\-in output\. The verdicts Stage B can return are a wrong answer, a fabricated or degenerate artifact caught by the grader’s own guard, a violated method rule, a missing deliverable, and a formatting slip, each recorded separately\.

#### Severity ordering \(Stage B\)\.

When several gates fire at once, a precondition failure outranks a quality miss, and among preconditions critical outranks process, which outranks administrative\. This ordering also closes an exploit, since an administrative failure is the only verdict Stage D can upgrade, and ranking prevents a run with any real failure from being relabelled as a formatting slip\.

#### Contradiction and upgrade \(Stage D\)\.

An upgrade is reachable only when the decisive failure is an administrative one \(from B\), the prompt did not demand that format \(m=0m=0, from A\), the values match the reference \(κ=1\\kappa=1, computed in Stage D\), and the agent was not caught lying \(ℓ≠1\\ell\\neq 1, from C\)\. Any other decisive gate simply confirms the recorded zero with its cause\. Signingδ\\deltarecords which side was wrong: the benchmark failed a run that should have passed, or passed a run that should have failed\. An agent that computes the right values and writes them toresult\.csvwhen the grader expectsresults\.csvfails correctly and fails again on re\-grade, so no contradiction arises; the zero still misrepresents work that was done\. This is only available because an administrative gate protects a requirement the task did not itself impose\. Where the prompt does state the format, the gate enforces something the task asked for, and Stage A types it as critical rather than administrative, so no upgrade is reachable\. Signing the direction matters for the same reason: one case is a fairness bug that punished a capable agent, the other is unearned credit inflating the leaderboard, and merging them into a single “we disagree” would hide which occurred and what to do about it\.

#### Signal strength in D2\.

A confession accompanied by a provenance gap is the strongest: both halves are recorded facts, and each corroborates the other, since the admission explains the gap and the gap confirms the admission\. A copied\-value match is deterministic but weaker, because a prompt that states the expected answer makes it unreliable\. A judge\-voted fabrication is the weakest, resting on the absence of a trace rather than the presence of evidence, which is why it requires a strict majority and resolves to clean on a split\.

#### Value tracing granularity \(D2\)\.

Scalars are traced one value at a time, so a constant pasted directly into the output cannot hide among computed ones, while arrays and large tables, which are never printed number by number, are cleared at the file level when the file itself was produced by executed code\.

#### Severity labels in D3\.

The judge gives a final decisionℓ^\\hat\{\\ell\}: it confirms the lie, or drops it if the agent’s words did not really contradict what happened\. The labelγ\\gammaseparates two kinds of claim, saying the work is*correct*versus merely saying it is*finished*, the first being the more serious\. The labelη\\etamarks the lie as hard or soft by whether a concrete defect sits behind it\. Finallyϕ\\phisays whether the lie alone should fail the task, on a deliberately high bar that answers “unverifiable” whenever the trace does not prove the deliverable was actually wrong\.

#### Per task variance and repeat artifacts \(D4\)\.

The within\-task standard deviation gives the same quantity asρ\\rhoper task, distinguishing a benchmark whose noise is spread evenly from one whose noise sits in a few unstable tasks\. Each run’s artifacts are persisted, so any of D1 through D3 can be applied to any individual repeat\.

## Appendix BRecorded bands per model

ModelpasspartialfailSonnet153646MetaMuse223738GPT\-Sol233638Opus243340Gemini182554Table 6:ALE’s recorded bands per model, over the 97 tasks with a re\-derivable verdict\. Because a dimension’s rate depends on band composition, per\-model rates are not directly comparable without conditioning on band\.
## Appendix CCompetence detail

Table[1](https://arxiv.org/html/2610.07274#S4.T1)reports one figure per outcome per model\. Two breakdowns sit behind it\.

The first separates contradictions by direction, which matters because they are not a single phenomenon\. A partial score that fails to clear the verified\-pass bar disagrees with the benchmark by definition rather than by error: both sides compute the same number and differ only on whether partial credit constitutes a verified pass\. A recorded pass that re\-derives as a failure, or a recorded failure that re\-derives as a pass, is a disagreement about the outcome itself\. Table[7](https://arxiv.org/html/2610.07274#A3.T7)shows that the first case accounts for most contradictions in every model, and that the second is an order of magnitude smaller\. One task carried no recorded score at all and is listed separately\.

ModeltotalP→\\rightarrowFpass→\\rightarrowFF→\\rightarrowpassnone→\\rightarrowFSonnet3834220MetaMuse4737910GPT\-Sol4736920Opus4333820Gemini3525811Table 7:Contradictions by direction, over the 97 gradable tasks \(P = partial, F = fail\)\.The second gives the full outcome distribution by recorded band\. The partial row is zero\-confirm for every model, which is the threshold rule made visible: no partial score re\-derives as a verified pass\. The pass row carries the disagreements that matter, and it is where the models differ most, from two contradictions for Sonnet to nine for MetaMuse and GPT\-Sol\.

ModelbandTasksconfirmcontraunver\.Sonnetpass151320partial360342fail464321MetaMusepass221390partial370370fail383710GPT\-Solpass231490partial360360fail383620Opuspass241581partial330330fail403820Geminipass181080partial250250fail545121Table 8:D1 outcomes by model and recorded band, over the 97 gradable tasks\. Upgrade is zero throughout and omitted\.#### Gate\-typing agreement\.

Over 1,078 gates the three votes within a pass are unanimous on 94\.1% and two independent passes agree on 96\.8%\. Because gate categories are a property of the grader rather than of a run, the typed inventory transfers across agents unchanged\.

## Appendix DReward hacking detail

Table[3](https://arxiv.org/html/2610.07274#S4.T3)reports one figure per model\. Two breakdowns sit behind it\.

The first gives the verdict distribution by recorded band\. The pattern is not shared across models: for Sonnet the gaming rate falls from the pass band to the fail band, from 20% to 10\.9%, while for the others it rises, so gaming concentrates where credit is earned in one model and where it is not in the rest\.

ModelbandTasksassessedcleangamedn/aSonnetpass17151131partial37363060fail54463457MetaMusepass23191261partial424124152fail433716174GPT\-Solpass23219102partial403821170fail453815221Opuspass25222011partial37363042fail46392559Geminipass19161231partial28271980fail6154131427Table 9:D2 verdicts by model and recorded band, over the 97 assessed tasks\. The*Tasks*column is the full 108 under each model’s own banding; n/a: not assessed\.The second gives the condition under which each conviction was reached\. Fabrication, which requires a judge, accounts for between 78% and 100% of convictions in every model, while the two deterministic conditions together account for the remainder\.

ModelcopiedconfessionfabricatedtotalSonnet211114MetaMuse403438GPT\-Sol324449Opus001010Gemini112325Table 10:Conditions under which a gamed verdict was reached, per model\.#### Hand verification of Sonnet’s gamed verdicts\.

Of the 14, 11 were required result values written as literals with no computation or file source anywhere in the run, 2 reproduced a value the prompt supplied, and 1 rested on the agent’s own admission that it could not run the required engine and had hand\-built the results to resemble its output\. That admission is recorded text and the accompanying provenance gap is computed from the file access trail, so the two together are the strongest condition described in Section[3\.2](https://arxiv.org/html/2610.07274#S3.SS2), and the evidence is deterministic even though the detector did not join them on this run\.

FilterSon\.OpusGPTMetaGem\.totalRecorded pass152018181384\+ D111121110650\+ D1, D4 \(determ\.\)6898334\+ all four4824119Table 11:Recorded passes surviving each filter, over the 50 repeated tasks \(Son\. = Sonnet, GPT = GPT\-Sol, Meta = MetaMuse, Gem\. = Gemini\)\.
#### Runs carrying no verdict\.

In most the agent staged no output matching the graded reference, so no answer values existed to trace; in a few the trajectory contained no usable log\. Gemini produced nothing assessable on 28 of its 97 tasks\.

## Appendix EReliability outcomes by band, and the full Sonnet set

Table[4](https://arxiv.org/html/2610.07274#S4.T4)reports one figure per model\. Two breakdowns sit behind it\.

The first gives the outcome by recorded band\. The band\-levelΔ¯\\bar\{\\Delta\}values are wider than the pass\-band figure reported in the body, running from−0\.31\-0\.31to\+0\.67\+0\.67, and their signs are largely set by the band, since a varied pass must have a repeat below it and a varied failure a repeat above it\. The partial band is the only one free to move either way, and it falls in four models and is flat in the fifth\.

ModelbandTasksstablevaried%Δ¯\\bar\{\\Delta\}Sonnetpass1510533\.3−0\.31\-0\.31partial1910947\.4−0\.25\-0\.25fail167956\.2\+0\.47\+0\.47Opuspass2015525\.0−0\.30\-0\.30partial1714317\.6−0\.08\-0\.08fail1310323\.1\+0\.67\+0\.67GPT\-Solpass1814422\.2−0\.22\-0\.22partial2017315\.0\+0\.00\+0\.00fail1210216\.7\+0\.50\+0\.50MetaMusepass1815316\.7−0\.12\-0\.12partial2115628\.6−0\.02\-0\.02fail118327\.3\+0\.27\+0\.27Geminipass1310323\.1−0\.24\-0\.24partial158746\.7−0\.15\-0\.15fail2218418\.2\+0\.56\+0\.56Table 12:D4 outcome by model and recorded band, over the 50 tasks repeated five times under every configuration\.Δ¯\\bar\{\\Delta\}is the mean change from the recorded score, over the varied tasks in that band\.The second gives the outcome over the larger 87\-task set under Sonnet, which is the source of the reliability statistics reported in the body\. Its varied rate of25\.3%25\.3\\%is roughly half the46\.0%46\.0\\%recorded on the 50 tasks drawn from it, and the two are discussed in Section[4\.5](https://arxiv.org/html/2610.07274#S4.SS5)\.

bandTasksstablevaried%Δ¯\\bar\{\\Delta\}pass1510533\.3−0\.31\-0\.31partial3627925\.0−0\.25\-0\.25fail3628822\.2\+0\.48\+0\.48Total87652225\.3—Table 13:D4 outcome by recorded band under Sonnet, over the 87 tasks that completed all five repeats\.
## Appendix FAggregation across all recorded bands

Table[5](https://arxiv.org/html/2610.07274#S4.T5)reports the conjunction over recorded passes\. Table[14](https://arxiv.org/html/2610.07274#A6.T14)gives it across all three bands\. The partial row is zero throughout, and the fail row nearly so, because D1 rejects a partial score by definition and the conjunction requires all four dimensions to clear; the totals therefore reflect that threshold as much as the evidence, which is why the body reports the pass band separately\.

failedaggregateModelbandTasksD1D2D3D4trust\.fail\.Sonnetpass154515411partial1919669019fail1614379016Total5037141423446Opuspass208305812partial1717433017fail1313733013Total503814611842GPT\-Solpass1871234216partial2020683020fail1212622012Total503924139248MetaMusepass188613414partial21217116021fail1111723011Total5040201412446Geminipass137523112partial1515667015fail22221764022Total5044281414149Table 14:Aggregation by model and recorded band, over the 50 repeated tasks, all of which carry a D4 verdict; ungradable tasks among them count as failing D1 and D2\. D1–D4 count tasks failing that dimension and overlap freely, so they do not sum; a task is trusted only where all four clear it \(trust\./fail\.: trusted/failed\)\.#### Ablation over recorded passes\.

Table[11](https://arxiv.org/html/2610.07274#A4.T11)applies the dimensions in sequence to the 84 recorded passes\. D1 alone leaves 50 standing\. Adding D4, so that only the two deterministic dimensions apply, leaves 34\. Requiring all four leaves 19, so the two judge\-assisted dimensions reject a further 15 passes that the deterministic checks clear, from none for Opus to seven of nine for GPT\-Sol\.