SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs?

# SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs?
Source: [https://arxiv.org/html/2609.21190](https://arxiv.org/html/2609.21190)
George Ma1,Benjamin Mikek2,11footnotemark:1Haoyu Li3Ferhat Erata4Yuhao Zhang4Zeren Shui4Behrooz Omidvar Tehrani4Jun Huan4Murali Krishna Ramanathan4Somayeh Sojoudi1Hao Zhou4Anoop Deoras41UC Berkeley2Georgia Tech3UIUC4AWS AI Labs††thanks:Equal contribution\.

###### Abstract

Ensuring the correctness of LLM\-generated code is a core challenge for modern software engineering\. Benchmarks for agentic code generation check correctness with held\-out test suites, which are inherently incomplete and increasingly susceptible to memorization\. Formal verification avoids both problems, but existing work covers only standalone tasks whose specifications are given as input, not real issues, which touch large repositories and state intent in vague natural language\. We presentBenchproofer, a pipeline that turns a coding task with a known correct patch into a formally verified one: it writes a specification for the new code, summarizes the existing functions that code calls with axioms, and admits an instance only after mechanical and adversarial gates agree\. Applying it toSWE\-bench VerifiedyieldsSWE\-Proof,500500real issues whose correctness is formally verified rather than tested, and it extends toSWE\-bench Pro\. Across two frontier models, verification catches what tests miss: a quarter to a half of test\-passing patches admit counterexamples, which a structured natural\-language specification does not fix, while a correct formal one lifts resolution from85%85\\%to95%95\\%for Opus 4\.8\. Writing that specification is the hard part: models that must write their own gain nothing over an unaided baseline, and only62%62\\%of their specifications pass our audit\. The usual failure is faithfulness, a specification that constrains part of the required behavior and leaves the rest free\. Specification quality still tracks the outcome, failing on89%89\\%of unresolved instances against47%47\\%of resolved ones, making faithful specification synthesis a concrete open problem\.

### 1Introduction

The dominant paradigm for evaluating LLM\-based code generation agents is execution against a held\-out test suite: an agent is given a programming task and its output is checked against hidden test cases\. Testing is cheap, easy to implement, and reuses the tests already present in existing codebases\. It is the standard correctness metric for coding agent benchmarks\([Jimenez et al\., 2024](https://arxiv.org/html/2609.21190#bib.bib22);[OpenAI, 2024](https://arxiv.org/html/2609.21190#bib.bib38)\), and has driven rapid progress over the last two years\([Yang et al\., 2024](https://arxiv.org/html/2609.21190#bib.bib54);[Xia et al\., 2024](https://arxiv.org/html/2609.21190#bib.bib50);[Wang et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib47);[Zhang et al\., 2024b](https://arxiv.org/html/2609.21190#bib.bib62)\)\.

Tests are nevertheless a weak correctness criterion\. First, testing is inherently incomplete: a finite set of inputs cannot certify behavior on the ones it leaves out\. Audits ofSWE\-benchfind that many patches credited as correct pass only because the associated tests are too weak to separate a real fix from a superficial one\([Aleithan et al\., 2024](https://arxiv.org/html/2609.21190#bib.bib5)\)\. Second, agents can reward hack, exploiting that incompleteness to produce code that passes the given tests but is not general\.[Zhong et al\. \(2025\)](https://arxiv.org/html/2609.21190#bib.bib63)find test\-passing but incorrect solutions on a large proportion ofSWE\-benchand other benchmark problems\.

Formal verification addresses both problems\([Ye et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib57);[Thakur et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib45);[Loughridge et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib30)\)\. Existing approaches hand an agent a specification in a formal language and ask it for an implementation annotated with invariants and pre\- and post\-conditions, checked by a verifier such as Dafny\([Leino, 2010](https://arxiv.org/html/2609.21190#bib.bib27)\)or Verus\([Lattuada et al\., 2023](https://arxiv.org/html/2609.21190#bib.bib26);[Yang et al\., 2025a](https://arxiv.org/html/2609.21190#bib.bib52);[Chen et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib10)\)\. Verification certifies the program on its entire specified domain rather than on sampled inputs\.

A gap remains between these approaches and realistic coding tasks\. Verified generation has so far targeted small curated exercises\([Loughridge et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib30)\), standalone HumanEval\-style tasks\([Shefer et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib42)\), and competition mathematics\([Wu et al\., 2022](https://arxiv.org/html/2609.21190#bib.bib49)\), and most techniques take the formal specification as input\([Ye et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib57);[Thakur et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib45)\)\. Real programming tasks instead require new code to fit into large codebases with many files and dependencies, and state intent in vague natural language\. Neither existing approaches to verified code generationnor the benchmarks used to evaluate themapply a sound verification oracle to such tasks\.

We close this gap withSWE\-Proof, a benchmark that pairs realistic programming tasks with formal verification\.SWE\-Proofprovides ground truth formal specifications, implementations, and proofs of correctness for each of the500500natural language coding tasks inSWE\-bench Verified\([OpenAI, 2024](https://arxiv.org/html/2609.21190#bib.bib38)\), under three verification backends: theNaginiverifier for statically\-typed Python\([Eilers & Müller, 2018](https://arxiv.org/html/2609.21190#bib.bib14);[Müller et al\., 2016](https://arxiv.org/html/2609.21190#bib.bib37)\),Velvet, a DSL for imperative\-style program verification inLean\([Gladshtein et al\., 2026](https://arxiv.org/html/2609.21190#bib.bib19)\), and pureLean\([De Moura et al\., 2015](https://arxiv.org/html/2609.21190#bib.bib13)\), in which proof obligations are discharged by a proof\-writing agent rather than by an SMT solver\.

We buildSWE\-ProofwithBenchproofer, a construction pipeline that turns a coding task with a known ground truth solution into a formally verified one\. It rests on two ideas\. First, it keeps verification tractable by modeling only the new or modified code and summarizing its calls into unchanged functions with*axioms*, overapproximative summaries of the callee properties the fix relies on\. Second, it admits an instance only once a suite of correctness gates agrees, combining mechanical verification with adversarial LLM auditors\. We releaseBenchprooferwithSWE\-Proof, and show it generalizes by constructing a verified version ofSWE\-bench Pro\.

We then evaluate two frontier models, Claude Opus 4\.8\([Anthropic, 2026](https://arxiv.org/html/2609.21190#bib.bib6)\)and GPT\-5\.5\([OpenAI, 2026](https://arxiv.org/html/2609.21190#bib.bib39)\), and report five findings\. Between a quarter and a half of patches that pass every hidden test are still flawed, and a structured natural language “specification” does not close that gap\. Asking an agent to write a specification and verify against it does not improve resolution over an unaided baseline, but supplying a correct one raises it from85\.0%85\.0\\%to95\.1%95\.1\\%for Opus 4\.8 and from81\.2%81\.2\\%to94\.7%94\.7\\%for GPT\-5\.5\. Specification synthesis is the bottleneck: only62%62\\%of synthesized specifications survive our adversarial audit, and the dominant failure is faithfulness, a specification that constrains part of the required behavior and leaves the rest free\. That quality tracks resolution:89%89\\%of specifications from unresolved instances fail the audit, against47%47\\%from resolved ones\.

In summary, we contributeBenchproofer, a construction pipeline that turns an agentic coding task with a known ground truth solution into ground truth specifications, implementations, and proofs, using environment axiomatization and a sequence of mechanical and adversarial gates \([Section3](https://arxiv.org/html/2609.21190#S3)\);SWE\-Proof, to our knowledge the first benchmark to attach a verification oracle to real\-world software engineering tasks, covering all500500instances ofSWE\-bench Verifiedand extended to the Python fragment ofSWE\-bench Pro; and an evaluation that identifies specification synthesis as the key unmet challenge for verified code generation \([Section4](https://arxiv.org/html/2609.21190#S4)\)\.

### 2Overview

Box 2\.1: Natural Language Taskax\.histdensity not auto\-scaled whenhisttype='step'\.The histogram’s density axis is not automatically adjusted to fit the whole histogram\. The issue occurs whenhisttype='step', but is otherwise fine for other histogram types\.

Box 2\.2: Formal Specification[⬇](data:text/plain;base64,LS0gcG9zdGNvbmRpdGlvbjogcmVzdWx0IGlzIHRoZSB0aWdodAotLSBib3VuZGluZyBib3ggb2YgZXZlcnkgcG9pbnQgaW4gYHZlcnRzYApkZWYgcHJvYmxlbV9zcGVjICh2ZXJ0cyA6IExpc3QgKFJhdCDDlyBSYXQpKQogICAgKHJlc3VsdCA6IE9wdGlvbiAoUmF0w5dSYXTDl1JhdMOXUmF0KSkgOiBQcm9wIDo9CiAgbWF0Y2ggcmVzdWx0IHdpdGgKICB8IG5vbmUgPT4gdmVydHMgPSBbXQogIHwgc29tZSAoeG1pbiwgeG1heCwgeW1pbiwgeW1heCkgPT4KICAgIHZlcnRzIOKJoCBbXSDiiKcKICAgICjiiIAgdiDiiIggdmVydHMsIHhtaW4g4omkIHYuMSDiiKcgdi4xIOKJpCB4bWF4IOKIpwogICAgICAgICAgICAgICAgICB5bWluIOKJpCB2LjIg4oinIHYuMiDiiaQgeW1heCkg4oinCiAgICAo4oiDIGEg4oiIIHZlcnRzLCBhLjEgPSB4bWluKSDiiKcKICAgICjiiIMgYiDiiIggdmVydHMsIGIuMSA9IHhtYXgpIOKIpwogICAgKOKIgyBjIOKIiCB2ZXJ0cywgYy4yID0geW1pbikg4oinCiAgICAo4oiDIGQg4oiIIHZlcnRzLCBkLjIgPSB5bWF4KQ==)–postcondition:resultisthetight–boundingboxofeverypointin‘verts‘defproblem\_spec\(verts:List\(Rat×Rat\)\)\(result:Option\(Rat×Rat×Rat×Rat\)\):Prop:=matchresultwith\|none=\>verts=\[\]\|some\(xmin,xmax,ymin,ymax\)=\>verts≠\[\]∧\(∀v∈verts,xmin≤v\.1∧v\.1≤xmax∧ymin≤v\.2∧v\.2≤ymax\)∧\(∃a∈verts,a\.1=xmin\)∧\(∃b∈verts,b\.1=xmax\)∧\(∃c∈verts,c\.2=ymin\)∧\(∃d∈verts,d\.2=ymax\)

Box 2\.3: Callee Axiom[⬇](data:text/plain;base64,b3BhcXVlIHNhbXBsZWRWZXJ0aWNlcyA6CiAgICBMaXN0IChSYXQgw5cgUmF0KSDihpIgQm9vbCDihpIgTGlzdCAoUmF0IMOXIFJhdCkKLS0ga2VlcC1hbGwgc2FtcGxpbmcgKGBmYWxzZWApIHJldHVybnMKLS0gZXhhY3RseSB0aGUgZGVmaW5pbmcgcG9pbnRzLCBpbiBvcmRlcgpheGlvbSBzYW1wbGVkVmVydGljZXNfa2VlcGFsbCA6IOKIgCB2ZXJ0cywKICAgIHNhbXBsZWRWZXJ0aWNlcyB2ZXJ0cyBmYWxzZSA9IHZlcnRz)opaquesampledVertices:List\(Rat×Rat\)→Bool→List\(Rat×Rat\)–keep\-allsampling\(‘false‘\)returns–exactlythedefiningpoints,inorderaxiomsampledVertices\_keepall:∀verts,sampledVerticesvertsfalse=verts

Box 2\.4: Implementation[⬇](data:text/plain;base64,ZGVmIGltcGxlbWVudGF0aW9uICh2ZXJ0cyA6IExpc3QgKFJhdCDDlyBSYXQpKSA6CiAgICBPcHRpb24gKFJhdMOXUmF0w5dSYXTDl1JhdCkgOj0gSWQucnVuIGRvCiAgLS0ga2VlcC1hbGw6IHJldGFpbiBldmVyeSBwb2ludAogIG1hdGNoIChzYW1wbGVkVmVydGljZXMgdmVydHMgZmFsc2UpIHdpdGgKICB8IFtdID0+IHJldHVybiBub25lCiAgfCBwMCA6OiByZXN0ID0+CiAgICBsZXQgbXV0IHhtaW4gOj0gcDAuMTsgbGV0IG11dCB4bWF4IDo9IHAwLjEKICAgIGxldCBtdXQgeW1pbiA6PSBwMC4yOyBsZXQgbXV0IHltYXggOj0gcDAuMgogICAgZm9yIHYgaW4gcmVzdCBkbwogICAgICBpZiB2LjEgPCB4bWluIHRoZW4geG1pbiA6PSB2LjEKICAgICAgaWYgdi4xID4geG1heCB0aGVuIHhtYXggOj0gdi4xCiAgICAgIGlmIHYuMiA8IHltaW4gdGhlbiB5bWluIDo9IHYuMgogICAgICBpZiB2LjIgPiB5bWF4IHRoZW4geW1heCA6PSB2LjIKICAgIHJldHVybiBzb21lICh4bWluLCB4bWF4LCB5bWluLCB5bWF4KQ==)defimplementation\(verts:List\(Rat×Rat\)\):Option\(Rat×Rat×Rat×Rat\):=Id\.rundo–keep\-all:retaineverypointmatch\(sampledVerticesvertsfalse\)with\|\[\]=\>returnnone\|p0::rest=\>letmutxmin:=p0\.1;letmutxmax:=p0\.1letmutymin:=p0\.2;letmutymax:=p0\.2forvinrestdoifv\.1<xminthenxmin:=v\.1ifv\.1\>xmaxthenxmax:=v\.1ifv\.2<yminthenymin:=v\.2ifv\.2\>ymaxthenymax:=v\.2returnsome\(xmin,xmax,ymin,ymax\)

Box 2\.5: Proof[⬇](data:text/plain;base64,dGhlb3JlbSBjb3JyZWN0IDog4oiAIHZlcnRzLAogICAgcHJvYmxlbV9zcGVjIHZlcnRzIChpbXBsZW1lbnRhdGlvbiB2ZXJ0cykgOj0gYnkKICBpbnRybyB2ZXJ0cwogIGNhc2VzIGhzIDogc2FtcGxlZFZlcnRpY2VzIHZlcnRzIGZhbHNlIHdpdGgKICB8IG5pbCAgPT4gLS0gZW1wdHk6IGBpbXBsZW1lbnRhdGlvbmAgPSBub25lLAogICAgICAgICAgICAtLSBhbmQgYHZlcnRzID0gW11gIGJ5IHRoZSBheGlvbQogICAgcncgW25vbmVfYnJhbmNoIHZlcnRzIGhzXQogICAgc2ltcCBvbmx5IFtwcm9ibGVtX3NwZWNdCiAgICBleGFjdCAoc2FtcGxlZFZlcnRpY2VzX2tlZXBhbGxfbmlsIHZlcnRzKS5tcCBocwogIHwgY29ucyBwMCByZXN0ID0+IC0tIGxvb3AgPSBjb21wb25lbnR3aXNlIGJveAogICAgcncgW2ltcGxfc29tZV9jaGFyIHZlcnRzIHAwIHJlc3QgaHNdCiAgICBzaW1wIG9ubHkgW3Byb2JsZW1fc3BlY10KICAgIGV4YWN0IOKfqHZlcnRzX25lX25pbCAuLiwgYm91bmRzX2FsbCAuLiwKICAgICAgICAgICBjb29yZF9hdHRhaW5lZCAuLuKfqQ==)theoremcorrect:∀verts,problem\_specverts\(implementationverts\):=byintrovertscaseshs:sampledVerticesvertsfalsewith\|nil=\>–empty:‘implementation‘=none,–and‘verts=\[\]‘bytheaxiomrw\[none\_branchvertshs\]simponly\[problem\_spec\]exact\(sampledVertices\_keepall\_nilverts\)\.mphs\|consp0rest=\>–loop=componentwiseboxrw\[impl\_some\_charvertsp0resths\]simponly\[problem\_spec\]exact⟨verts\_ne\_nil\.\.,bounds\_all\.\.,coord\_attained\.\.⟩

Figure 1:TheSWE\-Proofartifacts formatplotlib\-24177withLeanas the backend\. Box[1](https://arxiv.org/html/2609.21190#S2.F1)is taken fromSWE\-bench Verified; all other artifacts are newly\-constructed forSWE\-Proof\.We illustrateSWE\-Proofwith one instance,matplotlib\-24177fromSWE\-bench Verified\([Jimenez et al\., 2024](https://arxiv.org/html/2609.21190#bib.bib22)\), whose artifacts are shown in[Figure1](https://arxiv.org/html/2609.21190#S2.F1)\.

##### Task and specification

The input is a natural language bug report \(Box[1](https://arxiv.org/html/2609.21190#S2.F1)\): the bounds of a generated plot are computed from a simplified vertex set, so parts of the plot are cut off\.SWE\-Proofadds a formal specification of the intended behavior \(Box[1](https://arxiv.org/html/2609.21190#S2.F1)\), here stating that each side of the bounding box is tightly aligned with the furthest outlying point inverts\.

##### Axioms

Generating a specification and proof for every function the fix calls, and their transitive closure, is infeasible\. We therefore axiomatize each*unchanged callee*, a function the new code calls but does not modify\. Box[1](https://arxiv.org/html/2609.21190#S2.F1)summarizessampledVertices, asserting only that its keep\-all mode returns exactly the defining points \(verts\), in order\.

##### Implementations and patch

A verifier can only check code in its backend’s language, so each instance carries the fix in that language \(Box[1](https://arxiv.org/html/2609.21190#S2.F1)\), here aLeanloop that widens the bounds as it walks the vertices\. It carries two implementations: a*reference implementation*, which must verify, and a*pre\-fix implementation*, which must fail to verify and so witnesses that the specification rules the bug out\. Since the hidden tests run against the repository, each instance also carries an*equivalent patch*, the Python diff making the same change;[Claim1](https://arxiv.org/html/2609.21190#Thmclaim1)guarantees that the equivalent patch of a verified implementation resolves the issue\.

##### Proof

Box[1](https://arxiv.org/html/2609.21190#S2.F1)excerpts theLeanproof that the implementation satisfies the specification, using the axiom of Box[1](https://arxiv.org/html/2609.21190#S2.F1)as a lemma\. Once the compiler accepts it, the proof certifies the implementation subject to the axioms\.[Section3\.2](https://arxiv.org/html/2609.21190#S3.SS2)describes how we synthesize and validate all of these artifacts\.

##### Backends and challenges

SWE\-Proofsupports three verification backends\. UnderLeanthe proof is agent\-written and compiler\-checked, whileNaginiandVelvetdischarge the obligations with an SMT solver, which is faster but offers no recourse when the solver times out; we compare the trade\-offs in Appendix[D](https://arxiv.org/html/2609.21190#A4)\. The key challenge is ensuring these artifacts are correct and free of reward hacking and hallucination, which is what the gates of[Section3](https://arxiv.org/html/2609.21190#S3)check\.

### 3Methodology

[Figure2](https://arxiv.org/html/2609.21190#S3.F2)gives an overview ofBenchproofer, the construction pipeline we use to generate the ground truth artifacts forSWE\-Proof\.

![Refer to caption](https://arxiv.org/html/2609.21190v1/workflow.png)Figure 2:TheBenchprooferworkflow\. From a natural language issue description,Benchprooferbuilds a specification, an implementation, and a verification certificate, and admits each artifact only after it passes a suite of gates that combine mechanical checks with adversarial audits\.#### 3\.1The Guarantee

Every instance ofSWE\-Proofis built to establish a single claim, and the rest of this section exists to secure it\.

###### Claim 1\(Verify⇒\\RightarrowResolve\)\.

Consider an instance ofSWE\-Proofwith specificationSSand axiomsAA\. LetIIbe an implementation written in the language of the instance’s verification backend \(Nagini,Velvet, orLean\), and letPPbe a Python patch that is equivalent toIIin the repository \([Section3\.4](https://arxiv.org/html/2609.21190#S3.SS4)\)\. IfIIverifies againstSSunderAA, thenPPresolves the instance by passing the hidden tests inSWE\-bench\.

Verification buys more than the tests can: a verifiedIIis correct on*every input*SSadmits, including the ones no test exercises\. The equivalence ofIIandPPis a hypothesis of the claim rather than a consequence of it\. An agent evaluated onSWE\-Proofsupplies its own equivalent patch, and we report how we check it in[Section4](https://arxiv.org/html/2609.21190#S4); for the reference implementation we release, the equivalence gate of[Section3\.3](https://arxiv.org/html/2609.21190#S3.SS3)audits it against the ground\-truth patch\.

#### 3\.2Benchmark Construction

##### Specification synthesis\.

Realistic tasks state intent in natural language that is often vague, or open to several formal readings\. An agent turns that description into a formal specification and refines it in a feedback loop\. During construction, though never during evaluation \([Section4](https://arxiv.org/html/2609.21190#S4)\), it also sees the ground\-truth patch, the buggy code it replaces, the test patch, and the instance’sFAIL\_TO\_PASS\(F2P\) andPASS\_TO\_PASSlists, and calibrates the specification’s strength by asking whether an implementation could satisfy it and still fail an F2P test\. Rather than solve the issue from scratch, the agent renders the patched and the buggy code into the backend’s language, which yields the reference and pre\-fix implementations of[Section2](https://arxiv.org/html/2609.21190#S2)\.[Table1](https://arxiv.org/html/2609.21190#S3.T1)lists the components of an instance and shows which construction step and which gate touches each\.

##### Environment axiomatization\.

Verifying every function the new code calls, together with their transitive closure, quickly exceeds what automated verifiers can handle, and some callees are library routines whose source is unavailable\. We therefore summarize each unchanged callee with an axiom stating the properties the fix relies on\. Axioms are the boundary between what we verify and what we assume\.

Benchprooferproduces and validates them with an agentic procedure\. It first collects the unchanged calleesf1,f2,…f\_\{1\},f\_\{2\},\\ldotsthat the new implementation invokes, from the repository and from libraries alike, and for eachfif\_\{i\}builds a fuzzer that mixes random inputs with cases an agent writes\. An agent then drafts a candidate axiomAiA\_\{i\}, which two opposing loops refine\. The first checks*correctness*: the fuzzer exercisesfif\_\{i\}, and any input that violatesAiA\_\{i\}sends the axiom back for revision\. The second checks*usefulness*: the verifier tries to close the proof relating the reference implementation to the specification usingAiA\_\{i\}as a lemma, and failure sends it back as well\. The loops repeat until the axiom is both consistent with observed behavior and strong enough to finish the proof\. Unlike the specification, an axiom need not be precise: it may over\-approximatefif\_\{i\}, constraining less than the callee really guarantees, and still yield a correct instance, as long as the proof closes\.

Table 1:The components that each construction step and each correctness gate relates\. Columns are the components of one instance: a\) the originalSWE\-bench Verifiedartifacts, whose issue description and human\-written tests together express the intent, and b\) the newly\-constructedSWE\-Proofartifacts\. A∙\\bulletmarks each component the step or gate touches; a gate spanning multiple components audits their correspondence jointly\.

#### 3\.3Correctness Gates

No single check establishes[Claim1](https://arxiv.org/html/2609.21190#Thmclaim1)end to end, soBenchproofersecures it with a suite of*gates*\. Once specification synthesis and environment axiomatization produce a candidate instance, it must pass every gate before we accept it, and any counterexample sends it back for revision\. The gates are of two kinds:*mechanical*checks, such as running an SMT solver or theLeancompiler, and*adversarial*gates, in which LLM auditors try to refute the instance with a concrete counterexample\.[Table1](https://arxiv.org/html/2609.21190#S3.T1)lists the components each gate relates\.

##### Mechanical gates\.

The*verification check*confirms that the reference implementation verifies under the instance’s backend; under pureLeanthis means the agent\-written proof is accepted by the kernel\. The*discrimination check*pairs the same specification with the pre\-fix implementation and confirms that it fails to verify, which establishes that the specification is strong enough to catch the defect\. The*mutation check*extends this to first\-order mutations of the reference implementation, one operator at a time, and requires that the variants no longer verify; a surviving mutant means the specification misses a local perturbation, so we require a high kill rate\. The*hygiene check*confirms that the instance uses only what the verifier can check soundly: imports come from an allowlist, the specification is well formed, and the proof uses no escape hatch such asLean’ssorry\. The*resolution check*applies the ground\-truth patch in the official Docker harness and confirms that the instance resolves\. It tests the original benchmark rather than anything we build, which is what makes the pipeline safe to point at a new benchmark\.

##### Axiom\-soundness gate\.

This gate re\-runs the fuzzing check of[Section3\.2](https://arxiv.org/html/2609.21190#S3.SS2)with independent adversarial auditors, adjudicating their inputs by executing the real callee inside the instance’s Docker image, so a single observed violation rejects the axiom\. It also confirms that each axiomatized function is genuinely unchanged rather than one the fix modifies\. The resulting axiom tests ship as re\-runnable evidence\.

##### Conformance gate\.

This gate checks that the reference implementation agrees with the repository code it stands for\. It runs an executable*shadow*of the implementation against the real patched repository function inside the instance’s container, on at least10510^\{5\}generated inputs, and reports any disagreement\. UnderNaginithe shadow is the implementation with its annotations stripped, so nothing is translated; underVelvetandLean, whose languages cannot be executed, it is an agent\-written Python translation, which we treat as the weakest link in the guarantee\.

##### Soundness and completeness gates\.

A panel of independent adversarial auditors attacks the specification from two directions\. The*soundness*attack tries to show that the specification is too strong, by hunting for an input on which the specification and the repository code disagree; such an input means the specification has drifted from the behavior the repository actually has, so correct code need not verify\. The*completeness*attack tries to show that it is too weak, by building an incorrect implementation that verifies anyway, for instance one that satisfies a postcondition without doing the real work\. Together the two require that verification be achievable by correct code and unachievable by incorrect code\.

##### Property\-based falsification\.

This gate makes those two attacks reproducible\. For each modeled function, adversarial auditors write a*property\-based test suite*: implementations labeled with the verdict they should receive against the frozen specification, correct ones to verify and buggy ones to be rejected\. Each case is adjudicated by re\-running the verifier, so a disagreement with a label puts the specification at fault\. The suites ship with the benchmark as reusable evidence\.

##### Equivalence gate\.

This gate audits the correspondence between the reference implementation and the ground\-truth patch\. The auditor walks the documented mapping and confirms that each step is one of three behavior\-preserving operations: a*type refinement*, which replaces a repository type with one the verifier can express, such as a matrix row as a sequence of integers; an*axiomatized callee*, which replaces a call to an unchanged function with the axiom summarizing it; or an*identical operation*, which reproduces the patch verbatim\. Any other step is a divergence and sends the instance back, for example an axiom standing in for a callee the patch modifies\.

##### Leakage gate\.

A final gate guards a secondary property: the*specification view*, the specification with its axioms, is what an agent sees at evaluation time, and it must reveal nothing about the fix beyond what the issue already discloses\. A mechanical screen looks for views exposing patched identifiers, file paths, or narration of the fix, and an adversarial auditor then tries to reconstruct the ground\-truth patch from the view alone\. If either succeeds the instance is rejected, since a leaked view would hand the answer to an agent being evaluated onSWE\-Proof\.

#### 3\.4Boundaries of the Guarantee

Three boundaries limit where a true correspondence between artifacts can be established\. The first is the intent\-specification boundary: no guarantee can be established that a specification matches informal natural language intent\. The intent is supplied to construction as two inputs, the issue description and the human\-writtenSWE\-bench Verifiedtests, together with the ground\-truth patch that implements them \([Table1](https://arxiv.org/html/2609.21190#S3.T1)\)\. The property\-based falsification gate of[Section3\.3](https://arxiv.org/html/2609.21190#S3.SS3)is what guards the correspondence between the specification and that intent\.

The remaining two are the harder ones, and they share a cause: each backend verifies code in its own language while the repository is Python, so a formal artifact and the code it stands for can never be the same text\. The repository side has two parts, the unchanged code and the patch, and each is paired with a formal artifact\. No formal guarantee can establish the correspondence within either pair, so we close both with the adversarial gates of[Section3\.3](https://arxiv.org/html/2609.21190#S3.SS3)\. The second boundary pairs an axiom with the unchanged callee it summarizes, and rests on the axiom\-soundness gate together with the fuzzing procedure of[Section3\.2](https://arxiv.org/html/2609.21190#S3.SS2)\. The third pairs the reference implementation with the patch; several gates bear on it, but the conformance and equivalence gates address it directly\.

Each instance ships additional artifacts that can be used to reproduce the checking of the instance: the axiom tests and the property suite the gates produced, the operation\-by\-operation correspondence between the reference implementation and the patch, a record of which repository entity each modeled symbol abstracts, and the outcome of every gate\. Appendix[B](https://arxiv.org/html/2609.21190#A2)documents them and Appendix[H](https://arxiv.org/html/2609.21190#A8)walks through one instance, while[Section5](https://arxiv.org/html/2609.21190#S5)covers the remaining trust assumptions\.

#### 3\.5Extensibility

To see how farBenchproofergeneralizes, we applied it toSWE\-bench Pro, a second set of natural language GitHub issues whose tasks are considerably larger thanSWE\-bench Verified’s: they modify or add an average of nine functions and146146lines, against two functions and1414lines\.Benchprooferbuilt ground truth artifacts that pass every correctness gate under every backend for242242of the266266Python tasks\.

The2222tasks that fail under all backends fail for one reason: the patch does not alter behavior that pre\- and post\-condition specifications can observe\. Most only rename or relocate code, and the rest change signatures, internal data structures, or side effects outside the value domain\. The remaining two tasks are blocked by backend\-specific limits and build successfully elsewhere\. Appendix[A\.3](https://arxiv.org/html/2609.21190#A1.SS3)diagnoses each case, and Appendix[G\.1](https://arxiv.org/html/2609.21190#A7.SS1)reports theSWE\-bench Proevaluation across the full grid of settings\.Benchproofertherefore applies to any task with a known ground truth patch whose change an existing verifier can model\.

### 4Experiments

Our experiments ask what verification is worth\.[Section4\.2](https://arxiv.org/html/2609.21190#S4.SS2)asks whether the hidden tests are complete and, where they are not, whether a specification has to be formal to close the gap;[Section4\.3](https://arxiv.org/html/2609.21190#S4.SS3)whether an agent gains from writing and verifying its own specification, and[Section4\.4](https://arxiv.org/html/2609.21190#S4.SS4)whether being handed a correct one helps\.[Section4\.5](https://arxiv.org/html/2609.21190#S4.SS5)and[Section4\.6](https://arxiv.org/html/2609.21190#S4.SS6)then ask what goes wrong when models write specifications themselves\.

#### 4\.1Evaluation protocol

##### Settings\.

All settings share one agent scaffold, a shell and a submission action following[Yang et al\. \(2024\)](https://arxiv.org/html/2609.21190#bib.bib54), and differ only in what the agent is given and in what a pass requires;[Table2](https://arxiv.org/html/2609.21190#S4.T2)lists both\. The agent can run the in\-repository tests but never sees the hidden ones, and a sanitization step strips the version\-control history that would leak the fix\.

##### Metrics\.

Rows scored on tests alone report the*resolution rate*, the fraction of instances whose patch passes the hidden tests under the official harness\. Rows 3, 5, and 7 are scored jointly: an instance passes only when the patch resolves, the implementation verifies against the specification, and a majority of three adversarial auditors agree that the implementation corresponds to the patch\.

##### Models and verifiers\.

We evaluate Claude Opus 4\.8\([Anthropic, 2026](https://arxiv.org/html/2609.21190#bib.bib6)\)and GPT\-5\.5\([OpenAI, 2026](https://arxiv.org/html/2609.21190#bib.bib39)\)under identical scaffolds, prompts, and budgets, and run the full grid underNagini,Velvet, andLean\. Every number is over the full500500instances\.

Table 2:Evaluation settings and results over all500500SWE\-bench Verifiedinstances \(%\)\.Providedis what the agent is given: the*verifier*as a callable tool,*edit localization*\(which functions and files the fix touches\), and the ground\-truth*specification*view\.Evaluatedis what a pass requires\.*Tests*means the submitted patch resolves the issue under the officialSWE\-benchharness\.*Verify*means the submitted implementation verifies against a specification, the agent’s own in rows 3 and 5 or the ground\-truth one in row 7\. Rows 3, 5, and 7 further require a judge to confirm that the verified implementation and the submitted patch agree\. Rows 0, 1, and 4 supply neither a specification nor a verifier, so one run per model serves every backend \(gray rules\)\. Two marks carry extra meaning\. In row 1,✓∗\\checkmark^\{\*\}means the row 0 patches were re\-scored by an adversarial audit that looks for inputs on which the agent\-generated patch differ from the ground\-truth one, and an instance passes when no such input is found\. In rows 6 and 7,✓†\\checkmark^\{\\dagger\}means the provided specification already reveals the localization\. Finally, EARS is natural language and has no verifier, so in the EARS columns*Verify*is that same adversarial audit in row 1, and in the rows 3, 5 and 7 EARS agent works without a verifier tool\.

#### 4\.2Finding 1: Tests are incomplete; only formal specifications close the gap

Passing the hidden tests is not the same as being equivalent to the ground\-truth patch, because the suite is finite: a patch can satisfy every hidden test and still diverge from the ground\-truth patch on inputs no test exercises\. Rows 0 and 1 measure how often that happens\. For every patch Row 0 accepted, an adversarial auditor writes new tests to distinguish it from the ground\-truth patch, and a test counts only if it fails under the agent’s patch and passes under the ground\-truth one\. Such a test is evidence that the patch is wrong even though the suite accepted it\. Under this audit, 26\.8% of the patches that pass every test inSWE\-bench Verified’s are overturned \(Row 0 minus Row 1\), indicating that the benchmark’s test suites are substantially incomplete:passing all tests does not establish correctness\.

Rows 6 and 7 ask whether a specification closes that gap, and whether the specification has to be formal\. Both rows hand the agent a specification, with the structured EARS format\([Mavin et al\., 2009](https://arxiv.org/html/2609.21190#bib.bib34)\)as the non\-formal control \([SectionD\.4](https://arxiv.org/html/2609.21190#A4.SS4)\)\. Row 7 then checks the result against ground truth: the formal backends verify the implementation against the ground\-truth specification, while EARS uses the adversarial auditor above\. Both check the same ground\-truth specification, and verification is the stronger of the two: it proves that no violating input exists, where the audit only searches for one\. First, the formal backends close the gap: averaged over the three, a patch that resolves almost always survives verification as well, losing only0\.90\.9points for Opus 4\.8 and0\.30\.3for GPT\-5\.5 from Row 6 to Row 7\. Second, EARS does not close it:17\.817\.8points of Opus 4\.8’s pass rate and46\.046\.0of GPT\-5\.5’s come from patches that remain distinguishable from the ground\-truth patch despite passing every hidden test\. Rows 2 and 3 show the same split when the agent writes the specification itself, where the formal average loses0\.10\.1and0\.80\.8points against EARS’s27\.027\.0and49\.249\.2\.A structured non\-formal specification confers none of the improvement that a formal one does\.

#### 4\.3Finding 2: Self\-constructed specifications do not help

In the end\-to\-end setting \(Rows 2 and 3\) the agent is prompted to write a specification and verify an implementation against it before submitting its patch\. It gets no more information than the unaided baseline of Row 0\.Resolution does not improve over the baseline under any backend\(Row 2 against Row 0\)\. The best cell gains0\.60\.6points \(85\.6%85\.6\\%against85\.0%85\.0\\%, Opus 4\.8 underNagini\); the other five are flat or negative, the worst by1\.21\.2\. Row 3, which also requires the implementation to verify against the self\-constructed specification and to match the patch, stays within1\.61\.6points of Row 2 under every formal backend, so the bottleneck is not writing the implementation or the proof but the self\-constructed specification \([Section4\.5](https://arxiv.org/html/2609.21190#S4.SS5)\)\.

#### 4\.4Finding 3: A correct specification helps substantially

Handing the agentSWE\-Proof’s ground\-truth specification raises resolution sharply\(Row 6\): from85\.0%85\.0\\%to96\.2%96\.2\\%for Opus 4\.8 and from81\.2%81\.2\\%to94\.4%94\.4\\%for GPT\-5\.5 underNagini, and to94\.0%94\.0\\%and94\.8%94\.8\\%underVelvetand95\.2%95\.2\\%and95\.0%95\.0\\%underLean\. Some of the gain comes from localization, since the specification reveals which functions are in scope, but most is not: against Row 4, which supplies localization alone, the specification is still worth\+6\.9\+6\.9points for Opus 4\.8 and\+7\.7\+7\.7for GPT\-5\.5, averaged over the three backends\. The gain also survives the stricter scoring of Row 7 \([Section4\.2](https://arxiv.org/html/2609.21190#S4.SS2)\), where the equivalence judges accept the submitted patch as making the same change as the verified implementation on99\.5%99\.5\\%of the judged episodes \([Table24](https://arxiv.org/html/2609.21190#A7.T24)\), so these scores are not inflated by patches that diverge from the implementation they were derived from\.

The telling comparison is Row 7 against Row 3, where the agent had to write the specification itself \([Section4\.3](https://arxiv.org/html/2609.21190#S4.SS3)\)\. Being handed a correct one is worth\+11\.0\+11\.0points for Opus 4\.8 and\+14\.2\+14\.2for GPT\-5\.5 underNagini,\+8\.6\+8\.6and\+15\.0\+15\.0underVelvet, and\+9\.0\+9\.0and\+14\.8\+14\.8underLean\. The two settings differ only in whether the ground\-truth specification is given or the agent must construct it, so a gap this large shows thatproducing a correct specification is the bottleneck for verified code generation\.

#### 4\.5Finding 4: Specification synthesis is the bottleneck, and faithfulness is why

We score a specification against five properties\. Three are criteria proposed by[Feng et al\. \(2026\)](https://arxiv.org/html/2609.21190#bib.bib17), and constrain the specification of each function\. A specification is*admissible*if its precondition excludes no input the corrected code legitimately handles, since an over\-strong precondition lets an implementation verify on a shrunken domain\. It is*sound*if the intended behavior of each modeled function satisfies its contract, so a correct implementation can verify, and*complete*if no buggy implementation can verify against it\.

Two novel properties are specific to repairing code inside a repository\. A specification has*sound axioms*if every axiom is true of the real callee and describes only code the fix leaves alone, never behavior the fix introduces\. It is*faithful*if the functions it models cover the whole behavioral surface the issue requires\. The first three ask whether each modeled function is pinned down correctly; faithfulness asks whether enough of the task was modeled at all\. It matters because a real fix usually spans several functions and files, and whatever the specification leaves out stays unconstrained: a specification can be admissible, sound, and complete on the functions it models, yet an implementation that verifies against it still fails the issue by being wrong on the rest\.

Table 3:Specification synthesis by backend \(%\)\. The first row is the share of specifications passing the full five\-property audit; the rest are per\-property failure rates\.To locate the bottleneck of[Section4\.4](https://arxiv.org/html/2609.21190#S4.SS4), we isolate specification synthesis: the agent gets only the issue and the repository and must produce a specification and a witness implementation that verifies under it, with no patch\. Three adversarial auditors score it against the five properties, running candidate implementations through the verifier and probing the axioms against the real callees; it passes only if a majority accept all five\. Pass rates fall far below what the same models reach when a specification is supplied:46\.0%46\.0\\%for Opus 4\.8 and64\.2%64\.2\\%for GPT\-5\.5 underNagini,60\.0%60\.0\\%and72\.0%72\.0\\%underVelvet, and61\.4%61\.4\\%and69\.2%69\.2\\%underLean\. The failures are lopsided \([Table3](https://arxiv.org/html/2609.21190#S4.T3)\)\. Sound axioms, admissibility, and completeness almost never fail, each on at most3\.9%3\.9\\%of instances, so the specifications do reject the pre\-fix implementation and do not shrink the input domain\. Soundness fails more often, on13\.1%13\.1\\%of instances for Opus 4\.8 and15\.7%15\.7\\%for GPT\-5\.5\.*Faithfulness*dominates, failing on42\.5%42\.5\\%for Opus 4\.8 and30\.0%30\.0\\%for GPT\-5\.5 and remaining the most violated property in every cell\.Models write specifications that are correct on the functions they model but cover too few of them, so faithful specification synthesis is the open problem\.

#### 4\.6Finding 5: Specification correctness tracks patch correctness

[Section4\.5](https://arxiv.org/html/2609.21190#S4.SS5)scores synthesized specifications on their own; here we ask how that correctness relates to resolving the issue\. In the end\-to\-end setting of Rows 2 and 3 the model writes both a specification and a patch, so we re\-run the five\-property audit on the specification it wrote for itself and split the results by whether the patch resolved\. The auditors never see the patch or its test outcome, so their verdict is independent of the result\. The two groups separate sharply in every cell \([Table29](https://arxiv.org/html/2609.21190#A7.T29)\): averaged over the six cells, specifications from unresolved instances fail the audit89\.4%89\.4\\%of the time, against47\.3%47\.3\\%on resolved ones\. The gap holds in every cell for three of the five properties, soundness, completeness, and faithfulness, and is widest on the two that govern how much behavior a specification constrains, faithfulness \(44\.344\.3points on average\) and soundness \(26\.726\.7points\); faithfulness stays the most violated property on both sides\.Specification correctness tracks resolution, and faithfulness accounts for most of the gap\.

### 5Discussion and Limitations

The verify\-implies\-resolve claim is not a proof\. It rests on two audited trust points, axiom soundness and the fidelity of the refinement from the verified implementation to the executed patch \([Section3\.3](https://arxiv.org/html/2609.21190#S3.SS3)\), which we establish by execution, fuzzing, and adversarial audit\. Trust points of this kind are normal for verified systems: a verified kernel or compiler also holds only under assumptions its authors make explicit\. We release the evidence behind ours for readers to attack\. Three narrower limits apply\. First, each backend inherits the expressiveness of its language, and a few instances needed manual modeling where a faithful specification fell outside it, such as an import\-timing side effect a partial\-correctness logic cannot express\. Second, a few instances relax the secondary no\-leakage property \(Appendix[A\.4](https://arxiv.org/html/2609.21190#A1.SS4)\): their specification view reveals a little more about the fix than the issue does, which weakens the evaluation for them but leaves[Claim1](https://arxiv.org/html/2609.21190#Thmclaim1)intact\. Third, the specification audit is a calibrated panel of LLM auditors, so its verdicts are evidence rather than proof\.

### 6Conclusion

We presentedBenchproofer, a pipeline that turns a coding task with a known correct patch into a formally verified one, andSWE\-Proof, the benchmark it builds for all500500instances ofSWE\-bench Verifiedand the Python fragment ofSWE\-bench Pro, each instance pairing a formal specification with a verifying reference implementation and the ground\-truth patch\. Hidden tests miss a quarter to a half of the defects verification catches, and a correct specification lifts resolution well above the unaided baseline, but agents gain nothing writing one themselves: what they write misses part of the behavior the issue requires\. Faithful specification synthesis from informal intent is the open problem\.

#### Reproducibility statement

We will release both artifacts described in this paper\.Benchprooferis the construction pipeline of[Section3](https://arxiv.org/html/2609.21190#S3), including its gate suite, agent scaffolds, and prompts, so that the corpora can be rebuilt from scratch or the pipeline pointed at a new benchmark\.SWE\-Proofis the corpus it produced, and it ships, for every instance, the specification module, the verifying reference implementation, the pre\-fix implementation that must fail verification, the equivalent patch, the operation correspondence, the axiom provenance, and the recorded gate outcomes, so that each artifact and each step of the verify\-implies\-resolve guarantee can be re\-checked independently \([Section3\.3](https://arxiv.org/html/2609.21190#S3.SS3)and Appendix[B](https://arxiv.org/html/2609.21190#A2)\)\. Construction and evaluation settings, including scaffolds, budgets, and sanitization, are described in Appendix[F\.4](https://arxiv.org/html/2609.21190#A6.SS4); the specification\-synthesis audit procedure and its calibration are detailed in Appendix[E\.3](https://arxiv.org/html/2609.21190#A5.SS3); and licensing and intended use are set out in Appendix[J](https://arxiv.org/html/2609.21190#A10)\.

#### AI use statement

We used generative AI for three kinds of task that require disclosure\. The first is generating synthetic data\. The artifacts ofSWE\-Proof, including its specifications, axioms, reference and pre\-fix implementations, proofs, fuzzers, and property suites, are model\-generated and are admitted only once they pass the gates of[Section3\.3](https://arxiv.org/html/2609.21190#S3.SS3)\. This also covers assisting in the writing of proofs and supplying the ingredients each instance’s proof rests on, translating code between a repository’s language and each backend’s language, and cleaning and reformatting the upstream task data\. The second is implementing methods: the construction pipeline, the evaluation harness, and the analysis scripts were written with AI coding assistance\. The third is that the adversarial auditors and judges are themselves language models\. They are a component of the method rather than an authoring aid, and we describe them in[Section3\.3](https://arxiv.org/html/2609.21190#S3.SS3)and[Section4\.1](https://arxiv.org/html/2609.21190#S4.SS1)\.

We did not use generative AI to develop the conceptual framework of this work, to formulate or argue its claims, to propose or refine hypotheses, to design the methodology or the experiments, or to interpret results\. Under recommended disclosure, we used AI assistance to write and polish prose and to review code\.

We have reviewed all AI\-assisted work\. All code and all paper content have been checked by the authors\. Manually re\-deriving every released bundle is not tractable at this scale, so the corpus rests on the mechanical and adversarial gates of[Section3\.3](https://arxiv.org/html/2609.21190#S3.SS3), whose per\-instance outcomes we release; the instances we did inspect by hand were correct and legitimate\. We take responsibility for the final content of this work, including text, claims, and artifacts produced with the aid of generative AI\.

### References

- Agarwal et al\. \(2026\)Shubham Agarwal, Alexander Krentsel, Shu Liu, Mert Cemri, Audrey Cheng, Rui Meng, Tomas Pfister, Chun\-Liang Li, Sylvia Ratnasamy, Aditya Parameswaran, Matei Zaharia, Ion Stoica, and Mohsen Lesani\.Inductive Deductive Synthesis: Enabling AI to Generate Formally Verified Systems\.*arXiv preprint arXiv:2605\.23109*, 2026\.URL[https://arxiv\.org/abs/2605\.23109](https://arxiv.org/abs/2605.23109)\.
- Aggarwal et al\. \(2025\)Pranjal Aggarwal, Bryan Parno, and Sean Welleck\.AlphaVerus: Bootstrapping Formally Verified Code Generation through Self\-Improving Translation and Treefinement\.In*Proceedings of the 42nd International Conference on Machine Learning*, pp\. 587–615\. PMLR, 2025\.URL[https://proceedings\.mlr\.press/v267/aggarwal25a\.html](https://proceedings.mlr.press/v267/aggarwal25a.html)\.
- Ahmed et al\. \(2024\)Toufique Ahmed, Martin Hirzel, Rangeet Pan, Avraham Shinnar, and Saurabh Sinha\.TDD\-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?*arXiv preprint arXiv:2412\.02883*, 2024\.URL[https://arxiv\.org/abs/2412\.02883](https://arxiv.org/abs/2412.02883)\.
- Ahmed et al\. \(2025\)Toufique Ahmed, Jatin Ganhotra, Rangeet Pan, Avraham Shinnar, Saurabh Sinha, and Martin Hirzel\.Otter: Generating Tests from Issues to Validate SWE Patches\.In*Proceedings of the 42nd International Conference on Machine Learning*, 2025\.URL[https://arxiv\.org/abs/2502\.05368](https://arxiv.org/abs/2502.05368)\.
- Aleithan et al\. \(2024\)Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang\.SWE\-Bench\+: Enhanced Coding Benchmark for LLMs\.*arXiv preprint arXiv:2410\.06992*, 2024\.URL[https://arxiv\.org/abs/2410\.06992](https://arxiv.org/abs/2410.06992)\.
- Anthropic \(2026\)Anthropic\.Claude Opus 4\.8\.[https://www\.anthropic\.com/news/claude\-opus\-4\-8](https://www.anthropic.com/news/claude-opus-4-8), 2026\.Accessed 2026\-07\-02\.
- Badertdinov et al\. \(2025\)Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel\.SWE\-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents\.In*Proceedings of the 39th Conference on Neural Information Processing Systems*, 2025\.URL[https://arxiv\.org/abs/2505\.20411](https://arxiv.org/abs/2505.20411)\.
- Bursuc et al\. \(2025\)Sergiu Bursuc, Theodore Ehrenborg, Shaowei Lin, Lacramioara Astefanoaei, Ionel Emilian Chiosa, Jure Kukovec, Alok Singh, Oliver Butterley, Adem Bizid, Quinn Dougherty, Miranda Zhao, Max Tan, and Max Tegmark\.A benchmark for vericoding: formally verified program synthesis\.*arXiv preprint arXiv:2509\.22908*, 2025\.URL[https://arxiv\.org/abs/2509\.22908](https://arxiv.org/abs/2509.22908)\.
- Chakraborty et al\. \(2023\)Saikat Chakraborty, Shuvendu K\. Lahiri, Sarah Fakhoury, Madanlal Musuvathi, Akash Lal, Aseem Rastogi, Aditya Senthilnathan, Rahul Sharma, and Nikhil Swamy\.Ranking LLM\-Generated Loop Invariants for Program Verification\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, pp\. 9164–9175, 2023\.URL[https://arxiv\.org/abs/2310\.09342](https://arxiv.org/abs/2310.09342)\.
- Chen et al\. \(2025\)Tianyu Chen, Shuai Lu, Shan Lu, Yeyun Gong, Chenyuan Yang, Xuheng Li, Md Rakib Hossain Misu, Hao Yu, Nan Duan, Peng Cheng, Fan Yang, Shuvendu K Lahiri, Tao Xie, and Lidong Zhou\.Automated Proof Generation for Rust Code via Self\-Evolution\.In*Proceedings of the 13th International Conference on Learning Representations*, 2025\.URL[https://arxiv\.org/abs/2410\.15756](https://arxiv.org/abs/2410.15756)\.
- Chen et al\. \(2024\)Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou\.Teaching Large Language Models to Self\-Debug\.In*Proceedings of the 12th International Conference on Learning Representations*, 2024\.URL[https://arxiv\.org/abs/2304\.05128](https://arxiv.org/abs/2304.05128)\.
- Chen et al\. \(2026\)Zaoyu Chen, Jianbo Dai, Boyu Zhu, Jingdong Wang, Huiming Wang, Xin Xu, Haoyang Yuan, Zhijiang Guo, and Xiao\-Ming Wu\.CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation\.*arXiv preprint arXiv:2604\.12268*, 2026\.URL[https://arxiv\.org/abs/2604\.12268](https://arxiv.org/abs/2604.12268)\.
- De Moura et al\. \(2015\)Leonardo De Moura, Soonho Kong, Jeremy Avigad, Floris Van Doorn, and Jakob von Raumer\.The Lean Theorem Prover \(system description\)\.In*International Conference on Automated Deduction*, pp\. 378–388\. Springer, 2015\.URL[https://link\.springer\.com/chapter/10\.1007/978\-3\-319\-21401\-6\_26](https://link.springer.com/chapter/10.1007/978-3-319-21401-6_26)\.
- Eilers & Müller \(2018\)Marco Eilers and Peter Müller\.Nagini: A Static Verifier for Python\.In*International Conference on Computer Aided Verification*, pp\. 596–603\. Springer, 2018\.URL[https://link\.springer\.com/chapter/10\.1007/978\-3\-319\-96145\-3\_33](https://link.springer.com/chapter/10.1007/978-3-319-96145-3_33)\.
- Endres et al\. \(2024\)Madeline Endres, Sarah Fakhoury, Saikat Chakraborty, and Shuvendu K\. Lahiri\.Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?*Proceedings of the ACM on Software Engineering*, 1\(FSE\):1889–1912, 2024\.URL[https://arxiv\.org/abs/2310\.01831](https://arxiv.org/abs/2310.01831)\.
- Ernst et al\. \(2007\)Michael D\. Ernst, Jeff H\. Perkins, Philip J\. Guo, Stephen McCamant, Carlos Pacheco, Matthew S\. Tschantz, and Chen Xiao\.The Daikon system for dynamic detection of likely invariants\.*Science of Computer Programming*, 69\(1\-3\):35–45, 2007\.URL[https://www\.sciencedirect\.com/science/article/pii/S016764230700161X](https://www.sciencedirect.com/science/article/pii/S016764230700161X)\.
- Feng et al\. \(2026\)Yueyang Feng, Dipesh Kafle, Vladimir Gladshtein, Vitaly Kurin, George Pîrlea, Qiyuan Zhao, Peter Müller, and Ilya Sergey\.Certified Program Synthesis with a Multi\-Modal Verifier\.*arXiv preprint arXiv:2604\.16584*, 2026\.URL[https://arxiv\.org/abs/2604\.16584](https://arxiv.org/abs/2604.16584)\.
- First et al\. \(2023\)Emily First, Markus N\. Rabe, Talia Ringer, and Yuriy Brun\.Baldur: Whole\-Proof Generation and Repair with Large Language Models\.In*Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering*, pp\. 1229–1241, 2023\.URL[https://arxiv\.org/abs/2303\.04910](https://arxiv.org/abs/2303.04910)\.
- Gladshtein et al\. \(2026\)Vladimir Gladshtein, Vitaly Kurin, Yueyang Feng, Dipesh Kafle, George Pîrlea, Qiyuan Zhao, and Ilya Sergey\.Velvet: A Foundational Multi\-Modal Verifier for Imperative Programs in Lean\.In*International Conference on Computer Aided Verification*, 2026\.URL[https://ilyasergey\.net/assets/pdf/papers/velvet\-cav26\.pdf](https://ilyasergey.net/assets/pdf/papers/velvet-cav26.pdf)\.
- Jain et al\. \(2025\)Naman Jain, King Han, Alex Gu, Wen\-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar\-Lezama, Koushik Sen, and Ion Stoica\.LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code\.In*Proceedings of the 13th International Conference on Learning Representations*, 2025\.URL[https://arxiv\.org/abs/2403\.07974](https://arxiv.org/abs/2403.07974)\.
- Jiang et al\. \(2023\)Albert Q\. Jiang, Sean Welleck, Jin Peng Zhou, Wenda Li, Jiacheng Liu, Mateja Jamnik, Timothée Lacroix, Yuhuai Wu, and Guillaume Lample\.Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs\.In*Proceedings of the 11th International Conference on Learning Representations*, 2023\.URL[https://arxiv\.org/abs/2210\.12283](https://arxiv.org/abs/2210.12283)\.
- Jimenez et al\. \(2024\)Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan\.SWE\-bench: Can Language Models Resolve Real\-world Github Issues?In*Proceedings of the 12th International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=VTF8yNQM66](https://openreview.net/forum?id=VTF8yNQM66)\.
- Kamath et al\. \(2023\)Adharsh Kamath, Aditya Senthilnathan, Saikat Chakraborty, Pantazis Deligiannis, Shuvendu K\. Lahiri, Akash Lal, Aseem Rastogi, Subhajit Roy, and Rahul Sharma\.Finding Inductive Loop Invariants using Large Language Models\.*arXiv preprint arXiv:2311\.07948*, 2023\.URL[https://arxiv\.org/abs/2311\.07948](https://arxiv.org/abs/2311.07948)\.
- Kupferman & Vardi \(2003\)Orna Kupferman and Moshe Y\. Vardi\.Vacuity detection in temporal model checking\.*International Journal on Software Tools for Technology Transfer*, 4\(2\):224–233, 2003\.URL[https://link\.springer\.com/article/10\.1007/s100090100062](https://link.springer.com/article/10.1007/s100090100062)\.
- Lahiri \(2024\)Shuvendu K\. Lahiri\.Evaluating llm\-driven user\-intent formalization for verification\-aware languages\.In*Formal Methods in Computer\-Aided Design*, pp\. 142–147\. IEEE, 2024\.URL[https://arxiv\.org/abs/2406\.09757](https://arxiv.org/abs/2406.09757)\.
- Lattuada et al\. \(2023\)Andrea Lattuada, Travis Hance, Chanhee Cho, Matthias Brun, Isitha Subasinghe, Yi Zhou, Jon Howell, Bryan Parno, and Chris Hawblitzel\.Verus: Verifying Rust Programs using Linear Ghost Types\.*Proceedings of the ACM on Programming Languages*, 7\(OOPSLA1\):286–315, 2023\.URL[https://dl\.acm\.org/doi/abs/10\.1145/3586037](https://dl.acm.org/doi/abs/10.1145/3586037)\.
- Leino \(2010\)K\. Rustan M\. Leino\.Dafny: An Automatic Program Verifier for Functional Correctness\.In*International Conference on Logic for Programming Artificial Intelligence and Reasoning*, pp\. 348–370\. Springer, 2010\.URL[https://link\.springer\.com/chapter/10\.1007/978\-3\-642\-17511\-4\_20](https://link.springer.com/chapter/10.1007/978-3-642-17511-4_20)\.
- Li et al\. \(2024\)Zenan Li, Yifan Wu, Zhaoyu Li, Xinming Wei, Xian Zhang, Fan Yang, and Xiaoxing Ma\.Autoformalize Mathematical Statements by Symbolic Equivalence and Semantic Consistency\.In*Proceedings of the 38th Conference on Neural Information Processing Systems*, 2024\.URL[https://arxiv\.org/abs/2410\.20936](https://arxiv.org/abs/2410.20936)\.
- Liang et al\. \(2025\)Shanchao Liang, Spandan Garg, and Roshanak Zilouchian Moghaddam\.The SWE\-Bench Illusion: When State\-of\-the\-Art LLMs Remember Instead of Reason\.*arXiv preprint arXiv:2506\.12286*, 2025\.URL[https://arxiv\.org/abs/2506\.12286](https://arxiv.org/abs/2506.12286)\.
- Loughridge et al\. \(2025\)Chloe Loughridge, Qinyi Sun, Seth Ahrenbach, Federico Cassano, Chuyue Sun, Ying Sheng, Anish Mudide, Md Rakib Hossain Misu, Nada Amin, and Max Tegmark\.Dafnybench: A benchmark for formal software verification\.*Transactions on Machine Learning Research*, 2025\.ISSN 2835\-8856\.URL[https://openreview\.net/forum?id=yBgTVWccIx](https://openreview.net/forum?id=yBgTVWccIx)\.
- Ma et al\. \(2025\)Lezhi Ma, Shangqing Liu, Yi Li, Xiaofei Xie, and Lei Bu\.SpecGen: Automated Generation of Formal Program Specifications via Large Language Models\.In*IEEE/ACM 47th International Conference on Software Engineering*, pp\. 16–28\. IEEE, 2025\.URL[https://arxiv\.org/abs/2401\.08807](https://arxiv.org/abs/2401.08807)\.
- Ma et al\. \(2026\)Lezhi Ma, Shangqing Liu, Yi Li, Qiong Wu, Han Wang, and Lei Bu\.SpecSyn: LLM\-based Synthesis and Refinement of Formal Specifications for Real\-world Program Verification\.*arXiv preprint arXiv:2604\.21570*, 2026\.URL[https://arxiv\.org/abs/2604\.21570](https://arxiv.org/abs/2604.21570)\.
- Madaan et al\. \(2023\)Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark\.Self\-Refine: Iterative Refinement with Self\-Feedback\.In*Proceedings of the 37th Conference on Neural Information Processing Systems*, 2023\.URL[https://arxiv\.org/abs/2303\.17651](https://arxiv.org/abs/2303.17651)\.
- Mavin et al\. \(2009\)Alistair Mavin, Philip Wilkinson, Adrian Harwood, and Mark Novak\.Easy Approach to Requirements Syntax \(EARS\)\.In*2009 17th IEEE International Requirements Engineering Conference*, pp\. 317–322\. IEEE, 2009\.
- Misu et al\. \(2024\)Md Rakib Hossain Misu, Cristina V\. Lopes, Iris Ma, and James Noble\.Towards AI\-Assisted Synthesis of Verified Dafny Methods\.In*Proceedings of the ACM on Software Engineering*, pp\. 812–835, 2024\.URL[https://arxiv\.org/abs/2402\.00247](https://arxiv.org/abs/2402.00247)\.
- Mugnier et al\. \(2025\)Eric Mugnier, Emmanuel Anaya Gonzalez, Nadia Polikarpova, Ranjit Jhala, and Yuanyuan Zhou\.Laurel: Unblocking Automated Verification with Large Language Models\.*Proceedings of the ACM on Programming Languages*, 9\(OOPSLA1\):1519–1545, 2025\.URL[https://arxiv\.org/abs/2405\.16792](https://arxiv.org/abs/2405.16792)\.
- Müller et al\. \(2016\)Peter Müller, Malte Schwerhoff, and Alexander J\. Summers\.Viper: A Verification Infrastructure for Permission\-Based Reasoning\.In*International Conference on Verification, Model Checking, and Abstract Interpretation*, pp\. 41–62\. Springer, 2016\.URL[https://link\.springer\.com/chapter/10\.1007/978\-3\-662\-49122\-5\_2](https://link.springer.com/chapter/10.1007/978-3-662-49122-5_2)\.
- OpenAI \(2024\)OpenAI\.Introducing SWE\-bench Verified\.[https://openai\.com/index/introducing\-swe\-bench\-verified](https://openai.com/index/introducing-swe-bench-verified), 2024\.Accessed 2026\-07\-02\.
- OpenAI \(2026\)OpenAI\.GPT\-5\.5\.[https://openai\.com/index/introducing\-gpt\-5\-5/](https://openai.com/index/introducing-gpt-5-5/), 2026\.Accessed 2026\-07\-02\.
- Pan et al\. \(2025\)Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang\.Training Software Engineering Agents and Verifiers with SWE\-Gym\.In*Proceedings of the 42nd International Conference on Machine Learning*, 2025\.URL[https://arxiv\.org/abs/2412\.21139](https://arxiv.org/abs/2412.21139)\.
- Richter & Wehrheim \(2025\)Cedric Richter and Heike Wehrheim\.Beyond Postconditions: Can Large Language Models Infer Formal Contracts for Automatic Software Verification?*arXiv preprint arXiv:2510\.12702*, 2025\.URL[https://arxiv\.org/abs/2510\.12702](https://arxiv.org/abs/2510.12702)\.
- Shefer et al\. \(2025\)Aleksandr Shefer, Igor Engel, Stanislav Alekseev, Daniil Berezun, Ekaterina Verbitskaia, and Anton Podkopaev\.Can LLMs Enable Verification in Mainstream Programming?*arXiv preprint arXiv:2503\.14183*, 2025\.URL[https://arxiv\.org/abs/2503\.14183](https://arxiv.org/abs/2503.14183)\.
- Sosso et al\. \(2026\)Alessandro Sosso, Akhil Arora, and Bas Spitters\.Agentic Proving for Program Verification\.In*The 3rd AI for Math Workshop at the 43rd International Conference on Machine Learning*, 2026\.URL[https://openreview\.net/forum?id=rlWpAywUII](https://openreview.net/forum?id=rlWpAywUII)\.
- Sun et al\. \(2024\)Chuyue Sun, Ying Sheng, Oded Padon, and Clark Barrett\.Clover: Closed\-Loop Verifiable Code Generation\.In*International Symposium on AI Verification*, pp\. 134–155\. Springer, 2024\.URL[https://arxiv\.org/abs/2310\.17807](https://arxiv.org/abs/2310.17807)\.
- Thakur et al\. \(2025\)Amitayush Thakur, Jasper Lee, George Tsoukalas, Meghana Sistla, Matthew Zhao, Stefan Zetzsche, Greg Durrett, Yisong Yue, and Swarat Chaudhuri\.Clever: A Curated Benchmark for Formally Verified Code Generation\.*Proceedings of the 39th Conference on Neural Information Processing Systems*, 2025\.URL[https://arxiv\.org/abs/2505\.13938](https://arxiv.org/abs/2505.13938)\.
- Tu et al\. \(2026\)Haoxin Tu, Huan Zhao, Yahui Song, Mehtab Zafar, Ruijie Meng, and Abhik Roychoudhury\.Agentic Verification of Software Systems\.In*Proceedings of the ACM International Conference on the Foundations of Software Engineering*, 2026\.URL[https://arxiv\.org/abs/2511\.17330](https://arxiv.org/abs/2511.17330)\.
- Wang et al\. \(2025\)Xingyao Wang, Boxuan Li, Yufan Song, Frank F\. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H\. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig\.OpenHands: An Open Platform for AI Software Developers as Generalist Agents\.In*Proceedings of the 13th International Conference on Learning Representations*, 2025\.URL[https://arxiv\.org/abs/2407\.16741](https://arxiv.org/abs/2407.16741)\.
- Wu et al\. \(2024\)Haoze Wu, Clark Barrett, and Nina Narodytska\.Lemur: Integrating Large Language Models in Automated Program Verification\.In*Proceedings of the 12th International Conference on Learning Representations*, 2024\.URL[https://arxiv\.org/abs/2310\.04870](https://arxiv.org/abs/2310.04870)\.
- Wu et al\. \(2022\)Yuhuai Wu, Albert Q\. Jiang, Wenda Li, Markus N\. Rabe, Charles Staats, Mateja Jamnik, and Christian Szegedy\.Autoformalization with Large Language Models\.In*Proceedings of the 36th Conference on Neural Information Processing Systems*, 2022\.URL[https://arxiv\.org/abs/2205\.12615](https://arxiv.org/abs/2205.12615)\.
- Xia et al\. \(2024\)Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang\.Agentless: Demystifying LLM\-based Software Engineering Agents\.*arXiv preprint arXiv:2407\.01489*, 2024\.URL[https://arxiv\.org/abs/2407\.01489](https://arxiv.org/abs/2407.01489)\.
- Xin et al\. \(2024\)Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang\.DeepSeek\-Prover: Advancing Theorem Proving in LLMs through Large\-Scale Synthetic Data\.*arXiv preprint arXiv:2405\.14333*, 2024\.URL[https://arxiv\.org/abs/2405\.14333](https://arxiv.org/abs/2405.14333)\.
- Yang et al\. \(2025a\)Chenyuan Yang, Xuheng Li, Md Rakib Hossain Misu, Jianan Yao, Weidong Cui, Yeyun Gong, Chris Hawblitzel, Shuvendu Lahiri, Jacob R\. Lorch, Shuai Lu, Fan Yang, Ziqiao Zhou, and Shan Lu\.AutoVerus: Automated Proof Generation for Rust Code\.*Proceedings of the ACM on Programming Languages*, 9\(OOPSLA2\):3454–3482, 2025a\.URL[https://arxiv\.org/abs/2409\.13082](https://arxiv.org/abs/2409.13082)\.
- Yang et al\. \(2025b\)Chenyuan Yang, Natalie Neamtu, Chris Hawblitzel, Jacob R Lorch, and Shan Lu\.VeruSAGE: A Study of Agent\-Based Verification for Rust Systems\.*arXiv preprint arXiv:2512\.18436*, 2025b\.URL[https://arxiv\.org/abs/2512\.18436](https://arxiv.org/abs/2512.18436)\.
- Yang et al\. \(2024\)John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press\.SWE\-agent: Agent\-Computer Interfaces Enable Automated Software Engineering\.In*Proceedings of the 38th Conference on Neural Information Processing Systems*, 2024\.URL[https://arxiv\.org/abs/2405\.15793](https://arxiv.org/abs/2405.15793)\.
- Yang et al\. \(2025c\)John Yang, Carlos E\. Jimenez, Alex L\. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R\. Narasimhan, Diyi Yang, Sida I\. Wang, and Ofir Press\.SWE\-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?In*Proceedings of the 13th International Conference on Learning Representations*, 2025c\.URL[https://arxiv\.org/abs/2410\.03859](https://arxiv.org/abs/2410.03859)\.
- Yang et al\. \(2023\)Kaiyu Yang, Aidan M\. Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar\.LeanDojo: Theorem Proving with Retrieval\-Augmented Language Models\.In*Proceedings of the 37th Conference on Neural Information Processing Systems*, 2023\.URL[https://arxiv\.org/abs/2306\.15626](https://arxiv.org/abs/2306.15626)\.
- Ye et al\. \(2025\)Zhe Ye, Zhengxu Yan, Jingxuan He, Timothe Kasriel, Kaiyu Yang, and Dawn Song\.VERINA: Benchmarking Verifiable Code Generation\.*arXiv preprint arXiv:2505\.23135*, 2025\.URL[https://arxiv\.org/abs/2505\.23135](https://arxiv.org/abs/2505.23135)\.
- Ye et al\. \(2026\)Zhe Ye, Hantao Lou, Yuechun Sun, Peiyang Song, Zhengxu Yan, Timothe Kasriel, Qingyang Zhang, Kaiyu Yang, Soonho Kong, Jingxuan He, and Dawn Song\.Vero: Can AI Agents Build Formally Verified Software Repositories?*arXiv preprint arXiv:2608\.13522*, 2026\.
- Zan et al\. \(2025\)Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang\.Multi\-SWE\-bench: A Multilingual Benchmark for Issue Resolving\.*arXiv preprint arXiv:2504\.02605*, 2025\.URL[https://arxiv\.org/abs/2504\.02605](https://arxiv.org/abs/2504.02605)\.
- Zhang et al\. \(2024a\)Lichen Zhang, Shuai Lu, and Nan Duan\.Selene: Pioneering Automated Proof in Software Verification\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 1776–1789, 2024a\.URL[https://arxiv\.org/abs/2401\.07663](https://arxiv.org/abs/2401.07663)\.
- Zhang et al\. \(2025\)Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang\.SWE\-bench Goes Live\!*arXiv preprint arXiv:2505\.23419*, 2025\.URL[https://arxiv\.org/abs/2505\.23419](https://arxiv.org/abs/2505.23419)\.
- Zhang et al\. \(2024b\)Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury\.AutoCodeRover: Autonomous Program Improvement\.In*Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis*, 2024b\.URL[https://arxiv\.org/abs/2404\.05427](https://arxiv.org/abs/2404.05427)\.
- Zhong et al\. \(2025\)Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini\.ImpossibleBench: Measuring LLMs’ Propensity of Exploiting Test Cases\.*arXiv preprint arXiv:2510\.20270*, 2025\.URL[https://arxiv\.org/abs/2510\.20270](https://arxiv.org/abs/2510.20270)\.
- Zhuo et al\. \(2025\)Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen\-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra\.BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions\.In*Proceedings of the 13th International Conference on Learning Representations*, 2025\.URL[https://arxiv\.org/abs/2406\.15877](https://arxiv.org/abs/2406.15877)\.

## Appendix

### Appendix AThe benchmark corpora

SWE\-Proofis released as two corpora built by the same pipeline from two different sources of issue\-resolution tasks\. The first covers all500500instances ofSWE\-bench Verified; the second covers all266266Python instances ofSWE\-bench Proand exists to test whether the construction methodology transfers to a task distribution it was not designed around\. Every instance is built independently under each of the four specification backends of[appendixD](https://arxiv.org/html/2609.21190#A4), so the unit of release is an*\(instance, backend\)*pair, which we call a*bundle*\.[Table4](https://arxiv.org/html/2609.21190#A1.T4)summarizes the release\.

Table 4:TheSWE\-Proofrelease\. A bundle is one instance under one specification backend\. “Green” counts bundles that pass the complete admission suite of[appendixE](https://arxiv.org/html/2609.21190#A5); theSWE\-bench Verifiedcorpus is green everywhere, and theSWE\-bench Proshortfall is analyzed in[SectionA\.3](https://arxiv.org/html/2609.21190#A1.SS3)\. Churn is added plus removed lines in the gold patch\.Two properties hold by construction across both corpora\. First, the underlying task is passed through untouched: the problem statement, base commit, gold patch, and hidden tests are byte\-identical to the source benchmark, and an integrity check re\-verifies this for every bundle at release time\. Any result onSWE\-Proofis therefore comparable to the corresponding result upstream\. Second, the gold patch of every released instance resolves in the official per\-instance container, which anchors the external correctness oracle to the one the source benchmark already defines\.

#### A\.1TheSWE\-bench Verifiedcorpus

The500500SWE\-bench Verifiedinstances span1212open\-source Python projects, with Django supplying just under half;[appendixJ](https://arxiv.org/html/2609.21190#A10)gives each project’s license\.[Table5](https://arxiv.org/html/2609.21190#A1.T5)gives the per\-project breakdown together with three indicators of how much formal modeling each project demanded: the number of repository functions modeled, the number of instances that required at least one axiom for an unchanged callee, and the number that required modeling more than one function\. The three prover backends are reported separately because they make different modeling choices on the same task, a point we return to in[SectionD\.5](https://arxiv.org/html/2609.21190#A4.SS5)\.

Table 5:Per\-project composition of theSWE\-bench Verifiedcorpus\. “Fns” is the number of repository functions modeled under that backend, summed over the project; “Ax\.” is the number of axioms for unchanged callees\. “≥\\geq1 axiom” and “Multi\-site” count instances that require, respectively, at least one axiom and more than one modeled function under at least one backend\.##### What the tasks look like\.

TheSWE\-bench Verifiedfixes are small and local\. The median gold patch adds four lines and removes two, touches one file, and consists of a single hunk;429429of500500patches are confined to one file, and the largest touches2121\. The upstream difficulty labels place194194instances at under fifteen minutes of developer time,261261between fifteen minutes and an hour,4242between one and four hours, and33above four hours\. The graded test sets are asymmetric: a median of one fail\-to\-pass test decides the instance, against a median of5050pass\-to\-pass tests that guard against regression\. The behavioral change is concentrated in a small number of functions, so a specification can pin it without modeling the whole repository\. The surrounding code still has to be respected; the axioms of[SectionB\.2](https://arxiv.org/html/2609.21190#A2.SS2)do that\.

##### Modeling load\.

UnderNagini, the corpus models704704repository functions, a mean of1\.411\.41and a maximum of77per instance, and assumes264264axioms spread over137137instances\.Velvetmodels611611functions with8080axioms over4444instances, and pureLeanmodels552552functions with only33axioms in total\. The downward trend in axiom count follows from how deeply each backend can model a callee, not from a difference in the tasks\.Leanover Mathlib can usually define an unchanged helper outright, leaving no assumed contract to state, whereasNaginimust summarize anything it cannot express in its verification fragment\. Taking the union over backends,150150instances \(30\.0%30\.0\\%\) need at least one axiom somewhere and158158\(31\.6%31\.6\\%\) are multi\-site somewhere; only a single instance requires an axiom under all three prover backends, and350350need none anywhere\.

##### Representation choices\.

Verification forces a choice of carrier for values that the prover cannot represent natively\. Each bundle records its choices\. UnderNagini,1111instances require reasoning about real\-valued arithmetic and are verified under an interpreted real encoding; the remaining489489need no floating\-point reasoning at all\. Strings are the other recurring decision:227227Naginibundles model string content explicitly, most often as a sequence of integer code points or as an interned identity token, while186186declare that the instance’s behavior does not depend on string content and treat strings opaquely\.Velvet, which sits insideLean, more often carries strings as a list of characters\.[SectionD\.5](https://arxiv.org/html/2609.21190#A4.SS5)works through the cases where this choice changes what a specification can say\.

#### A\.2TheSWE\-bench Procorpus

SWE\-bench Prodraws its tasks from three repositories thatSWE\-bench Verifieddoes not cover, and its patches are an order of magnitude larger: a mean of145\.5145\.5changed lines across3\.43\.4files and9\.89\.8hunks, against14\.314\.3lines in1\.251\.25files forSWE\-bench Verified\.[Table6](https://arxiv.org/html/2609.21190#A1.T6)gives the breakdown\. The three projects also differ in kind from theSWE\-bench Verifiedset\. Ansible is a configuration\-management engine whose behavior is dominated by process orchestration and templating; OpenLibrary is a web application with a large persistence surface; and qutebrowser is a desktop application built around an event loop and a Qt widget hierarchy\. None of the three is the kind of numerically\-oriented library that verification tooling is usually demonstrated on\. The issue text, however, is no longer: the median problem statement is1,2261\{,\}226characters against1,1851\{,\}185onSWE\-bench Verified\. ASWE\-bench Protask therefore asks for ten times the code change on the same amount of stated intent\. The added difficulty is in writing the specification, not in verifying it\.

Table 6:Per\-project composition of theSWE\-bench Procorpus\. Columns are as in[Table5](https://arxiv.org/html/2609.21190#A1.T5); “Churn” is the mean added plus removed lines in the gold patch\.The larger patches do not translate into proportionally larger specifications\. The number of modeled functions per instance is slightly*lower*than onSWE\-bench Verified\(1\.221\.22against1\.411\.41underNagini\), because aSWE\-bench Propatch usually spreads a single behavioral change over many call sites, boilerplate updates, and test\-support edits, only a few of which carry the behavior the graded tests observe\. Identifying that core is the part of construction that gets harder, not the verification itself\. Mean verification time on the greenSWE\-bench Probundles is comparable toSWE\-bench VerifiedunderNagini\(29\.429\.4s against25\.825\.8s\) and lower under bothLean\-based backends \(3\.63\.6s against6\.06\.0s underVelvet,4\.64\.6s against10\.510\.5s underLean\), because the modeled functions themselves are no more complex\.[Figure3](https://arxiv.org/html/2609.21190#A1.F3)puts the two quantities side by side\.

Figure 3:The two corpora on the two measurements that matter for construction\.Left:the size of the reference fix, median and mean, on a logarithmic scale\.Right:the number of functions a specification models per instance\. ASWE\-bench Profix is an order of magnitude larger by added lines and touches three times as many files, and yet its specifications model no more functions than anSWE\-bench Verifiedspecification does\. The gap between the two panels is the work of construction: finding the behavioral core of a large patch, rather than formalizing more of it\.The admission rate is where the difference shows\.996996of10641064SWE\-bench Probundles are green, against20002000of20002000onSWE\-bench Verified, and242242of266266instances are green under all four backends\. The shortfall is highly structured:2222instances fail under every prover backend for the same reason, which we analyze next, and only22more are backend\-specific\.[Figure4](https://arxiv.org/html/2609.21190#A1.F4)shows the pattern\.

Figure 4:Admission on theSWE\-bench Procorpus\.Left:green bundles per backend\.Right:the joint pattern over the three prover backends\. The2222instances that no prover backend admits fail for a shared, diagnosable reason \([SectionA\.3](https://arxiv.org/html/2609.21190#A1.SS3)\) rather than through independent per\-backend attrition\.
#### A\.3Tasks that admit no verified twin

Of the266266SWE\-bench Protasks,2222are not constructed on any prover backend:Nagini,Velvet, andLeanindependently agree that no*verify⇒\\Rightarrowcorrect*twin can be built\. A specification in our setting constrains the*values*a function computes, and each of these2222tasks changes something else: a name, a call shape, an internal representation, or an effect outside the value domain\. In every case the fail\-to\-pass test discriminates on that non\-value property\. A specification that pinned the computed values would be satisfied equally by the pre\-fix and post\-fix code, and the discrimination check of[SectionE\.2](https://arxiv.org/html/2609.21190#A5.SS2)would correctly refuse the instance\.

We list all2222below, grouped by root cause, with the diagnosis recorded for each task during construction\.

##### Renaming and relocation \(1616tasks\)\.

These tasks move or rename code without altering what it computes\. The fail\-to\-pass test therefore discriminates on a name, an import source, or a class identity, none of which a specification written over values can see\.

- •ansible\_\_ansible\-379058e10f3dbc0fdcaf80394bd09b18927e7d33\-v1055803c3a812189a1133297f7f5468579283f86 Swaps the import source for the collections ABCs; the only other changes are a changelog entry and a lint literal\.
- •ansible\_\_ansible\-502270c804c33d3bc963930dc85e0f4ca359674d\-v7eee2454f617569fd6889f2211f75bc02a35f9f8 Method bodies are moved verbatim intoCommandStrategyandFileStrategy\.
- •internetarchive\_\_openlibrary\-25858f9f0c165df25742acf8309ce909773f0cdd\-v13642507b4fc1f8d234172bf8129942da2c2ca26 A pure relocation whose fail\-to\-pass test checks only that anImportErrorno longer occurs\.
- •internetarchive\_\_openlibrary\-308a35d6999427c02b1dbf5211c033ad3b352556\-ve8c8d62a2b60610a3c4631f5f23ed866bada9818 ListandListChangesetare moved byte\-for\-byte; the test assertsisinstanceand registry membership\.
- •internetarchive\_\_openlibrary\-3aeec6afed9198d734b7ee1293f03ca94ff970e1\-v13642507b4fc1f8d234172bf8129942da2c2ca26 Wikidata methods are renamed to private names, with two of them merged by concatenation\.
- •internetarchive\_\_openlibrary\-757fcf46c70530739c150c57b37d6375f155dc97\-ve8c8d62a2b60610a3c4631f5f23ed866bada9818 Relocation only; before the patch the test fails solely because the name is missing\.
- •internetarchive\_\_openlibrary\-798a582540019363d14b2090755cc7b89a350788\-v430f20c722405e462d9ef44dee7d34c41e76fe7a Relocation only; the test asserts which module the class is imported from\.
- •internetarchive\_\_openlibrary\-8a5a63af6e0be406aa6c8c9b6d5f28b2f1b6af5a\-v0f5aece3601a5b4419f7ccec1dbda2071be28ee4 bash\_runandlimit\_serverare byte\-identical after the move\.
- •qutebrowser\_\_qutebrowser\-3d01c201b8aa54dd71d4f801b1dd12feb4c0a08a\-v5fc38aaf22415ab0b70567368332beee7955b367 Renames in the resource\-path module together with a dead\-import cleanup\.
- •qutebrowser\_\_qutebrowser\-3fd8e12949b8feda401930574facf09dd4180bba Six commands are renamed to acmd\-prefix, keeping deprecated\-name aliases\.
- •qutebrowser\_\_qutebrowser\-5fdc83e5da6222fe61163395baaad7ae57fa2cb4\-v363c8a7e5ccdf6968fc7ab84a2053ac78036691d parse\_font\_familiesis wrapped in aFontFamiliestype while the loop body stays identical, confirmed by fuzzing200,000200\{,\}000inputs\.
- •qutebrowser\_\_qutebrowser\-bedc9f7fadf93f83d8dee95feeecb9922b6f063f\-v2ef375ac784985212b1805e1d0431dc8f1b3c171 Relocation only; before the patch the test fails with anAttributeErroron the new export\.
- •qutebrowser\_\_qutebrowser\-de4a1c1a2839b5b49c3d4ce21d39de48d24e2091\-v2ef375ac784985212b1805e1d0431dc8f1b3c171 Relocation only; the test fails before the patch solely onImportError\.
- •qutebrowser\_\_qutebrowser\-ebfe9b7aa0c4ba9d451f993e08955004aaec4345\-v059c6fdc75567943479b23ebca7c07b5e9a7f34c The Qt message handler moves between logging modules, with an added initialization wrapper\.
- •qutebrowser\_\_qutebrowser\-f91ace96223cac8161c16dd061907e138fe85111\-v059c6fdc75567943479b23ebca7c07b5e9a7f34c hide\_qt\_warningandQtWarningFiltermove to the Qt logging utility module\.
- •qutebrowser\_\_qutebrowser\-fd6790fe8c02b144ab2464f1fc8ab3d02ce3c476\-v2ef375ac784985212b1805e1d0431dc8f1b3c171 bufferis renamed totab\_select, and the corresponding completion model totabs\.

##### Function signature changes \(22tasks\)\.

The discriminating difference is the shape of the call, not its result, so the test flips on aTypeErrorrather than on a value\.

- •ansible\_\_ansible\-39bd8b99ec8c6624207bf3556ac7f9626dad9173\-v1055803c3a812189a1133297f7f5468579283f86 A module\-runner entry point drops itsjob\_pathparameter in favor of a monkeypatchable global; the single test flips on aTypeError\.
- •qutebrowser\_\_qutebrowser\-e5340c449f23608803c286da0563b62f58ba25b0\-v059c6fdc75567943479b23ebca7c07b5e9a7f34c The logic genuinely changed, from a boolean return to accept/reject side effects, but the fail\-to\-pass test does not observe the difference\.

##### Internal data structure changes \(22tasks\)\.

Here the pre\- and post\-patch code compute the same values, and the test keys on a representation or call\-route difference instead\.

- •ansible\_\_ansible\-e0c91af45fa9af575d10fd3e724ebc59d2b2d6ac\-v30a923fb5c164d6cd18280c02422f75e611e8fb2 Pre\- and post\-patch code are extensionally identical, measured inside the instance image; the test discriminates only on which bound method is monkeypatched\. Two backends had previously admitted this task and were withdrawn on adjudication, because their pre\-fix models asserted a value footprint the real code never produces\.
- •internetarchive\_\_openlibrary\-3f580a5f244c299d936d73d9e327ba873b6401d9\-v0f5aece3601a5b4419f7ccec1dbda2071be28ee4 Alistbecomes atupleand adicta read\-only mapping proxy, with contents identical\.

##### External side effect changes \(22tasks\)\.

The change is observable only outside the value domain, in process, filesystem, or ordering state that the verifier does not model\.

- •ansible\_\_ansible\-8127abbc298cabf04aaa89a478fc5e5e3432a6fc\-v30a923fb5c164d6cd18280c02422f75e611e8fb2 The graded tests pass merely because the task executor and the worker process stopped accepting a standard\-input handle\. The backends disagreed on the cause here, and it was adjudicated as an arity\-only change\.
- •qutebrowser\_\_qutebrowser\-e57b6e0eeeb656eb2c84d6547d5a0a7333ecee85\-v2ef375ac784985212b1805e1d0431dc8f1b3c171 The same file objects and completion count are delivered in the same order before and after the patch\.

##### What the list shows\.

None of the2222is a verifier limitation\. In each case the task’s own fail\-to\-pass test is insensitive to the values the code computes, so the gold patch and its predecessor are behaviorally indistinguishable in the domain a functional specification talks about\. The bound on transfer is therefore a property of the task, not of the specification language: our methodology applies where a known gold patch exists and the requested change is observable as a change in computed values\. The remaining22non\-green bundles are per\-backend refusals on instances that other backends admit, so those instances stay usable under the backends that do admit them\.

#### A\.4Instances with a relaxed disclosure disposition

The specification view shown to a solver must reveal nothing about the fix beyond what the public problem statement already discloses\. For a minority of instances this requirement collides with faithfulness\. The behavior the hidden tests actually check appears only in developer discussion or in the gold change, never in the solver\-visible problem statement\. A faithful specification must then name a discriminator absent from the public text, while a strictly non\-leaking specification, judged against that text alone, cannot pin the tested behavior\. The two properties are simultaneously unachievable because the public task text and the gold fix are about different facets of the same symptom\.

For these instances we keep the bundle, permit the discriminator in the specification view, and document the relaxation per instance\. An instance qualifies only when an adversarial analysis establishes that the leak is unavoidable, that is, that no reformulation derivable from the public problem statement could pin the tested behavior\. Across theSWE\-bench Verifiedcorpus the disposition applies to1717Naginibundles,3737Velvetbundles,1111Leanbundles, and noEarsbundle; acrossSWE\-bench Proit applies to11,11,22, and77respectively, eleven bundles in total\. The counts differ by backend: the identity of the unavoidable discriminator depends on how the behavior is encoded, and a phrase that is unavoidable in one language may be avoidable in another\.

The verification core, namely verification, discrimination, soundness, completeness, and axiom soundness, and the harness resolution of the gold patch all hold unchanged\. Only the secondary non\-disclosure property is relaxed, for a documented minority\. The bundle records the relaxation, so a downstream user can exclude those instances\.

scikit\-learn\-26194is representative\. The issue concerns the leading threshold returned by a receiver\-operating\-characteristic routine\. The gold fix prepends positive infinity and the hidden test asserts exactly that, whereas the public issue proposes the*opposite*remedy of clipping thresholds to at most one\. The discriminator “infinity” appears nowhere in the public text, so any faithful specification must introduce it, and any specification that avoids it is unfaithful to the graded behavior\.[SectionH\.3](https://arxiv.org/html/2609.21190#A8.SS3)shows the specification view for this instance alongside the public text\.

### Appendix BAnatomy of a released instance

Every bundle is self\-contained\. It carries the specification, both implementations that the specification is meant to separate, the repository diff that resolves the task, the two documents that bind the formal model to the real code, the executable evidence that substantiates each admission check, and the record of what was checked and how\. A reader who downloads a bundle can re\-run the verifier, re\-run the probes, and re\-derive every stamp\.

[Table7](https://arxiv.org/html/2609.21190#A2.T7)lists the artifact classes and their typical size under each backend\. TheSWE\-bench Verifiedcorpus ships3\.543\.54million lines of specification, proof, probe, and audit text in total, of which the specifications themselves are the smallest part; most of the volume is refutation machinery and recorded evidence\.

Table 7:Artifacts released with each bundle, and their median size in lines under each backend\. The first three rows are the objects the guarantee is stated over; the middle two bind the formal model to the repository; the last three are the evidence and the record\. Where an artifact is not carried by all500500bundles, a parenthesized count gives how many carry it; a dash means the backend does not use it\.Earsships a requirements document and an audit record only, since it has no verifier to drive\.ArtifactRoleNaginiVelvetLeanEarsSpecification moduleThe verified object: signatures, contracts, axioms, pure helpers, and a reference implementation with its proof annotations151100107106Pre\-fix twinThe same specification paired with the reported buggy behavior; required to fail verification153 \(497\)103 \(481\)97 \(479\)–Equivalent patchThe repository diff that applies at the base commit and passes the hidden testsshared across backendsCorrespondence mapThe operation\-by\-operation refinement from the reference implementation to the patch1028759–Provenance mapThe binding from each modeled symbol, type, and axiom to its repository counterpart958982–Property test suiteCandidate implementations with the verifier outcome each must produce529716742–Axiom probesExecutable checks that run each axiom against the real callee inside the instance container175 \(477\)143 \(262\)24 \(496\)–Differential harnessThe reference implementation run against the patched repository code on generated inputs300157 \(499\)144–Admission recordEvery check, its outcome, the verifier configuration, and the adjudication that closed it733603543366#### B\.1The central pair and the executed patch

The guarantee is a statement about two objects\. The*specification module*is verified; the*equivalent patch*is executed\. Neither alone is sufficient, so the bundle also carries the documents that connect them\.

##### The specification module\.

The module declares the signatures of the functions the fix touches, states their preconditions and postconditions, axiomatizes the callees the fix leaves alone, defines whatever pure helpers the postconditions need, and closes with a reference implementation carrying the annotations the verifier requires\. It is one file, and it is small: a median of151151lines underNagini,100100underVelvet,107107underLean, and106106underEars\.

The proof burden inside that file is not small\. ANaginimodule states a mean of6\.86\.8preconditions and12\.912\.9postconditions;264264of the500500modules need at least one loop invariant, at a mean of10\.710\.7where they appear, and443443state an explicit termination measure\. Roughly half the non\-blank content is specification rather than implementation: a mean of28\.828\.8specification lines against21\.121\.1lines of executable body\. TheLeanmodules declare a mean of1\.871\.87theorems and4\.54\.5supporting definitions each, spend a median of1313lines inside the proof, and no proof invokessorryanywhere in the corpus; across all500500modules there are exactly77local axiom declarations\.Velvetmodules declare a mean of1\.231\.23methods with2\.322\.32postconditions and hand each correctness obligation to the solver, so their proof regions are the shortest of the three \(median44lines\) while their pure definitions are the longest \(median3232\)\.

##### What the solver sees\.

The module as released contains the reference implementation, which would give the answer away\. Every evaluation setting that shows a specification to a solver shows a*view*of the module instead: the signatures, the contracts, the axioms, and the pure helpers, with the reference body removed\. The view is generated from the verified module\.[SectionF\.2](https://arxiv.org/html/2609.21190#A6.SS2)describes what else is withheld\.

##### The pre\-fix twin\.

Each bundle also ships the same specification paired with the implementation the issue complains about\. This artifact exists to*fail*\. It is the witness that the specification is strong enough to see the reported defect, and its failure is the discrimination check of[SectionE\.2](https://arxiv.org/html/2609.21190#A5.SS2)\. Where a fix repairs several independent behaviors, the bundle ships one twin per behavior:419419Naginibundles carry a single twin,6565carry two,1313carry three, and33carry four, for600600twins in total;Velvetships569569andLean540540\. A specification whose twin verifies is not admitted\.

##### The equivalent patch\.

The patch is the object that must actually resolve the task, and it is shared by all four backends of an instance because it is a property of the repository, not of the specification language\. It applies at the frozen base commit and passes the hidden tests in the task’s official container\. OnSWE\-bench Verifiedthe patches are small: a median of one file and one hunk\. The correspondence below is therefore tractable to audit by hand\.

##### The correspondence and provenance maps\.

The transfer from the verified object to the executed one rests on these two documents\.

The*correspondence map*walks the reference implementation against the patch one operation at a time and labels each step with the behavior\-preserving relationship that justifies it: an identical operation, a type refinement from the modeled carrier to the concrete repository type, or a call to an axiomatized callee\. A step that fits none of the three is a defect, and the equivalence check of[SectionE\.3](https://arxiv.org/html/2609.21190#A5.SS3)is an adversarial reading of exactly this document\. The maps run to a median of102102lines underNagini, longer than the specification modules they annotate\.

The*provenance map*records where the model came from\. For each modeled symbol it names the repository entity it abstracts: which attribute an integer field stands for, which class a modeled record corresponds to, which method a modeled operation implements\. For each axiom it names the exact callee and states the evidence for the contract asserted of it\. For each representation decision it states the decision and its justification: that strings are carried as sequences of code points because the tested behavior inspects characters, or that they are treated opaquely because it does not\. OnSWE\-bench Verified,227227Naginibundles model string content explicitly and186186declare that the instance’s behavior does not depend on it;1111carry an interpreted encoding for real arithmetic\.[SectionD\.5](https://arxiv.org/html/2609.21190#A4.SS5)works through the cases where the choice changes what a specification can express\.

#### B\.2Axioms for code the fix does not change

A fix to one function usually sits in the middle of code that the fix does not touch\. Verifying it end to end would mean formalizing that surrounding code, and for a real repository that is neither feasible nor useful: the issue is not about the database driver, the template engine, or the array library\. Instead, each unchanged callee that the specification depends on is given an*axiom*, which states the weakest property of that callee the proof actually needs, and nothing more\.

Box B\.1: An axiom for an unchanged callee, fromdjango\-11179\(abridged\)[⬇](data:text/plain;base64,QENvbnRyYWN0T25seQpkZWYgcmVtb3ZlX3Jvd3MoZGI6IERhdGFiYXNlLCBpZGVudGlmaWVyOiBpbnQpIC0+IGludDoKICAgICIiIlRoZSB1bmNoYW5nZWQgcm93LXJlbW92YWwgcHJpbWl0aXZlLiBJdCBpc3N1ZXMgZXhhY3RseSBvbmUKICAgIHJvdy1yZW1vdmFsIHN0YXRlbWVudCBmb3IgYGlkZW50aWZpZXJgLCByZXBvcnRzIHRoZSBudW1iZXIgb2YKICAgIHJvd3MgdGhhdCBzdGF0ZW1lbnQgcmVtb3ZlZCwgYW5kIGRvZXMgbm90IHRvdWNoIHRoZSBpbi1tZW1vcnkKICAgIG9iamVjdC4gVGhlIHN0b3JlIG1heSBpbnN0ZWFkIHJlZnVzZSB0aGUgc3RhdGVtZW50LCBhbmQgd2hpY2gKICAgIGFuc3dlciBhIGNhbGxlciBnZXRzIGlzIHRoZSBzdG9yZSdzIHRvIGRlY2lkZS4iIiIKICAgIFJlcXVpcmVzKE11c3RUZXJtaW5hdGUoMSkpCiAgICBSZXF1aXJlcygtKDIgKiogNjMpIDw9IGlkZW50aWZpZXIgYW5kIGlkZW50aWZpZXIgPCAyICoqIDYzKQogICAgUmVxdWlyZXMoUmVtb3ZhbEJ1ZGdldChMRURHRVIpKQogICAgRW5zdXJlcyhDYXJyaWVkT3V0UmVtb3ZhbChkYiwgaWRlbnRpZmllciwgUmVzdWx0KCkpKQogICAgRW5zdXJlcyhSZXN1bHQoKSA+PSAwKQogICAgRXhzdXJlcyhSZW1vdmFsUmVmdXNlZCwgUmVmdXNlZFJlbW92YWwoZGIsIGlkZW50aWZpZXIpKQ==)@ContractOnlydefremove\_rows\(db:Database,identifier:int\)\-\>int:”””Theunchangedrow\-removalprimitive\.Itissuesexactlyonerow\-removalstatementfor‘identifier‘,reportsthenumberofrowsthatstatementremoved,anddoesnottouchthein\-memoryobject\.Thestoremayinsteadrefusethestatement,andwhichansweracallergetsisthestore’stodecide\.”””Requires\(MustTerminate\(1\)\)Requires\(\-\(2\*\*63\)<=identifierandidentifier<2\*\*63\)Requires\(RemovalBudget\(LEDGER\)\)Ensures\(CarriedOutRemoval\(db,identifier,Result\(\)\)\)Ensures\(Result\(\)\>=0\)Exsures\(RemovalRefused,RefusedRemoval\(db,identifier\)\)

Box[B\.2](https://arxiv.org/html/2609.21190#A2.SS2)is representative\. The fix in that instance clears an in\-memory identifier after a single\-object fast delete; the row\-removal primitive it calls is untouched, and the axiom asserts only what the proof needs from it\. The axiom deliberately leaves several things open\. It does not say when the store refuses a removal, so no implementation can arrange the refusing outcome or predict it; it does not say what count comes back beyond non\-negativity; and it says nothing at all about the store’s own tallies\. An axiom that fixed the refusal condition would let an implementation branch on it and still verify, and an axiom that exposed the tallies would license implementations whose real form pays a database round trip per read\. The justification for each assertion, and the identity of the primitive the axiom stands for, are recorded in the provenance map; the executable probe of this axiom calls Django’s real deletion primitive\.

##### Axioms are a last resort, not a default\.

An axiom is an assumption, so a bundle that can avoid one does\. The preferred alternative is to model the callee outright, and the corpus does this far more often than it axiomatizes:362362Naginibundles model at least one callee deeply,500500such models in total, against264264axioms over137137instances\. The balance shifts sharply with the expressiveness of the language\.Velvetneeds8080axioms across4444instances, and pureLeanover Mathlib needs33in the entire500500\-instance corpus\. Taking the union over the three prover backends,150150SWE\-bench Verifiedinstances \(30\.0%30\.0\\%\) require at least one axiom somewhere,350350require none anywhere, and exactly one requires an axiom under all three\. OnSWE\-bench Prothe pattern repeats at lower intensity:5757axioms underNagini,99underVelvet,33underLean\.

##### Axioms are tested against the running repository\.

Each bundle ships an executable probe that constructs inputs, calls the*real*callee inside the task’s official container, and checks the observed behavior against what the axiom asserts\. The probes are substantial programs, at a median of175175lines underNagini\. Across both corpora they contribute25\.725\.7million recorded observations of real callee behavior\. The probes cover477477Naginibundles,262262Velvetbundles, and496496Leanbundles onSWE\-bench Verified; a bundle without a probe is a bundle with no axiom to probe\. No probe refutes its axiom on any admitted bundle\.[SectionE\.2](https://arxiv.org/html/2609.21190#A5.SS2)reports the check itself\.

##### What an axiom cannot be\.

An axiom must never contain the fix\. An axiom that asserted the corrected behavior of a callee would make the specification verify for the wrong reason and the pre\-fix twin fail for the wrong reason\. The audit of[SectionE\.3](https://arxiv.org/html/2609.21190#A5.SS3)attacks axioms specifically, and an axiom over a callee that the patch*does*touch is rejected outright\. In the released corpora every axiomatized callee lies outside the patch\.

#### B\.3Attached evidence and the admission record

A specification must admit legitimate implementations, accept correct ones, and reject wrong ones\. Those are claims about infinitely many programs, and the bundle discharges them with finite, re\-runnable evidence: a property test suite of candidate implementations with the verdict each must receive, a differential harness that compares the reference implementation against the real repository code, and the probes of[SectionB\.2](https://arxiv.org/html/2609.21190#A2.SS2)\.

##### The property test suite\.

Each bundle ships a set of candidate implementations of its modeled functions, each labeled with the verdict the verifier must return\. The labels fall into three lanes\.*Admissibility*cases are implementations shaped like the ones the task’s own tests exercise; if the verifier rejects them, the specification’s preconditions are too strong to be about the real task\.*Soundness*cases are correct implementations written differently from the reference \(statements reordered, a helper inlined, a different algorithm for the same function\); if the verifier rejects them, the specification has over\-fitted to one way of writing the answer\.*Discrimination*cases are wrong implementations, including the original bug and deliberate near misses; if the verifier accepts them, the specification is too weak to be worth having\.

Table 8:The released property test suites\. Each case is a candidate implementation carried in the bundle together with the verdict the verifier must return\. “Must verify” pools the admissibility and soundness lanes; “must be rejected” is the discrimination lane\. Every case in both corpora produces its required verdict\.LaneOutcomeBackendCasesAdmis\.Sound\.Discrim\.Must verifyMust rejectSWE\-bench Verified,*500 instances*Nagini5,0305339663,5311,494 / 1,4943,536 / 3,536Velvet5,2085261,2683,4141,794 / 1,7943,414 / 3,414Lean5,8805131,2494,1181,762 / 1,7624,118 / 4,118SWE\-bench Pro,*266 instances*Nagini2,2312665201,445786 / 7861,445 / 1,445Velvet2,5092637501,4961,013 / 1,0131,496 / 1,496Lean2,6242677331,6241,000 / 1,0001,624 / 1,624Total23,4822,3685,48615,62823,482 / 23,482[Table8](https://arxiv.org/html/2609.21190#A2.T8)reports the property test suites\. TheSWE\-bench Verifiedcorpus carries16,11816\{,\}118cases andSWE\-bench Proa further7,3647\{,\}364, a median of99to1111per bundle, and every case produces the verdict it is labeled with\. The single largest property test suite holds6262cases, for an instance whose specification had to be defended against an unusually large family of near misses\.

Two features of the property test suite make the discrimination lane meaningful\. First, a rejection is credited only when the verifier’s complaint is about the intended property\. Under the twoLean\-based backends a discrimination case is accompanied by a witness: a concrete input on which the candidate and the specification disagree\. The case counts as refuted only when that witness is confirmed against the candidate and cleared against the frozen reference\. The property test suites are therefore expensive: driving the16,11816\{,\}118SWE\-bench Verifiedcases takes32,66232\{,\}662verifier invocations and roughly7272hours of prover time\. Second, the property test suites are shipped as source, so anyone can add a case\. A specification that survives our23,48223\{,\}482candidates has survived a stated attack, and a reader can extend that attack\.

##### The differential harness\.

The property test suite checks the specification against candidate implementations\. The differential harness tests the*reference implementation*against the real thing\. It generates inputs, runs the patched repository code on them inside the task’s container, runs the modeled reference on the corresponding modeled inputs, and compares\. It catches a reference implementation that is faithful to the specification but not to the repository: a model that is internally consistent and describes a program nobody has\. Across theSWE\-bench Verifiedcorpus the harness compares129\.8129\.8million inputs, and acrossSWE\-bench Proa further57\.957\.9million\. There are no disagreements in either corpus\.

##### The admission record\.

Each bundle closes with a record of what was checked\. It lists every admission check of[appendixE](https://arxiv.org/html/2609.21190#A5)with its outcome, the reason for any check that does not apply, the verifier configuration used, the representation decisions of[SectionB\.1](https://arxiv.org/html/2609.21190#A2.SS1), and the adjudications that closed the checks requiring judgment\. It also carries, verbatim, the attacks that an audit attempted and the reason each failed\. The records are long, at a median of733733lines underNagini\. They let the corpus be re\-audited rather than re\-run: an audit that reconstructs a previous attack and reproduces its outcome costs nothing\.

A check that does not apply is recorded as inapplicable rather than as a pass, and the distinction matters when reading the per\-backend tables of[SectionE\.2](https://arxiv.org/html/2609.21190#A5.SS2)\. An axiom probe is inapplicable to a bundle with no axioms; a mutation check is inapplicable to a reference implementation with no operators to mutate, such as a body that only performs an assignment\. In those cases the burden falls on the checks that do apply: for an assignment\-only body, on the pre\-fix twin, which must still fail\. No bundle is admitted on inapplicability alone\.

### Appendix CRelated work

##### Issue\-resolution benchmarks and their oracles\.

SWE\-bench\([Jimenez et al\., 2024](https://arxiv.org/html/2609.21190#bib.bib22)\)established the real\-world issue\-resolution task with a set of hidden\-test oracles;SWE\-bench Verified\([OpenAI, 2024](https://arxiv.org/html/2609.21190#bib.bib38)\)refined it into a human\-validated 500\-instance subset\. Extensions address visual domains\([Yang et al\., 2025c](https://arxiv.org/html/2609.21190#bib.bib55)\), additional languages\([Zan et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib59)\), agent training\([Pan et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib40)\), and contamination\-resistant task streams\([Zhang et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib61);[Badertdinov et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib7)\)\. A parallel line scrutinizes the oracle:[Aleithan et al\. \(2024\)](https://arxiv.org/html/2609.21190#bib.bib5)document solution leakage and weak tests, while[Liang et al\. \(2025\)](https://arxiv.org/html/2609.21190#bib.bib29)attribute part of reported performance to memorization\. The community response has been to strengthen or refresh the tests\([Ahmed et al\., 2024](https://arxiv.org/html/2609.21190#bib.bib3);[Ahmed et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib4);[Jain et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib20);[Zhuo et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib64)\)\.SWE\-Proofreplaces tests with a formal specification, so that a verified fix is provably correct on its entire input domain\.

##### Verified code generation and verification benchmarks\.

Clover\([Sun et al\., 2024](https://arxiv.org/html/2609.21190#bib.bib44)\)accepts generations only when a verifier confirms mutual consistency among code, documentation, and a Dafny specification\.Verina\([Ye et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib57)\)jointly benchmarks code, specification, and proof generation in Lean, separating soundness from completeness as we do\.DafnyBench\([Loughridge et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib30)\)and multi\-language vericoding suites\([Bursuc et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib8)\)measure whether models can produce verifying artifacts\. A complementary thread develops proof automation: whole\-proof generation and repair\([First et al\., 2023](https://arxiv.org/html/2609.21190#bib.bib18)\), sound LLM\-in\-the\-loop frameworks\([Wu et al\., 2024](https://arxiv.org/html/2609.21190#bib.bib48)\), retrieval\-augmented proving\([Yang et al\., 2023](https://arxiv.org/html/2609.21190#bib.bib56)\), self\-improving verified translation\([Aggarwal et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib2)\), and agentic proof generation\([Yang et al\., 2025a](https://arxiv.org/html/2609.21190#bib.bib52);[Chen et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib10);[Mugnier et al\., 2025](https://arxiv.org/html/2609.21190#bib.bib36);[Misu et al\., 2024](https://arxiv.org/html/2609.21190#bib.bib35)\)\. AutoRocq\([Tu et al\., 2026](https://arxiv.org/html/2609.21190#bib.bib46)\)iteratively refines proofs via the Rocq theorem prover; Inductive Deductive Synthesis\([Agarwal et al\., 2026](https://arxiv.org/html/2609.21190#bib.bib1)\)jointly synthesizes implementation and proof for distributed systems200×200\\timesfaster than expert effort\. Project\-level efforts target real verified codebases such as an OS microkernel\([Zhang et al\., 2024a](https://arxiv.org/html/2609.21190#bib.bib60)\), while[Yang et al\. \(2025b\)](https://arxiv.org/html/2609.21190#bib.bib53)study 849 real proof tasks from open\-source Verus/Rust systems and[Sosso et al\. \(2026\)](https://arxiv.org/html/2609.21190#bib.bib43)show that agentic provers nearly saturate existing Lean 4 benchmarks and motivate harder suites like ours\. Our verifier\-feedback loop instantiates iterative repair from execution feedback\([Chen et al\., 2024](https://arxiv.org/html/2609.21190#bib.bib11);[Madaan et al\., 2023](https://arxiv.org/html/2609.21190#bib.bib33)\), with a sound verifier supplying the signal\.[Shefer et al\. \(2025\)](https://arxiv.org/html/2609.21190#bib.bib42)evaluate LLMs producing verified code in Dafny, Verus, and Nagini on HumanEval\-derived tasks;[Feng et al\. \(2026\)](https://arxiv.org/html/2609.21190#bib.bib17)validate generated specifications before synthesizing certified code against a Lean\-embedded verifier\. All above research works target tasks that synthesize programs from curated descriptions\. Closest to our setting,Vero\([Ye et al\., 2026](https://arxiv.org/html/2609.21190#bib.bib58)\)evaluates implementation and proof synthesis at the repository level inLean4, over4343instances sourced from real Dafny, Verus, Coq, and Python projects\. Each instance is a hand\-writtenLeanre\-formalization of the upstream code rather than the repository itself, and the agent fills a fixed scaffold instead of resolving a reported issue\. Its specifications are supplied and frozen, so specification synthesis is out of scope, and its audit certifies that they are satisfiable rather than that they capture the intended behavior\.SWE\-Proofdiffers from these works as it is built on fixes to real GitHub issues in code repositories with rich library dependencies\.

##### Specification synthesis from informal intent\.

Whether a specification faithfully captures intent is a recognized hard problem\.[Endres et al\. \(2024\)](https://arxiv.org/html/2609.21190#bib.bib15)measure postconditions’*discriminative power*to reject buggy behavior;[Lahiri \(2024\)](https://arxiv.org/html/2609.21190#bib.bib25)proposes symbolically testing specifications for quality;[Richter & Wehrheim \(2025\)](https://arxiv.org/html/2609.21190#bib.bib41)evaluate contracts by their ability to separate buggy from correct implementations; and[Ma et al\. \(2025\)](https://arxiv.org/html/2609.21190#bib.bib31)generate verifiable specifications with conversational refinement\. SpecSyn\([Ma et al\., 2026](https://arxiv.org/html/2609.21190#bib.bib32)\)strengthens specifications via mutation\-based semantic discrimination, while[Chen et al\. \(2026\)](https://arxiv.org/html/2609.21190#bib.bib12)find that the best frontier LLM attains only a pass rate of 20\.2% on repository\-scale specification generation, corroborating our finding that specification synthesis is the binding bottleneck\. The challenge echoes classical results: dynamically\-inferred invariants are unsound\([Ernst et al\., 2007](https://arxiv.org/html/2609.21190#bib.bib16)\), learned loop invariants are often non\-inductive\([Chakraborty et al\., 2023](https://arxiv.org/html/2609.21190#bib.bib9);[Kamath et al\., 2023](https://arxiv.org/html/2609.21190#bib.bib23)\), and specifications can be satisfied vacuously\([Kupferman & Vardi, 2003](https://arxiv.org/html/2609.21190#bib.bib24)\)\. The problem parallels autoformalization, where faithfulness is enforced through round\-trip checks\([Wu et al\., 2022](https://arxiv.org/html/2609.21190#bib.bib49);[Jiang et al\., 2023](https://arxiv.org/html/2609.21190#bib.bib21);[Li et al\., 2024](https://arxiv.org/html/2609.21190#bib.bib28);[Xin et al\., 2024](https://arxiv.org/html/2609.21190#bib.bib51)\)\. These works study faithfulness at the level of a single function or postcondition; the real\-world setting adds a distinctly harder demand, namely that a specification cover the full behavioral surface of a fix spanning many functions and files\.SWE\-Proofmakes this coverage requirement an explicit correctness criterion and contributes the first measurement of the resulting faithfulness gap on real\-world software issues, showing it to be the dominant obstacle to end\-to\-end verified issue resolution\.

### Appendix DSpecification languages and verifiers

Every instance is built four times, in four specification languages with four different notions of what it means to be checked\. Three are machine\-checkable and one is not\. The fourth separates two effects that are easy to conflate: the benefit of giving a solver a precise statement of the intended behavior, and the benefit of that statement being mechanically enforceable\. Without a prose backend, every result on the formal backends is confounded: a specification is also a very good problem description\.

[Table9](https://arxiv.org/html/2609.21190#A4.T9)states what each backend checks and what it costs\.

Table 9:The four specification backends\. “Trust base” is what must be correct for a passing check to mean anything\. Verification time is the wall clock of a single successful check of a released bundle on theSWE\-bench Verifiedcorpus\. Axioms are the total number of assumed contracts for unchanged callees across all500500instances\.NaginiVelvetLeanEarsNotationPython with contract annotationsImperative language embedded in Lean 4Lean 4 with MathlibStructured EnglishStatement formPre/postconditions, permissions, invariantsMethod contracts with mutable parametersA specification predicate and a theoremRequirements with acceptance criteriaDischarged byTranslation to an intermediate verification language, then SMTAn SMT\-backed tactic inside LeanThe Lean kernel–Trust baseTranslation, verification back end, SMT solverEmbedding semantics, tactic, SMT solverLean kernel and its three classical axiomsHuman readingMedian length \(lines\)151100107106Verify, mean / median \(s\)25\.8 / 18\.46\.0 / 5\.210\.5 / 9\.1–Verify, p95 / max \(s\)58\.2 / 288\.611\.9 / 33\.717\.7 / 84\.9–Axioms,500500instances264803–TerminationExplicit measures \(443443bundles\)Total\-correctness semanticsStructural, by construction–#### D\.1Contract\-annotated Python

The first backend states specifications in Python itself, annotated with contracts, and verifies them by translation to an intermediate verification language and from there to an SMT solver\. A specification declares the functions the fix touches with their real signatures, states preconditions and postconditions over their arguments and results, and supplies a reference body that the verifier checks against those contracts\.

Two features of this backend shape how the corpus looks\. The first is that it reasons about mutable heap state with explicit permissions: a specification must say which fields it may read and write, and a postcondition about a field is meaningless without the accompanying permission\. The backend can therefore describe the kind of behaviorSWE\-bench Verifiedissues are actually about: in\-place mutation of an object, aliasing between two references, a method that changes one field and must leave another alone\. The cost is considerable annotation\. The second is that termination is not free\. A specification states an explicit measure that must decrease, and443443of the500500released bundles do so, at a mean of2\.82\.8measures per bundle\. Without it the verifier will happily prove a postcondition of a function that never returns\.

The consequence is that this is the most expensive backend to write in and the most expressive about state\. Its modules are the longest \(151151lines at the median\), carry the most contracts \(a mean of6\.86\.8preconditions and12\.912\.9postconditions\), need the most loop invariants \(264264bundles, a mean of10\.710\.7where present\), and take the longest to check \(25\.825\.8s on average, up to288\.6288\.6s\)\. It is also the backend that needs the most axioms,264264over137137instances, because anything it cannot express in its verification fragment must be summarized rather than modeled\.

##### Exceptional behavior\.

Many repository fixes are about what happens when something fails, so the specification language must be able to state it\. This backend distinguishes the normal postcondition from the postcondition that holds when a particular exception propagates, and the corpus uses that distinction: a specification can require that a failing operation leaves the object exactly as it found it, which is a guarantee about the absence of a partial effect\. Box[B\.2](https://arxiv.org/html/2609.21190#A2.SS2)shows the pattern in an axiom, where the store’s refusal to remove a row is an outcome the specification allows but no implementation may arrange\.

#### D\.2An imperative language inside a proof assistant

The second backend writes the fix in a small imperative language that is embedded in Lean 4\. A specification declares a method with typed parameters, marks the ones it mutates, states preconditions, postconditions, and loop invariants, and closes with a directive that asks the tooling to prove the method correct against its contract\. The proof obligations are generated from the embedding’s semantics and discharged by an SMT\-backed tactic, so a specification in this language is a Lean object whose correctness claim is discharged automatically rather than by hand\.

The result is the lightest of the three formal backends to write in and the cheapest to check: a median of100100lines, a mean of1\.231\.23methods with2\.322\.32postconditions, and a mean verification time of6\.06\.0s\. Its proof regions are the shortest of the three \(a median of44lines\) because the tactic does the work\. Its pure definitions are the longest \(a median of3232lines\), since the interesting content moves into the functional model that the contracts are stated over\.

The characteristic modeling move in this backend is to*parameterize*rather than to axiomatize\. Where the contract\-annotated Python backend states an assumed contract for an unchanged callee, this backend often takes the callee’s result as an input to the method and states the specification relative to it\. Indjango\-11179, for example, the number of rows removed by the unchanged deletion primitive is simply a parameter, and the specification says that the identifier is cleared and the reported count equals that parameter\. That is a weaker statement: it does not pin what the primitive itself does\. This backend therefore needs8080axioms where the first needs264264\.[SectionD\.5](https://arxiv.org/html/2609.21190#A4.SS5)discusses what this trade costs\.

#### D\.3Dependent type theory over a mathematical library

The third backend states the specification in Lean 4 with Mathlib, as a predicate over inputs and outputs, gives a functional implementation, and proves a theorem that the implementation satisfies the predicate for all inputs\. There is no verification condition generator and no solver: the proof is a term the Lean kernel checks\.

This is the strictest trust base of the three, and the corpus is measured against it directly\. Across the released bundles no proof anywhere contains asorry, the entire500500\-instance corpus declares77local axioms, and an audit of what the theorems actually depend on finds only Lean’s three standard classical axioms \(propositional extensionality, cited by659659theorems; soundness of quotients,501501; and choice,127127\) plus eleven instance\-specific declarations that are themselves recorded as axioms in the bundles that use them\. A theorem in this backend does not appeal to anything we supplied\.

The characteristic modeling move here is*universal quantification over the unchanged*\. Where the other backends assume a contract for a callee they do not model, this backend can leave the callee as a variable and prove the theorem for every possible instantiation of it\. Indjango\-11179the decision of whether an object qualifies for the fast deletion path is an unchanged predicate\. The specification quantifies over all predicates and proves the outcome for each\. The backend therefore needs33axioms in500500instances: a callee that can be abstracted does not need to be assumed\.

The cost of that strictness appears elsewhere\. The backend has no native account of mutable heap state, so a fix about in\-place mutation must be recast as a function from a value to a value, and the recasting is the hard part of writing these bundles\. It is also the backend whose modeled site most often differs from the patch’s own file set, because the recasting frequently pulls in the caller that owns the data being transformed\.

#### D\.4Structured natural language

The fourth backend states the intended behavior in structured English, using a constrained requirements syntax in which each acceptance criterion follows one of a small number of sentence patterns: a trigger and a required response \(when…then…shall\), an unconditional obligation \(shall\), an undesired condition and its handling \(if…then\), or a continuous state \(while\)\. Each requirement is introduced by the developer\-facing need it serves and then decomposed into numbered criteria\.

The released documents are as long as the formal ones and considerably more readable: a median of106106lines, a mean of3\.73\.7requirements and14\.714\.7acceptance criteria, with17\.817\.8obligations stated per instance\. Of the7,3627\{,\}362criteria in theSWE\-bench Verifiedcorpus,57\.3%57\.3\\%are trigger\-and\-response,28\.4%28\.4\\%are unconditional,11\.8%11\.8\\%describe an undesired condition,1\.9%1\.9\\%describe a continuous state, and0\.6%0\.6\\%use no pattern keyword at all\.

These documents are held to the same standards of faithfulness, discrimination, and non\-disclosure as the formal bundles, and are audited by the same procedure\. They cannot be held to verification, because there is nothing to run\. A solver given one of these documents receives a complete, precise, non\-leaking account of the intended behavior and no way to check its own work, which is the condition against which the formal backends’ verification feedback is measured\. The requirements are also noticeably*broader*than their formal counterparts\. Because prose costs nothing to quantify over, these documents routinely state regression obligations that a formal specification would have to model in order to mention: that signals still fire, and that unrelated deletion behaviors are preserved\.

#### D\.5What each language can and cannot say

The four backends are not four encodings of one specification\. They are four specifications, and on the same task they pin different things\.

##### The same fix, four times\.

django\-11179asks that a single dependency\-free object being fast\-deleted have its in\-memory primary key cleared, as the general deletion path already does\. It is a one\-line patch\. The four specifications of it differ as follows\.

The contract\-annotated Python bundle models the removal as an operation against a database handle with an explicit entitlement to issue exactly one row\-removal statement\. Its postconditions pin five separate facts: that one statement was issued, that it targeted the identifier the object carried on entry, that the returned count is the one*that statement*reported rather than a number obtained by asking the store again, that the in\-memory identifier is cleared, and that on a refusal the object is left exactly as it was found\. Nothing about the store’s own tallies is readable by the implementation, which rules out implementations that pay a database round trip per read\.

TheVelvetbundle states the same fix in three lines of contract: the identifier is cleared, and the reported count equals the count it was given\. The unchanged removal is a parameter\.

TheLeanbundle models the collector’s data as groups of records, defines the outcome as “every record’s key cleared” plus a branch\-selected tally, and proves that its implementation meets that specification for every possible fast\-deletability predicate\.

TheEarsbundle states three requirements and thirteen acceptance criteria, including two that no formal bundle states: that the reset must use the model’s actual primary\-key attribute name so that renamed primary keys work, and that clearing the key must not issue an additional query\.

All four are admitted, and all four discriminate the pre\-fix behavior\. They are not equivalent: satisfying one does not imply satisfying another\.

##### Where the backends agree\.

Disagreement about how to state a fix does not imply disagreement about what the fix is about\. On the289289SWE\-bench Verifiedinstances where all three formal backends record a resolvable modeled location,244244\(84\.4%84\.4\\%\) model exactly the same set of repository files, the median Jaccard overlap of the three file sets is1\.01\.0, and pairwise agreement runs from87\.0%87\.0\\%to88\.4%88\.4\\%\. On364364of the500500instances all three agree on how many functions need modeling\. The independent construction of three formal twins therefore converges on the same site and differs on how to describe it\.

##### Carriers\.

The remaining differences are largely about representation\. Every specification must choose a carrier for values the language cannot represent natively, and the choices differ systematically: the contract\-annotated Python bundles carry strings as sequences of integer code points \(144144bundles\), as an interned identity token \(9191\), or as an explicit integer sequence \(262262use a sequence carrier of some kind\), and declare in186186cases that the tested behavior does not depend on string content at all\. TheLean\-based bundles, having a real character type available, carry strings as lists of characters \(140140bundles\) and reach for rationals \(4343\) where the Python backend needs an interpreted real encoding \(1111bundles\)\. None of these choices is free: a specification that treats strings opaquely cannot state a property about their content, and the provenance map records the decision so that a reader can see which properties were placed out of scope\.

##### What none of them can say\.

All four languages describe the values a program computes\. None describes the names it computes them under, the module a symbol is imported from, the class a value is an instance of, or an effect visible only in process or filesystem state\. When a task’s graded test discriminates on one of those, no specification in any of the four languages can separate the fix from the bug, and the instance is refused\. That boundary is the subject of[SectionA\.3](https://arxiv.org/html/2609.21190#A1.SS3), and it is a property of the task rather than of the language: all three formal backends run into it on exactly the same2222SWE\-bench Protasks\.

### Appendix EConstruction and admission

A bundle earns its place in the release\. It is drafted by agents that are allowed to see everything, then attacked by agents that are allowed to see almost nothing, then admitted only when thirteen independent checks all report a positive or explicitly inapplicable outcome\. This section describes that pipeline: what is built and by whom \([SectionE\.1](https://arxiv.org/html/2609.21190#A5.SS1)\), what the mechanical checks actually establish and at what cost \([SectionE\.2](https://arxiv.org/html/2609.21190#A5.SS2)\), how the adversarial audit is run and adjudicated \([SectionE\.3](https://arxiv.org/html/2609.21190#A5.SS3)\), and how a recorded outcome is kept honest as the checks themselves change \([SectionE\.4](https://arxiv.org/html/2609.21190#A5.SS4)\)\.

Two design commitments run through all of it\. The first is that*generation is adversarial and adjudication is mechanical*\. Every attack on a specification is produced by an agent whose instructions are to break the bundle, and then resolved by running the verifier, the repository, or the task’s own test suite\. An attack takes one of three forms: a wrong implementation that might slip through, an axiom that might be false of the real callee, a scenario the specification might mis\-describe\. An attacker’s opinion that a specification is weak is not a verdict; an implementation that verifies when it should not is\.

The second is that*inapplicable is not the same as passing*\. A check that does not apply to a bundle records that fact and its reason\. This makes the tables in this section read differently from the usual pass\-rate summary: the denominator of a check is the set of bundles it can speak about, and the burden of a check that cannot speak falls on the ones that can\.

#### E\.1How a bundle is constructed

Construction runs in four stages, with an information asymmetry between the first and the third: the agents that write a specification see the gold patch, and the agents that attack it do not\.

##### Stage one: modeling\.

A construction agent receives the task’s problem statement, the repository at the frozen base commit, the gold patch, and the task’s own test suite\. It decides what to model \(which functions the fix touches, what their arguments and results are, what carrier each value needs in the target language\) and writes the specification module, the reference implementation, and the pre\-fix twin\. The verifier is in the loop throughout, so the stage ends with an object that verifies and a twin that does not\. It also ends with the two documents that carry the transfer to the repository: the operation\-by\-operation correspondence and the provenance record of every modeling decision \([SectionB\.1](https://arxiv.org/html/2609.21190#A2.SS1)\)\.

This is the stage where the interesting choices are made, and they are recorded as they are taken\. When a specification treats strings opaquely because the tested behavior does not inspect their content, that is a decision with consequences for what the specification can express, and it is written down at the moment it is taken\.

##### Stage two: proof\.

The kernel\-checked backend needs a proof term, not a solver call, so a separate stage searches for one\. The output is a closed proof or nothing: a proof that merely elaborates while leaving its obligation unfinished is not accepted, and the check in[SectionE\.2](https://arxiv.org/html/2609.21190#A5.SS2)that looks for this is the one that separates a file that compiles from a theorem that is proved\. Across both released corpora no proof anywhere is left open\.

##### Stage three: attack\.

The bundle is then handed to four independent adversarial auditors, described in[SectionE\.3](https://arxiv.org/html/2609.21190#A5.SS3)\. They do not receive the gold patch\. Each has a single assignment: break the transfer, break the specification’s strength, break its agreement with the issue, or recover the fix from the view a solver would see\. Each produces concrete artifacts rather than judgments: a candidate implementation, a concrete input, a scenario\. The artifacts are adjudicated by running them\.

The output of this stage is the property test suite, the axiom probes, and the differential harness of[SectionB\.3](https://arxiv.org/html/2609.21190#A2.SS3), together with a written record of every attack that was attempted and the reason it failed\. That record makes the corpus re\-auditable rather than merely re\-runnable \([SectionE\.4](https://arxiv.org/html/2609.21190#A5.SS4)\)\.

##### Stage four: admission\.

Finally the thirteen checks are run and their outcomes recorded\. A negative outcome does not produce a patch to the bundle; it sends the instance back for revision at whichever stage owns the defect\. A specification that admits a wrong implementation goes back to stage one, not to stage three\. The alternative, adding a precondition until the attacker’s counterexample stops verifying, is how a specification stops being about the task\.

##### Refusal\.

Some instances cannot be built, and refusal is a recorded outcome with a stated reason rather than a silent omission\. Of the1,0641\{,\}064candidate bundles onSWE\-bench Pro,6868are refused, and their reasons fall into a small number of categories: the change has no value\-level footprint at all \(a rename, a relocation, an import cleanup\), the change is behavior\-preserving so no differential witness exists that could separate it from the code it replaces, or the change’s only observable effect is an irreversible external side effect that the specification languages cannot describe\.[SectionA\.3](https://arxiv.org/html/2609.21190#A1.SS3)works through the2222SWE\-bench Protasks that no backend can model\. The remaining refusals are per\-backend: an instance can be refused under one language and admitted under another, and22furtherSWE\-bench Protasks are in that position, so2424of the266266carry at least one refusal\.

##### What construction cost\.

[Table10](https://arxiv.org/html/2609.21190#A5.T10)totals the machine work behind the two released corpora\. Verifying the specifications takes under nine hours of prover time in total, so the figures are dominated by trying to refute them\. Driving the property test suites takes an order of magnitude longer than verifying every released bundle, and the differential and axiom evidence is measured in hundreds of millions of executions of real repository code\.

Table 10:Machine effort behind the released corpora, pooled over both and over all backends\. “Volume” counts the units of work admission performed, and excludes the verifier calls made while a specification was still being drafted\. Prover time is totalled for the two activities that are pure verification; a dash means the activity was not timed separately\. Wall\-clock totals are serial prover time; admission itself runs in parallel\.

#### E\.2The mechanical checks

Thirteen checks decide admission\.[Table11](https://arxiv.org/html/2609.21190#A5.T11)lists them with what each one establishes and how often it applies\. Ten are decided by running something \(a verifier, a repository, a test suite\) and three by reading the bundle statically\. None of them involves a judgment call; the two that rest on adversarial work read a verdict that was already adjudicated mechanically, and they are the subject of[SectionE\.3](https://arxiv.org/html/2609.21190#A5.SS3)\.

Table 11:The thirteen admission checks, pooled over both corpora and all four backends \(3,0643\{,\}064candidate bundles\)\. “Applies” is the number of bundles the check can speak about; “inapplicable” is the number where it explicitly cannot, almost always because the check is meaningless for that backend \(the prose backend has no verifier\) or for that bundle \(no axioms to probe, no operators to mutate\)\. “Decided by” distinguishes a static reading of the bundle, an execution inside the task’s official container, and a verifier verdict\.CheckWhat a positive outcome establishesDecided byAppliesInapp\.VerificationThe released specification module verifies, and every correctness obligation in it is genuinely dischargedverifier2,2332\{,\}233784784DiscriminationEvery pre\-fix twin fails verification under the byte\-identical specificationverifier2,2332\{,\}233775775Gold agreementThe implementation that mirrors the gold patch verifies, and the twins still fail against itverifier2,2332\{,\}233768768MutationSystematic mutants of the reference body are rejected at or above the backend’s floorverifier2,1332\{,\}133882882Property test suiteEvery case in the property test suite receives the verdict it is labeled withverifier2,2312\{,\}231784784ResolutionThe equivalent patch applies at the base commit and passes the task’s hidden testscontainer3,0013\{,\}00155Axiom soundnessEvery axiom holds of the real callee on generated and boundary inputscontainer1,8961\{,\}8961,1271\{,\}127ConformanceThe reference implementation and the patched repository function agree on a large generated input setcontainer2,0932\{,\}093934934Task identityThe frozen task row is unmodified and the bundle adds only permitted filesstatic3,0163\{,\}01600HygieneRegion markers are intact, the solver\-facing view is stable, and no prohibited construct appearsstatic3,0003\{,\}00033DisclosureThe solver\-facing view does not reveal the fixstatic2,9302\{,\}9301010Refinement fidelityAn adversarial reading of the correspondence found no unjustified stepaudit3,0003\{,\}00022Attacker panelThree blind attackers found no weakness, no unfaithfulness, and no disclosureaudit3,0023\{,\}00200Of the3,0643\{,\}064candidate bundles,3,0023\{,\}002carry all thirteen outcomes; the6262that do not are refusedSWE\-bench Probundles that never reached admission,2727of which carry no outcome at all\. On the2,9962\{,\}996admitted bundles every applicable check is positive\.

##### Verification, and what it costs\.

[Table12](https://arxiv.org/html/2609.21190#A5.T12)gives the wall\-clock cost of a single successful check of a released bundle\. The three formal backends differ by a factor of five in the mean and by a factor of three in the tail, and the ordering is the one the languages predict: reasoning about mutable heap state through an intermediate verification language and an SMT solver is the most expensive, a solver\-backed tactic over a small imperative embedding is the cheapest, and kernel checking of an already\-found proof sits between them\.

Table 12:Verification wall clock for one successful check, in seconds\. Every bundle whose verification passes contributes one measurement, which onSWE\-bench Prois a few more bundles than are released, since a bundle can verify and still be refused on another check\. The totals are serial prover time\.Figure 5:The same measurements as[Table12](https://arxiv.org/html/2609.21190#A5.T12), drawn on one logarithmic scale\. Each row is a backend on a corpus\. The band spans the median to the 95th percentile, the tick marks give the median, the 90th, and the 95th percentile, the open circle is the mean, and the cross is the slowest single bundle in that cell\. The three formal backends separate without overlap onSWE\-bench Verified, and the two Lean\-based backends move*left*onSWE\-bench Proeven thoughSWE\-bench Profixes are five times larger\.SWE\-bench Protasks are five times larger thanSWE\-bench Verifiedtasks by patch size, and yet under the two Lean\-based backends they verify*faster*\([Figure5](https://arxiv.org/html/2609.21190#A5.F5)\), because the specifications written for them are more abstract\. A larger fix forces more of the surrounding behavior to be parameterized rather than modeled, and a parameterized specification generates smaller proof obligations\. Under the contract\-annotated Python backend, where the surrounding state has to be modeled to be talked about at all, the expected direction reappears:29\.429\.4s onSWE\-bench Proagainst25\.825\.8s onSWE\-bench Verified, with a median that rises by half\. The cost of a specification tracks how much of the program it commits to describing, not how large the patch is\.

The tail is dominated by a small number of bundles\. Under the contract\-annotated Python backend the slowest released bundle takes288\.6288\.6s where the median takes18\.418\.4s, a factor of sixteen, and the p90\-to\-p95 step is steep, so a tight fixed timeout would act as a filter on which specifications can be admitted\. A solver working under a per\-call verifier budget works against this distribution;[SectionG\.2](https://arxiv.org/html/2609.21190#A7.SS2)reports what it does with it\.

##### Discrimination and gold agreement\.

These two checks keep a merely consistent specification out of the release\. The first requires every pre\-fix twin to fail under the exact specification the reference implementation passes under;1,7091\{,\}709twins across theSWE\-bench Verifiedcorpus must fail, and all of them do\. The second requires that the implementation mirroring the gold patch also verifies and that the twins still fail against it, so the specification is satisfied by the repository’s own answer and not only by ours\. Where a fix repairs several independent behaviors the bundle ships one twin per behavior, so these checks are not one bit of information per bundle:8181Naginibundles carry two or more\.

##### Mutation\.

[Table13](https://arxiv.org/html/2609.21190#A5.T13)reports the mutation check\. It generates systematic single\-point mutants of the reference implementation’s body, holding the specification fixed, and requires the verifier to reject them\. Pooled over both corpora,25,16325\{,\}163of25,45725\{,\}457mutants are rejected, a rate of99\.17%99\.17\\%, and1,9961\{,\}996of2,1332\{,\}133eligible bundles reject every mutant generated for them\.[Figure6](https://arxiv.org/html/2609.21190#A5.F6)splits the same counts by corpus\.

Table 13:The mutation check\. “At100%100\\%” counts bundles that reject every mutant generated for them\. “Survivors” counts mutants that verified against the frozen specification\. The admission floor is80%80\\%per bundle for all backends; the pooled rate is far above it\.Figure 6:The mutation check by backend and corpus, splitting the totals of[Table13](https://arxiv.org/html/2609.21190#A5.T13)\.Left:the share of bundles that reject every mutant generated for them; the number inside each bar is how many bundles those are, and the number beside it is how many of the cell’s bundles left at least one survivor\.Right:the pooled rejection rate over all mutants of the cell, on an axis that begins at97%97\\%; the per\-bundle admission floor of80%80\\%is far below the bottom of this range\. Survivors are concentrated in one backend and one corpus\.The floor is80%80\\%rather than100%100\\%, and it bounds specification tightness rather than measuring correctness\. A surviving mutant means the specification does not distinguish the mutated body from the reference one\. Sometimes that is a real weakness\. Sometimes it is not: a mutant that changes a value the specification deliberately does not constrain \(a subexpression whose result is discarded, an operation on a field the postconditions leave free\) is genuinely indistinguishable, and a specification that killed it would be over\-fitted to one way of writing the answer\. Demanding100%100\\%would therefore reject sound bundles, and it would reject them preferentially on the backends where the proof is found by search rather than supplied, since there a survivor can also mean the search did not find the refutation\. The tighter check on specification strength is the discrimination lane of the property test suite and the attacker panel, both of which use targeted wrong implementations rather than random perturbations\.

The per\-backend spread is informative about the languages rather than about the specifications\. The contract\-annotated Python backend has ten times the survivor count of the other two, on a comparable number of mutants\. Its specifications talk about mutable state through explicit permissions, which leaves more of the reference body outside what the postconditions constrain; the two Lean\-based backends state their obligations over pure values, where almost every operation in the body feeds the result\. The lower score is a property of what the language chooses to leave unconstrained, and the released bundles are the ones that clear the floor with the survivors individually reviewed\.

882882bundles are inapplicable, of which766766are prose bundles with no verifier to run and116116have reference bodies with nothing to mutate\. A body that performs a single assignment offers no operator to perturb\. Those bundles carry the burden on the twin and the property test suite instead\.

##### Resolution\.

The equivalent patch must apply at the frozen base commit and pass the task’s hidden tests in the task’s own official container\. All3,0013\{,\}001admitted bundles resolve\. Twenty\-three needed more than one attempt and are flagged as flaky, all of them onSWE\-bench Verified, distributed as1010under the contract\-annotated Python backend,99under the embedded imperative language and44under the kernel\-checked backend\. The flakiness is in the repository’s own test suite, not in the specification; the same patch resolves on retry\. The flag is recorded so that a downstream user comparing evaluation results against these instances knows which ones have a noisy oracle\. NoSWE\-bench Probundle needed a retry\.

##### Axiom soundness\.

An axiom is the one place in a bundle where something is asserted rather than proved, so each is tested against the running repository\.1,8931\{,\}893probe runs constructed inputs, called the real callee inside the task’s official container, and compared the observed behavior against what the axiom claims\.428428runs concentrated on boundary inputs \(428,609428\{,\}609cases\) and526526ran generated inputs at volume \(20\.620\.6M cases\), for25\.725\.7M recorded observations of real callee behavior in total\.164164runs were performed against the patched tree rather than the base commit, which is the correct choice when the probe needs a data structure the fix introduces\. No probe refutes its axiom on any admitted bundle\.

1,1271\{,\}127bundles are inapplicable, and815815of the runs report that the bundle they were built for declares no axioms at all\. That is the expected outcome under the kernel\-checked backend, where a callee that can be abstracted does not need to be assumed \([SectionB\.2](https://arxiv.org/html/2609.21190#A2.SS2)\)\.

##### Conformance\.

The conformance check compares the reference implementation against the real patched repository function on a large generated input set, inside the task’s container, with a pinned seed\. It is a refutation instrument: it cannot prove that the model is faithful, but a single disagreement proves it is not\. Across2,0932\{,\}093recorded verdicts there are no disagreements anywhere in either corpus, over187\.9187\.9M compared inputs, and1,4611\{,\}461verdicts individually confirm at least10510^\{5\}inputs\.

What the check establishes differs by backend\. Under the contract\-annotated Python backend the reference implementation is Python, so it is executed directly against the repository function:695695verdicts are of this kind\. Under the two Lean\-based backends the reference implementation is not executable in the repository’s language, so the comparison runs against a transliteration of the model into Python:1,4011\{,\}401verdicts are of this kind\. A transliterated comparison tests the model as transliterated, which is a weaker claim, and it is recorded as such rather than pooled with the direct comparisons\.934934bundles are inapplicable:766766prose bundles,146146with no executable reference region to compare,66whose model uses a construct the comparison cannot represent, and one with no comparison built\.

##### The property test suite\.

Every case in every property test suite receives the verdict it is labeled with:23,48223\{,\}482of23,48223\{,\}482, across2,2312\{,\}231bundles, with no failures in either corpus\.[Table8](https://arxiv.org/html/2609.21190#A2.T8)gives the per\-backend breakdown by lane\.

Driving the property test suites is the single most expensive part of admission\. The23,48223\{,\}482cases consume87\.087\.0hours of serial prover time at a mean of13\.313\.3s and a median of8\.18\.1s per case\. The slowest case, at2,7652\{,\}765s, is a wrong implementation that the contract\-annotated Python backend’s solver had to work hard to reject\. That single case costs more prover time than verifying every releasedVelvetbundle in theSWE\-bench Verifiedcorpus\.

##### Task identity, hygiene, and disclosure\.

Three static checks guard properties that would be easy to violate by accident\.

Task identity requires that the frozen task record is byte\-identical to the upstream row and that the bundle adds only permitted files\. It rules out admission against an edited task: a relaxed test list, a different base commit, an amended problem statement\.

Hygiene is a structural reading of the bundle: that the region markers which define the solver\-facing view are intact and in the right order, that projecting the view twice gives the same result as projecting it once, that no module outside a fixed permitted set is imported, and that no construct appears which would let a specification be satisfied vacuously\. The last of these is per\-backend, because each language admits vacuity differently: a contract\-declaration form used outside the axiom region, a missing termination directive in the embedded imperative language, a divergence\-admitting construct or an unfinished proof in the kernel\-checked one\. View stability does the most work in practice, since the view is what a solver sees\.

Disclosure screens that view for content that would reveal the fix\.2,9302\{,\}930bundles pass it outright\.7676carry a documented relaxation, meaning the view discloses more about the intended fix than the problem statement alone does\. The relaxation is written down and the affected instances are listed in[SectionA\.4](https://arxiv.org/html/2609.21190#A1.SS4)\. A solver’s patch is still graded by the repository’s own tests\. Ten bundles are recorded as inapplicable and one as failing the screen; none of them is in the release\.

#### E\.3The adversarial audit

The mechanical checks establish that a specification is internally consistent, tight against perturbation, and true of the repository on the inputs anyone tried\. They cannot establish that it is about the right thing\. Four properties resist mechanization because they are claims about the relationship between a formal object and an informal one: that the transfer from the verified model to the executed patch is sound, that the specification is strong enough to be worth verifying, that it agrees with what the issue asked for, and that it does not give the answer away\. Each is assigned to a blind adversarial auditor\.

##### The four roles\.

The*fidelity*role attacks the transfer\. It reads the correspondence map and the provenance record and looks for a step that is neither an identical operation, nor a type refinement from a modeled carrier to a concrete repository type, nor a call to an axiomatized callee\. A step that fits none of the three is a hole in the argument that verifying the model says anything about running the patch, and the role’s job is to find one and exhibit the behavior it permits\.

The*strength*role attacks the specification\. Its assignment is to write an implementation that a reader of the issue would call wrong and that nonetheless verifies against the frozen specification\. The role delivers a program, and the program is run\.

The*faithfulness*role attacks the agreement between the specification and the issue\. Its assignment is to describe a concrete scenario in which the specification and the issue’s stated intent come apart: an input on which the specification demands behavior the issue does not ask for, or permits behavior the issue forbids\. This is the hardest of the four, and it is the property that dominates failure when solvers write specifications themselves \([SectionG\.3](https://arxiv.org/html/2609.21190#A7.SS3)\)\.

The*disclosure*role attacks the view\. It receives only what a solver in the specification\-provided setting would receive and tries to reconstruct the fix from it\. A successful reconstruction is a finding whether or not the mechanical screen of[SectionE\.2](https://arxiv.org/html/2609.21190#A5.SS2)flagged anything, because the leak it finds is semantic rather than lexical\.

##### Blindness, and what it means\.

The attacking agents do not receive the gold patch, and they run separately from construction\. An agent that has seen the intended fix cannot judge whether a specification discloses it, and an agent that wrote a specification is a poor judge of whether it is faithful to an issue it has already interpreted\. Each role runs against the bundle as committed, with the reference implementation withheld from the roles for which seeing it would be disqualifying\.

##### Adjudication\.

A role’s report is recorded but never taken as a verdict on its own\. When the strength role produces a candidate implementation, that implementation is run against the frozen specification, and the outcome, not the report, decides the check\. When the fidelity role claims a step is unjustified, the claim is checked against the repository\. When the disclosure role reconstructs a fix, the reconstruction is compared against the gold patch by a separate process that does have it\.

Attacks that succeed do not become bundle patches\. They become property test suite cases, and the instance is sent back for revision\. The discrimination lane of the released property test suites is three times the size of the other two lanes combined, most of it the accumulated residue of attacks that once worked\.

##### What the audit found\.

[Table14](https://arxiv.org/html/2609.21190#A5.T14)reports the outcomes\. All four roles ran on3,0023\{,\}002bundles\. The strength role reports the specification sound on2,9972\{,\}997; two are recorded as broken and two as inconclusive, and none of those four is in the release\. The faithfulness role clears all3,0023\{,\}002\. The disclosure role returns1818findings and2,9842\{,\}984clean reports; every finding is dispositioned, either repaired by narrowing the view or recorded as one of the documented relaxations of[SectionA\.4](https://arxiv.org/html/2609.21190#A1.SS4)\. The fidelity role clears3,0003\{,\}000and rejects two\.

Table 14:Adversarial audit outcomes, pooled over both corpora and all four backends\. Each role produces one report per bundle\. “Inconclusive” means the role could neither construct an attack nor certify the bundle within its budget, which is recorded as a negative outcome for admission purposes\.The table records that a stated attack, run by a specific attacker under a specific budget, did not break these specifications\. Attacks that succeeded during construction were resolved by sending the instance back for revision rather than by recording a failure, so the finding rate on released bundles is close to zero by construction\. The audit’s calibration is measured on specifications that were*not*built this way\.[SectionE\.4](https://arxiv.org/html/2609.21190#A5.SS4)reports that calibration, and[SectionG\.3](https://arxiv.org/html/2609.21190#A7.SS3)reports what the same procedure finds when it is pointed at solver\-written specifications, where it rejects a third to a half of the specifications written under specification synthesis and close to two thirds of those written end\-to\-end\.

##### Judges\.

The audit roles were run with a frontier model as the attacking agent \(2,7412\{,\}741of the3,0023\{,\}002bundles per role\) and a stronger successor model on the remainder \(261261\)\. Where the two overlap, the successor model is the calibration instrument of[SectionE\.4](https://arxiv.org/html/2609.21190#A5.SS4)rather than the recorded verdict\.

#### E\.4Keeping a recorded outcome honest

An admission record computed once, by code that has since changed, is a historical claim\. Three mechanisms keep the record current\.

##### Every outcome is versioned\.

Each of the48,22748\{,\}227recorded outcomes carries the identity of the code that produced it:39,78639\{,\}786carry the checking code’s version and18,84018\{,\}840additionally carry a digest of the harness configuration in force\. The record spans five weeks of construction, from mid\-July to mid\-August, during which the checking code changed many times\.625625distinct versions appear across the two corpora, together with3737distinct harness configurations\. When a check’s definition changes, the outcomes produced under the old definition become identifiably stale and can be re\-derived;102102outcomes were re\-derived in this way after a check tightened\.

##### Admission is recomputed, never stored\.

A bundle’s admitted status is derived, on every read, from the recorded outcomes of the thirteen checks by a single procedure, and every consumer of the corpus uses that one procedure\. The counts in this appendix therefore agree with the counts in[Table4](https://arxiv.org/html/2609.21190#A1.T4)and with what a downloaded bundle reports about itself\. The only way a bundle becomes admitted is by acquiring a positive outcome, and a positive outcome requires the check to have run\.

##### Re\-auditing is cheap because the attacks are kept\.

Each bundle’s record contains, verbatim, the attacks the audit attempted and the reason each failed, including the candidate implementations\. A later audit that wants to know whether a bundle is still sound can replay the previous attacks against the current specification for the cost of running the verifier, and only has to generate new attacks where the specification has changed since\. Re\-auditing the corpus is therefore a routine operation rather than a full reconstruction: an attack that failed in July still fails in August, and a reader can check that\.

##### Calibrating the audit\.

The audit’s accuracy cannot be measured on the released corpus, where it almost never finds anything\. The measurement is made instead on the specifications that solvers write during evaluation, where findings are common enough to compare judges against each other\. On those, a blind re\-audit by a stronger model, run with no access to the original verdicts, reaches a Fleissκ\\kappaof0\.8610\.861among its three independent ballots on the faithfulness property and0\.8710\.871on the overall verdict, and its verdicts are strongly associated with the independent outcome of whether the solver’s patch resolved the task \(a Mantel–Haenszel odds ratio of13\.513\.5,95%95\\%CI\[7\.96,22\.93\]\[7\.96,22\.93\]\)\.[SectionG\.3](https://arxiv.org/html/2609.21190#A7.SS3)reports that analysis in full\. The same procedure, applied to specifications built without the pipeline above, rejects between a third and two thirds of them, depending on the model and the setting\. The near\-zero finding rate on the released corpus is thus a property of the construction process and not of the audit\.

### Appendix FEvaluation protocol

The evaluation is built around a single discipline: between any two settings we compare, exactly one thing changes\. The model, the agent scaffold, the container, the step budget, the way the issue text is presented, and the sanitization of the working tree are identical\. What differs is either the formal information the agent is handed or the rule by which its submission is graded, and never both at once\. Every claim in[Section4](https://arxiv.org/html/2609.21190#S4)is a difference between two settings that satisfy that condition\.

This appendix records the protocol in four parts\.[SectionF\.1](https://arxiv.org/html/2609.21190#A6.SS1)states what each of the nine settings gives the agent and what it demands in return\.[SectionF\.2](https://arxiv.org/html/2609.21190#A6.SS2)describes the environment that is held fixed across all of them, and reports what an episode actually looks like from the inside\.[SectionF\.3](https://arxiv.org/html/2609.21190#A6.SS3)describes the graders, three mechanical and three agentic, and the order in which they run\.[SectionF\.4](https://arxiv.org/html/2609.21190#A6.SS4)records the models, budgets, and repetitions behind every number we report\.

#### F\.1The nine settings

A setting is fixed by three choices: what the agent is told, what it must hand back, and what has to be true of what it hands back\.[Table15](https://arxiv.org/html/2609.21190#A6.T15)states all three for each of the nine rows of[Table2](https://arxiv.org/html/2609.21190#S4.T2)\.

Table 15:The nine evaluation settings\. “Local\.” is edit localization: the functions and files the fix touches\. “Spec\.” is the specification view of[appendixB](https://arxiv.org/html/2609.21190#A2): the specification and its axioms with the reference implementation and all proof annotations removed\. “Verifier” means the agent can invoke the backend’s verifier on its own artifact during the episode\. Row numbers match[Table2](https://arxiv.org/html/2609.21190#S4.T2)\. The table states the protocol for the three prover\-backed backends; the structured\-requirements backend deviates as described at the end of this subsection\.##### The three axes\.

The first axis is*localization*\. Rows 4–7 name the functions and files the fix touches; rows 0–3 and row 8 do not\. Localization is a genuine hint about the repository and nothing more: it names sites, never behavior\. It is computed once, offline, from the frozen corpus, so the evaluation process never opens a gold patch while an episode is running\.

The second axis is the*specification*\. Rows 6 and 7 hand the agent the specification view: the formal contract, the axioms that summarize the callees it depends on, and the operation correspondence, with the reference implementation and every proof annotation removed\. Rows 2, 3, 5, and 8 ask the agent to write that artifact itself\. This is the axis the paper is about\. The pair \(row 7, row 3\) isolates it exactly: identical scaffold, identical grading rule, and the only difference is whether the specification arrives with the task or has to be constructed\.

The third axis is*what is graded*\. Five settings are graded on the patch alone, by the official held\-out test suite\. Three are graded on the formal artifact as well: the submitted implementation must verify, and an adversarial panel must confirm that it and the patch describe the same behavior\. Row 8 is graded on the specification alone and asks for no patch at all\.

##### The designed contrasts\.

Six differences between rows are load\-bearing, and each is a comparison in which everything but one axis is fixed\.

- •Row 4−\-row 0measures localization alone\. Both settings are graded by the held\-out tests and neither mentions formal machinery, so the difference is the value of knowing where to edit\.
- •Row 6−\-row 4measures the specification and its verifier*on top of*localization\. Row 6 gives everything row 4 gives, plus the specification view and the verifier, and is graded identically\. This is the contrast that separates the benefit of a correct specification from the benefit of the localization it necessarily reveals\.
- •Row 2−\-row 0measures the effect of asking an agent to verify while changing nothing about its inputs\. Row 2 has the verifier and the instruction to use it; the issue text, the repository, the budget, and the grading rule are those of row 0\.
- •Row 7−\-row 3measures a supplied specification against a constructed one under a single grading rule\. Both rows require a verifying implementation, a resolving patch, and a confirmed equivalence; only the origin of the specification differs\.
- •Row 3−\-row 2androw 7−\-row 6measure the price of the stricter grade\. In each pair the agent’s environment is identical and only the pass predicate tightens, from “the patch resolves” to “the patch resolves and the formal artifact holds up\.” A large drop in either pair would mean that resolution rates in the loose settings are being carried by submissions whose formal half does not survive inspection\.
- •Row 1−\-row 0measures the price of an independent counterexample search on a submission that the held\-out tests already accepted\. It is the control for the claim that a test suite is an incomplete correctness signal; the two rows differ in the grader and in nothing else\.

Rows 5 and 8 stand slightly apart\. Row 5 completes the grid: it is row 3 with localization added, or equivalently row 7 with the specification withheld\. It therefore separates the two halves of what row 7 supplies\. Row 8 is not a patch\-writing task\. The other eight settings measure specification quality only through its downstream effect on a patch; row 8 measures it directly\.

##### Why the settings come in pairs\.

Five of the nine rows are the loose\-graded twin of a strict\-graded row: \(0, 1\), \(2, 3\), \(4, 5\), and \(6, 7\)\. The pairing is deliberate\. An agent’s behavior depends on what it is asked to produce, and a grading rule that is announced in the prompt is part of what it is asked to produce; a grading rule applied after the fact is not\. Rows 1 and 3 differ from rows 0 and 2 in both respects, so neither pair on its own separates the effect of the information from the effect of the rule\. Reading the four pairs together does: rows 0, 2, 4, and 6 vary the information under a fixed rule, and each strict row is anchored to the loose row it was built from\.

##### What the specification view withholds\.

The view given in rows 6 and 7 is not the corpus artifact\. The reference implementation is removed, every proof annotation and lemma is removed, and the view is screened for text that would name the fix rather than describe the required behavior;[SectionA\.4](https://arxiv.org/html/2609.21190#A1.SS4)describes the screen and the cases where a relaxation was recorded instead\. What remains is a formal statement of what the changed functions must do, the axioms that pin down the behavior of the callees they rely on, and the correspondence between the abstract operations and the repository’s own names\. An agent in row 6 therefore knows exactly what correctness means for this issue and still has to write repository code that achieves it\.

##### The natural\-language backend\.

The structured\-requirements backend has no prover, so the protocol changes shape for it\. Its verify slot holds a well\-formedness check on the requirements document, not a proof: “verified” for one of its episodes means the document is a well\-formed set of requirements with triggered acceptance criteria, never that anything was proved\. Nothing about it can be graded on verification, on equivalence, or on a specification audit\. Every one of its settings is therefore graded on the patch, and the strict half of each pair is graded by the adversarial counterexample audit of[SectionF\.3](https://arxiv.org/html/2609.21190#A6.SS3)\. That audit uses the Lean sibling of the same instance as formal ground truth: the two artifacts describe the same task, so a formal oracle is available even where the submitted document cannot itself be checked\.

The audit decides rows 3, 5, and 6 for this backend; rows 2, 4, and 7 are decided by the held\-out tests alone\. Each pair is agent\-side identical \(same prompt, same tools, same budget\), so within a pair the entire difference lives in the grader\. In rows 6 and 7 the requirements document handed to the agent is complete rather than a view with parts elided, since there is no implementation or proof to remove\. There is consequently no formal artifact for the agent to construct in those two rows\. Row 8 does not apply at all: there is no formal object for a specification audit to check\.

#### F\.2The agent’s environment

Everything in this subsection is identical across all nine settings and all four backends, with the single exception of which tool occupies the verify slot\.

##### Scaffold\.

The agent runs a single reason\-and\-act loop in the style of[Yang et al\. \(2024\)](https://arxiv.org/html/2609.21190#bib.bib54): it emits one tool call at a time, receives the result, and continues\. There is no planner, no retriever, no subagent, no self\-consistency, and no external memory\. Three actions are available\. A shell runs commands inside the task’s container\. A verify action runs the backend’s verifier on the artifact the agent has written, and returns its verdict\. A submit action ends the episode\. Settings 0, 1, and 4 expose the shell and submit only; the verify action is absent\.

The scaffold is deliberately plain\. A stronger harness could recover a weak specification by trying more implementations, and a weaker one could fail to exploit a good specification; a minimal loop keeps the measured difference attributable to the difference in information\.

##### The container\.

Each episode runs in the task’s own official evaluation image, with the working tree at the base commit, the state of the repository immediately before the fix\. The held\-out tests are never present in that image\. Before the agent is given a shell, the version\-control history is sanitized: every reference except the base commit is removed and unreachable objects are pruned, so the future of the branch is not recoverable from inside the container\.

The task description the agent sees is a projection of the corpus record\. The gold patch, the test patch, and the lists of tests that must newly pass and must keep passing are all dropped\. Maintainer hint text, where the upstream dataset carries it, is excluded: it frequently contains the eventual fix in prose\.

##### The verify action cannot leak\.

The verify action runs on the host, not in the container, and it is given exactly one input: the artifact the agent wrote\. It has no access to the repository, to the held\-out tests, to the gold patch, or to any part of the corpus bundle for the instance\. Its output is the verifier’s own verdict on the agent’s own file\. The tool that tells an agent whether its proof went through cannot also tell it what the answer is\.

##### Budget and termination\.

Every episode has a budget of250250agent steps in every setting, model, and backend\. An episode ends when the agent submits or when the budget is exhausted, and both are outcomes that are scored\. An idle turn \(a reply with no tool call and no submission\) draws a reminder of the remaining budget rather than being counted as a submission\.

##### What an episode looks like from the inside\.

[Table16](https://arxiv.org/html/2609.21190#A6.T16)reports the interaction profile of an episode, averaged over sampled episodes in every completed run\. Three patterns stand out\.

Table 16:Interaction profile of an episode, by setting, model, and backend\. Columns are per\-episode means: model turns, shell calls, shell calls returning a nonzero status, verify calls, and verify calls that returned a passing verdict\. “Records” is the length of the recorded episode in events \(mean / median / maximum\)\. Rows aggregate over the repetitions of a cell;ssis the number of repetitions\. Sampled uniformly at6060episodes per repetition\.ShellVerifyRecordsSettingModelBackendssTurnscallsfailcallspassmeanmed\.max0 Unaided baselineOpus 4\.8–412\.211\.20\.25––26\.417\.8162GPT\-5\.5–429\.328\.21\.65––60\.552\.21824 Localization ProvidedOpus 4\.8–410\.29\.20\.27––22\.412\.8163GPT\-5\.5–419\.318\.31\.11––40\.730\.01302 End\-to\-EndOpus 4\.8Nagini422\.618\.90\.542\.741\.2347\.235\.9228GPT\-5\.5Nagini435\.030\.22\.293\.802\.0472\.167\.01523 Verified End\-to\-EndOpus 4\.8Nagini422\.918\.80\.583\.211\.2847\.936\.0206GPT\-5\.5Nagini434\.028\.72\.274\.232\.1269\.967\.81946 Specification ProvidedOpus 4\.8Nagini419\.215\.10\.353\.271\.2340\.627\.9186GPT\-5\.5Nagini426\.022\.01\.533\.011\.7354\.047\.41647 Verified from SpecificationOpus 4\.8Nagini419\.515\.00\.333\.611\.2741\.227\.2238GPT\-5\.5Nagini426\.121\.81\.053\.301\.6354\.346\.51868 Specification SynthesisOpus 4\.8Nagini418\.112\.00\.295\.111\.6338\.126\.5325GPT\-5\.5Nagini422\.315\.10\.956\.082\.0646\.541\.81615 Verified from LocalizationOpus 4\.8Nagini121\.917\.60\.573\.421\.3246\.036\.0150Opus 4\.8Velvet138\.529\.20\.808\.551\.5280\.276\.0226Opus 4\.8Lean120\.017\.20\.952\.051\.2243\.033\.01402 End\-to\-EndOpus 4\.8Ears116\.714\.80\.621\.051\.0235\.528\.0128GPT\-5\.5Ears128\.625\.91\.601\.801\.8059\.258\.01303 Verified End\-to\-EndOpus 4\.8Ears117\.916\.10\.501\.031\.0338\.127\.0132GPT\-5\.5Ears116\.715\.21\.151\.051\.0335\.938\.0112First, the two models work very differently at similar accuracy\. GPT\-5\.5 takes roughly twice as many turns as Opus 4\.8 in every setting and issues four to six times as many shell commands that return an error:2\.32\.3per episode against0\.60\.6in the end\-to\-end settings\. Its episodes are correspondingly longer to read\. The resolution rates of the two models differ by a few points; the interaction cost differs by a factor\.

Second, verify calls scale with how much formal work a setting demands, and they peak where no patch is required at all\. An agent handed a specification \(rows 6 and 7\) calls the verifier about3\.33\.3times; an agent constructing one \(rows 2 and 3\) calls it2\.72\.7–4\.24\.2times; an agent doing nothing but specification synthesis \(row 8\) calls it5\.15\.1–6\.16\.1times\. The verifier is the only feedback channel in row 8, so the agent leans on it hardest exactly where the artifact is all there is\.

Third, the ratio between calls and passing verdicts is a direct measure of how hard a backend is to satisfy\. UnderNaginian agent needs about2\.62\.6calls per passing verdict\. UnderVelvetit needs8\.558\.55calls for1\.521\.52passes, better than five to one\. That is the largest such ratio anywhere in the grid, and it matches the construction\-side picture of[Table12](https://arxiv.org/html/2609.21190#A5.T12)\.Leansits at the other end,2\.052\.05calls to1\.221\.22passes, because its failures are reported as concrete unsolved goals rather than as a search that did not converge\. Backend difficulty is not only a property of the corpus; it is something an agent pays for turn by turn\.

##### The anti\-cheat screen\.

Every recorded episode is screened for the four ways an agent could obtain the answer rather than derive it: recovering the fix from version\-control history, reading a held\-out test, reaching the network, and touching the corpus bundle for its own instance\. The screen is deliberately over\-broad\. It matches on surface patterns and is tuned to over\-report, so a match is a pointer to be read rather than a verdict\. Only a high\-severity match, which means the pattern plausibly reached protected material, marks an episode unclean; browsing sanitized history is recorded and left at the lowest severity, because the history no longer contains the fix\.

Across all53,50053\{,\}500episodes the screen raised2,2342\{,\}234matches on711711episodes:1,2921\{,\}292informational,124124low, and818818high\. The high matches fall on332332episodes,0\.6%0\.6\\%of the total, and carry three labels\. Most,708708, are searches over the path where bundles live, which is not mounted in the agent’s container and so returned nothing\. Ninety are network patterns, and twenty match an agent writing its own regression test into an existing test file, which is a legitimate action whose result the grader ignores\. No episode in any reported cell was excluded for cheating, and the counts above are the screen’s raw output\.

#### F\.3Grading a submission

Six graders decide the numbers in[Table2](https://arxiv.org/html/2609.21190#S4.T2): three mechanical and three agentic\.[Table17](https://arxiv.org/html/2609.21190#A6.T17)states, for each, what a positive outcome establishes and what it does not\.

Table 17:The graders\. “Mechanical” means the verdict is a program’s output on fixed inputs and is reproducible bit for bit; “agentic” means a model panel decided it\. Every agentic grader runs only on a submission that has already passed the mechanical graders that apply to its setting\.##### Resolution\.

The submitted patch is applied to a clean checkout of the repository at the base commit, inside a fresh instance of the official evaluation image, and the official harness runs the held\-out suite\. Nothing the agent did to its own container survives into this step: the patch is the only thing carried across\. An instance resolves when every test that must newly pass does, and every test that must keep passing does\. A small number of instances have tests that are genuinely nondeterministic; a failure that flips to a pass on a bounded retry is recorded as a retry, and[SectionE\.2](https://arxiv.org/html/2609.21190#A5.SS2)reports the rate\.

##### Verification and anti\-fakery\.

In the three strict settings, the agent’s implementation is checked against the agent’s specification by the same verifier and under the same hygiene screen that the corpus admission uses \([SectionE\.2](https://arxiv.org/html/2609.21190#A5.SS2)\)\. A submission is therefore held to exactly the standard the corpus itself was held to\. The screen rejects the obvious degeneracies: a contract weakened until it says nothing, a proof discharged by an escape hatch, a precondition that no input satisfies\. It is mechanical, so it applies uniformly\.

##### Equivalence\.

Verification and resolution are checks on two different objects: a formal implementation and a repository patch\. Passing both separately is not enough, because an agent could verify a trivial implementation and, independently, write a patch that resolves\. The equivalence grader closes that gap\. Three independent judges each receive the submitted specification, the submitted implementation, the submitted patch, and the issue, and decide whether the implementation and the patch describe the same behavior; a majority carries\. When the specification was supplied, the judges additionally check that it is unchanged from the one given, so the strict grade in row 7 cannot be reached by relaxing the target\. When the specification was constructed, they check that it is a faithful reading of the issue\. That is the same question the row 8 audit asks, and it is reported the same way\.

[SectionG\.3](https://arxiv.org/html/2609.21190#A7.SS3)gives the agreement between the three votes, how often the majority overrides a dissent, how much machine evidence each vote rests on, and what the panel’s verdicts predict about an outcome it never sees\.

##### The specification audit\.

Row 8 asks for a specification and a witness implementation and grades the specification against five properties: that its axioms hold of the real callees, that its preconditions admit the inputs the issue is about, that it is true of the intended behavior, that it constrains enough of that behavior, and that it is a faithful reading of the issue\. Each property is voted on by three judges, and the specification passes only if a majority accepts all five\. The judges are tool\-using: a judge can put a candidate implementation through the verifier and can probe a claimed axiom against the real callee in the instance’s own image, so a refutation usually comes with a concrete witness\. Per\-property verdicts are recorded\. The attribution in[Table3](https://arxiv.org/html/2609.21190#S4.T3)rests on them: a specification does not merely fail, it fails on faithfulness\.

##### The counterexample audit\.

The remaining grader inverts the question\. Instead of asking whether a submission is right, it hunts for a behavior on which the submission is wrong despite having passed\. An adversary is given the formal ground truth for the instance from the frozen corpus, the issue, the held\-out tests, and a checkout with the agent’s patch applied, and is asked to exhibit a concrete input on which the patch violates that ground truth\. Its claim is then re\-run mechanically, and must clear two bars: the exhibited behavior must fail under the agent’s patch, and it must hold under the gold patch\. A claim that fails either bar is discarded\. The two bars keep the adversary from demoting a correct patch by inventing a requirement the task never had\.

Outcomes fall into five categories: a real violation, a claim that turned out to test something the task does not require, a claim that did not reproduce, an inconclusive run, and a clean search\. Only a real violation moves a grade, and it only ever moves it downward\. Inconclusive counts as a pass\. The audit never runs on a submission that did not already resolve, so its denominator is the resolved set\.

In row 3 on the structured\-requirements backend with Opus 4\.8,122122of421421resolved submissions \(29\.0%29\.0\\%\) carry a mechanically confirmed violation of the formal ground truth, which turns an84\.2%84\.2\\%resolution rate into a59\.8%59\.8\\%pass rate\. With GPT\-5\.5, on the227227episodes of that cell that completed,9696of191191resolved submissions \(50\.3%50\.3\\%\) do\. These are the sharpest available measurements of how much a held\-out suite lets through\. Row 1 exists as a designed cell for exactly that purpose: the same audit applied to the unaided baseline is the cross\-backend control for these two numbers\. That cell has not been run, so the two rates stand as a measurement on one backend\.

##### Order of adjudication\.

The graders run mechanical\-first, and no agentic grader can create a pass\. In a strict setting a submission must resolve and verify and clear the hygiene screen before a judge sees it; a judge can then only decline to confirm equivalence\. In row 1 and the audited requirements rows the adversary runs only on a submission the tests already accepted, and can only demote it\. Row 8 is the one setting whose grade rests on an agentic verdict alone, since there is no patch to check mechanically\. Its judges are tool\-using, its audit is reported per property, and[SectionG\.3](https://arxiv.org/html/2609.21190#A7.SS3)calibrates it against independent re\-adjudication\.

#### F\.4Models, repetitions, and run accounting

##### Models\.

We evaluate Claude Opus 4\.8\([Anthropic, 2026](https://arxiv.org/html/2609.21190#bib.bib6)\)and GPT\-5\.5\([OpenAI, 2026](https://arxiv.org/html/2609.21190#bib.bib39)\)\. The scaffold, the prompts, and the tool schemas are identical for the two models\. Decoding is left at the provider default in every setting; we vary information, not sampling temperature\.

##### Repetitions\.

The main text reports single\-repetition numbers\. Underneath, eachNaginicell was run four independent times, apart from rows 1 and 5, which have one repetition each;[SectionG\.4](https://arxiv.org/html/2609.21190#A7.SS4)reports all four together with the spread\. The other backends were run once per cell\. Repetitions are independent replays of the same cell: the same instances, the same prompts, the same budget, a different sampling stream\. They bound run\-to\-run noise, not variation across instances\.

##### Budget\.

Every episode in every cell has the same250250\-step budget, and no cell was given a wall\-clock extension\. Episodes ran concurrently, so elapsed time is not a per\-episode quantity;[SectionG\.6](https://arxiv.org/html/2609.21190#A7.SS6)reports the token and step cost of each setting\.

##### Coverage\.

[Table18](https://arxiv.org/html/2609.21190#A6.T18)records the repetitions behind every cell\. The design calls for5252cells and all5252were run, each repetition over all500500instances ofSWE\-bench Verified\.

Table 18:Cell coverage: the number of repetitions of each cell, every one of them over all500500instances\. “N/A” marks a cell the design does not call for\.
##### Accounting rules\.

Every rate in this paper is recomputed from the per\-episode records rather than read from a run summary, and every repetition behind it covers all500500instances\. Where a cell has four repetitions, a table that pools them says so and gives the pooled denominator\.

### Appendix GAdditional results

[Section4](https://arxiv.org/html/2609.21190#S4)reports one number per cell: a single repetition, one criterion, no decomposition\. This appendix reports everything behind those numbers that does not fit in the main table\.

The six parts answer six separate questions\.[SectionG\.1](https://arxiv.org/html/2609.21190#A7.SS1)reports the full evaluation grid on theSWE\-bench Procorpus, the transfer test for every finding of the main text\.[SectionG\.2](https://arxiv.org/html/2609.21190#A7.SS2)asks what the verifier actually costs an agent during an episode, and uses the answer to rank the four backends by how hard they are to satisfy from the inside\.[SectionG\.3](https://arxiv.org/html/2609.21190#A7.SS3)asks what happens when a grader is asked to adjudicate a specification the solver wrote: how the three votes agree, how much machine evidence each vote rests on, and what the verdicts predict about an outcome the graders never see\.[SectionG\.4](https://arxiv.org/html/2609.21190#A7.SS4)repeats everyNaginicell four times and reports the spread\.[SectionG\.5](https://arxiv.org/html/2609.21190#A7.SS5)splits the corpus by structural properties of the reference fix and asks where a supplied specification helps and where it does not\.[SectionG\.6](https://arxiv.org/html/2609.21190#A7.SS6)reports the token, step, and wall\-clock cost of the whole campaign\.

Two conventions hold throughout\. Settings are named and numbered as in[Table2](https://arxiv.org/html/2609.21190#S4.T2); a row number always refers to that table\. And a*cell*is a model, backend, and setting, measured over all500500instances of the corpus\.[Table18](https://arxiv.org/html/2609.21190#A6.T18)records the repetitions behind each one\.

[Figure7](https://arxiv.org/html/2609.21190#A7.F7)draws the main table before any of that decomposition begins, because two of its features organize everything that follows: a supplied specification moves every backend above both specification\-free settings, and requiring the agent to produce one moves every backend far below them\.

Figure 7:Every backend\-dependent cell of[Table2](https://arxiv.org/html/2609.21190#S4.T2), one panel per model\. Each series is one prover backend across the five settings whose outcome depends on the backend\. The two vertical rules are the settings that use no specification and are therefore shared by all backends: the unaided baseline \(row 0, dashed\) and localization provided \(row 4, dotted\)\. The three prover backends track each other closely everywhere except specification synthesis, where they separate by1414points under Opus 4\.8 and1212under GPT\-5\.5\. The prose backend is omitted here and reported on its own in[Table40](https://arxiv.org/html/2609.21190#A7.T40)\.#### G\.1Evaluation on theSWE\-bench Procorpus

Table 19:The full evaluation grid of[Table2](https://arxiv.org/html/2609.21190#S4.T2)repeated on theSWE\-bench Procorpus, over the242242instances whose bundles are green under all four backends, plus row 8 \(specification synthesis\), which[Table2](https://arxiv.org/html/2609.21190#S4.T2)omits\. Columns, settings, and mode numbering are exactly those of[Table2](https://arxiv.org/html/2609.21190#S4.T2), and✓∗\\checkmark^\{\*\}and✓†\\checkmark^\{\\dagger\}mean what they mean there\. In row 8 the agent writes a specification and a verifying witness but no patch, so nothing is scored on tests and✓‡\\checkmark^\{\\ddagger\}marks the adversarial audit of the specification itself\. As in[Table2](https://arxiv.org/html/2609.21190#S4.T2), the EARS columns have no verifier, so their*Verify*entries are the adversarial counterexample audit, and row 8 does not apply to EARS\.[Table19](https://arxiv.org/html/2609.21190#A7.T19)repeats the full evaluation grid of[Table2](https://arxiv.org/html/2609.21190#S4.T2)on theSWE\-bench Procorpus, over the242242instances whose bundles are green under all four backends\. The absolute level is far from saturation everywhere: the unaided baseline resolves61\.2%61\.2\\%for Opus 4\.8 and30\.2%30\.2\\%for GPT\-5\.5, a gap of3131points whereSWE\-bench Verifiedseparates the two models by under four\. The value of formal information also redistributes on these larger tasks\. Localization is the dominant aid for the weaker model \(\+21\.5\+21\.5points for GPT\-5\.5, flat for Opus 4\.8\), and a supplied specification no longer adds to it: both models score*below*their localization\-provided setting when handed the specification \(55\.855\.8–59\.9%59\.9\\%against61\.2%61\.2\\%for Opus 4\.8,37\.837\.8–39\.4%39\.4\\%against51\.7%51\.7\\%for GPT\-5\.5\), and prompting an agent to construct its own specification end\-to\-end costs Opus 4\.8 roughly ten points against the baseline\. On tasks that change a mean of nine functions across three files, a specification of the behavioral core evidently concentrates the agent’s attention on a slice of the required edit in a way thatSWE\-bench Verified’s single\-site patches never exposed\. Two of the main text’s findings do transfer intact\. First, the jointly scored settings track their test\-scored twins closely under every prover backend \(within22points on most cells,55at worst\), so carrying a specification through to a verified, equivalent patch remains essentially free once the patch resolves\. Second, specification synthesis remains the bottleneck: the five\-property audit accepts31\.031\.0–49\.2%49\.2\\%of Opus 4\.8’s specifications and43\.443\.4–53\.3%53\.3\\%of GPT\-5\.5’s\. Finally, the audited settings sharpen the incompleteness finding of the main text: of the61\.2%61\.2\\%/29\.3%29\.3\\%of instances the baseline agents resolve, the adversarial counterexample audit confirms only17\.4%17\.4\\%/3\.7%3\.7\\%against the formal ground truth — a far larger demotion than onSWE\-bench Verified, consistent with the hidden tests of these larger tasks exercising a smaller fraction of the changed behavior\.

#### G\.2Verification effort and backend difficulty

Every setting that hands an agent a verifier is a setting in which the agent can choose how much to use it\. This subsection reports what agents chose, and what that reveals about the four backends\.

##### Engagement is universal\.

In every cell where a verify tool exists, essentially every episode calls it: the engagement rate is1\.0001\.000in3636of the4040cells whose artifact is graded, and at least0\.9980\.998in the other four\. This is a property of the protocol more than of the agents: rows 3, 5, 7, and 8 refuse a submission that carries no verifying artifact, so an agent that never calls the verifier cannot finish\. But rows 2 and 6 impose no such requirement, and under every prover backend their engagement rates are1\.0001\.000as well\. An agent offered a prover uses it whether or not it is forced to\.

Table 20:Verifier use per episode, pooled over the complete repetitions of each cell\. “Calls” is the number of verify invocations in an episode\. Acceptance is reported two ways:*pooled*is accepting verdicts divided by all calls in the cell, and*per ep\.*is the mean over episodes of that episode’s own ratio\. The second is higher wherever a minority of episodes issue a long run of failing calls\. “Verify share” is the fraction of episode wall\-clock spent inside the verifier, again pooled and per\-episode\. The lower block is row 5, the one setting in which all three provers were run with both models, and is the basis of the backend comparison in the text\.Calls per episodeAcceptanceVerify shareSettingModelBackendnnmeanmed\.maxpooledper ep\.pooledper ep\.2 End\-to\-EndOpus 4\.8Nagini20002\.762330\.4680\.7270\.2010\.150GPT\-5\.5Nagini20003\.893270\.5310\.6760\.1270\.1233 Verified End\-to\-EndOpus 4\.8Nagini20002\.962750\.4250\.6980\.2400\.158GPT\-5\.5Nagini20004\.073250\.5170\.6610\.1170\.1156 Specification ProvidedOpus 4\.8Nagini20003\.481880\.4080\.7330\.2520\.178GPT\-5\.5Nagini20003\.422500\.5270\.7780\.1960\.1707 Verified from Spec\.Opus 4\.8Nagini20003\.521700\.3850\.7300\.2430\.170GPT\-5\.5Nagini20003\.312610\.5100\.7850\.1730\.1418 Specification SynthesisOpus 4\.8Nagini20005\.1621170\.3040\.5790\.2720\.134GPT\-5\.5Nagini20005\.584500\.3430\.4560\.1280\.1022 End\-to\-EndOpus 4\.8Ears5001\.04130\.9900\.9950\.0000\.000GPT\-5\.5Ears5001\.98251\.0001\.0000\.0000\.0003 Verified End\-to\-EndOpus 4\.8Ears5001\.05130\.9940\.9970\.0000\.000GPT\-5\.5Ears5002\.02250\.9991\.0000\.0000\.0005 Verified from Local\.Opus 4\.8Nagini5003\.992570\.3520\.6000\.4070\.244GPT\-5\.5Nagini5004\.694180\.4830\.5830\.2590\.282Opus 4\.8Velvet5008\.928460\.1660\.1790\.1850\.205GPT\-5\.5Velvet50010\.6910440\.2850\.2990\.2350\.264Opus 4\.8Lean5002\.06290\.5970\.7530\.2250\.248GPT\-5\.5Lean5002\.832180\.6640\.7600\.2740\.292Figure 8:How hard an agent leans on the verifier, contract\-annotated Python backend, pooled over the four repetitions of each cell\.Left:verifier calls per episode\. The full bar is every call; the dark inner bar is the calls that returned an accepting verdict, with that share printed inside it\.Right:the share of episode wall time spent inside the verifier\. The two settings that supply no specification make no verifier calls and are omitted\. Specification synthesis is the setting in which both models call the verifier most and are accepted least\.
##### Calls scale with the formal work the setting demands\.

[Table20](https://arxiv.org/html/2609.21190#A7.T20)orders the settings the way one would expect if the verifier were being used as a feedback channel rather than as a formality\. An agent handed a specification \(rows 6 and 7\) calls the verifier about3\.43\.4times\. An agent that must write the specification and the patch \(rows 2 and 3\) calls it2\.82\.8to4\.14\.1times\. An agent that writes nothing but a specification and a witness \(row 8\) calls it5\.25\.2to5\.65\.6times\. Row 8 is the setting in which the verifier is the*only*signal available: there is no patch, so there are no tests to run against one\. It is also the setting in which agents lean on the verifier hardest\.

The medians tell a second story that the means hide\. Under Opus 4\.8 the median episode in rows 6 and 7 makes exactly one verify call: it writes an implementation, submits it to the verifier, and is accepted\. The mean of3\.53\.5is produced by a minority of episodes that iterate hard, with a maximum of8888calls in a single episode of row 6 and117117in row 8\. GPT\-5\.5 has almost no such tail \(its per\-episode maximum never exceeds6161\), but its median is22to44\. The two models spend a similar total amount of verifier effort by opposite strategies: Opus 4\.8 usually succeeds immediately and occasionally grinds, while GPT\-5\.5 iterates a little on almost every instance\.

##### Acceptance rate is a measure of backend difficulty\.

The fraction of verify calls that return an accepting verdict is the cleanest available measure of how hard a backend is to satisfy, because it is measured from inside the agent’s own loop rather than from the construction side\. Ordered by that fraction, the four backends separate sharply\.Earsaccepts0\.990\.99to1\.001\.00of calls: its verify slot is a structural check on a requirements document, not a proof, and a competent agent writes a well\-formed document on the first attempt\.Leanaccepts0\.600\.60to0\.660\.66\.Naginiaccepts0\.300\.30to0\.530\.53, depending on the setting\.Velvetaccepts0\.170\.17to0\.290\.29: three to six calls for every accepting verdict, against1\.51\.5to1\.71\.7forLean\.

That ordering matches the construction\-side effort of[Table12](https://arxiv.org/html/2609.21190#A5.T12)and has the same explanation\. A failingLeanattempt reports concrete unsolved goals, so the next attempt is informed\. A failingVelvetattempt reports that a search did not converge, which is far less actionable, and the agent’s next move is closer to a guess\. Backend difficulty is not only a property of how long the reference bundles took to build; it is something an agent pays for turn by turn\.

##### The verifier is a large but not dominant share of an episode\.

Pooled over the campaign, agents spent896896hours inside verifiers against5,9375\{,\}937hours of total episode wall\-clock, or15%15\\%\. Per cell the share runs from12%12\\%to41%41\\%underNaginiand is0%0\\%underEarsby construction \([Figure8](https://arxiv.org/html/2609.21190#A7.F8), right\)\. The pooled share exceeds the per\-episode mean share in nearly every cell, by as much as sixteen points under Opus 4\.8 in row 5, which says that verifier time is concentrated in the same minority of episodes that make many calls\.

Table 21:Distribution of verifier wall\-clock per episode, in seconds, pooled over the complete repetitions of eachNaginicell\. TheEarscells are omitted: their verify slot returns in well under a tenth of a second and every percentile is0\.00\.0\. The last two columns give the corresponding step\-count tail, where250250is the budget\.
##### The tail is long and it lives in specification synthesis\.

[Table21](https://arxiv.org/html/2609.21190#A7.T21)makes the concentration explicit\. The median episode spends about half a minute in the verifier in everyNaginicell, which is the cost of one or two accepting calls on a reference\-sized problem\. The9999th percentile is2020to5050times that\. The extreme is row 8 under Opus 4\.8: a median of3838seconds, a9999th percentile of4,7234\{,\}723seconds, and a single episode that spent14,69614\{,\}696seconds, just over four hours, inside the prover\. Those episodes are agents that have written a specification they cannot discharge and are attempting successive witnesses against it, each of which times out\.

The step\-count tail behaves differently\. The9999th percentile of agent steps never exceeds103103against a budget of250250, and the budget was reached in2525of the campaign’s53,50053\{,\}500episodes\. Agents do not run out of turns; when they fail they fail while still holding budget, which means the reported rates measure capability rather than an interaction limit\.

##### Whether the formal artifact comes first\.

In the settings that require an agent to produce both a verifying artifact and a patch, the order matters: an agent that gets its artifact verifying before it edits the repository is using the formal object to drive the fix, while an agent that patches first and formalizes afterwards is using it as a certificate for a decision already made\. We record which happened by comparing, in each episode, the time of the first accepting verify call against the time of the first shell command that edits repository source\.

Table 22:Share of episodes in which the first accepting verify call precedes the first repository edit\. Rows 3 and 7 also require a patch, so the ordering is a genuine strategy choice; row 8 requires no patch at all, and the residual below100%100\\%is agents that write exploratory edits they later discard\.[Table22](https://arxiv.org/html/2609.21190#A7.T22)shows a large and consistent split between the two models\. Opus 4\.8 patches first in roughly three quarters of episodes; GPT\-5\.5 formalizes first in roughly two thirds\. The gap is stable across settings and backends, is much larger than any seed\-to\-seed variation, and does not translate into a difference in outcome: the two models’ rates in rows 3 and 7 differ by one to six points while their working order differs by forty\. Two very different strategies reach a similar place\.

The backend also matters, and in the direction the acceptance rates predict\. UnderVelvetonly1\.6%1\.6\\%of row\-5 episodes get a verifying artifact before touching the repository, and underLean4\.0%4\.0\\%, against26\.6%26\.6\\%underNaginifor the same model and setting\. When the artifact is expensive to discharge, agents postpone it\. Row 8 is the control: with no patch to write, three quarters to nine tenths of episodes verify before making any repository edit at all, and the residual is exploratory editing that is later abandoned\.

##### Three provers on the same instances\.

Row 5 is the only setting in which all three formal backends were run with the same model, so it is the only place a like\-for\-like prover comparison is available\. All three runs cover the same500500instances\.[Table23](https://arxiv.org/html/2609.21190#A7.T23)reports the comparison\.

Table 23:The three formal backends under row 5 with Opus 4\.8, over all500500instances\. “Verified” is the share whose submitted artifact was accepted by the prover; “Pass” additionally requires the patch to resolve the hidden tests and the equivalence judge to confirm that the patch and the verified artifact agree\. The last two columns repeat the per\-episode verifier statistics of[Table20](https://arxiv.org/html/2609.21190#A7.T20)for reference\.The outcome rates are within2\.42\.4points of each other, and the difference is well inside the sampling noise of a500500\-instance comparison\. What differs by a factor of four is the effort:8\.928\.92verifier calls per episode underVelvetagainst2\.062\.06underLean\. Given a localized target and a prover to satisfy, an agent reaches the same place under all three backends; how much work that takes is a property of the backend, not of the task\.

#### G\.3Adjudicating a solver’s specification

Three of the nine settings ask a grader to decide something about an artifact the solver wrote: whether a verified implementation agrees with the submitted patch \(rows 3, 5, 7\), and whether a synthesized specification is a correct and faithful statement of the issue \(row 8\)\. This subsection reports how those graders behave\. It also reports what happens when the same five\-property audit is turned on the specifications solvers wrote in row 3, which were never audited during scoring\.

##### The equivalence judge rarely refuses\.

The equivalence judge runs only on submissions that have already cleared verification, resolution, and the anti\-fakery check\. Its question is narrow: whether the patch implements the same behavior as the artifact that verified, and whether a handed\-down specification still says what it said\. Three independent ballots decide by majority\. Across the twenty\-two adjudicated runs the judge confirmed between99\.2%99\.2\\%and100\.0%100\.0\\%of the submissions it was given:3434rejections out of9,7639\{,\}763adjudicated submissions, a third of one percent\.

The grounds of those3434rejections correspond to two different failures\. Thirty\-one cited a behavioral disagreement only: the verified artifact and the submitted patch do not do the same thing\. Two cited both a behavioral disagreement and an alteration of the given specification\. One, a single GPT\-5\.5 submission in the fourth repetition of row 7, was rejected on the specification alone: the patch and the artifact agreed, but the artifact no longer contained the contract the solver had been handed\. That is one submission in1,8951\{,\}895adjudicated row\-7 submissions, and[Table31](https://arxiv.org/html/2609.21190#A7.T31)below measures the same question mechanically over all of them\.

Dissent is slightly more common than rejection, because a lone dissenting ballot is overridden\. There were120120rejecting ballots against3434rejections, and in two runs the panel recorded rejecting ballots but no rejection at all\.

No ballot in any run abstained, so every adjudicated submission carries a confirmation or a refutation\.

Table 24:The equivalence judge, one row per repetition\. “Judged” is the number of episodes that reached the judge, out of500500; the rest failed verification, resolution, or the anti\-fakery check and were recorded as non\-equivalent without a ballot\. “Conf\.” is the share of judged episodes the majority accepted\. The grounds columns classify each rejection as citing a behavioral disagreement between patch and artifact, an alteration of the given specification, or both\. The ballot columns are the individual votes underlying the majorities; no ballot in any run abstained, so an abstention column would be zero throughout and is omitted\.
##### What the five\-property panel looks like from the inside\.

The row\-8 audit asks five questions of a synthesized specification: that its axioms hold of the real callees, that it is satisfiable by some implementation, that it is sound with respect to the reference behavior, that it excludes the original bug, and that it is a faithful reading of the issue\. Each question is answered by three independent ballots; the specification passes only if a majority accepts all five\.

Two properties of the panel matter for reading its verdicts\. First, every ballot is grounded in machine evidence\. A judge has two tools: it can submit a candidate implementation to the prover against the specification under review, and it can probe a claimed axiom against the real callee\. Across the11,97011\{,\}970decisive ballots cast in row 8, all but four issued at least one tool call and received at least one real machine signal, so almost no verdict rests on the prose alone\. A ballot made4\.04\.0tool calls on average, and96%96\\%to98%98\\%of ballots saw both an accepting and a rejecting prover verdict during their deliberation\. Recomputing every record’s verdict using only tool\-grounded ballots changes nothing, because there are no others\.

Second, the panel is mostly unanimous but not degenerately so\. Of all row\-8 records,82%82\\%to83%83\\%were unanimous in one direction or the other and17%17\\%split22–11; the residual is a handful of records with an abstention\. A majority therefore overrode a dissent on roughly one specification in six\. Mean self\-reported confidence was0\.850\.85on accepting ballots and0\.860\.86on rejecting ones\. The panel is not more confident when it rejects\.

Table 25:Behavior of the two audit panels\. “Row 8” is the panel that scores specification synthesis during evaluation; “row 3 re\-audit” is the same procedure applied afterwards, by a stronger judge with no access to the original verdicts, to the specifications solvers wrote in row 3\. Ballots are counted over all repetitions; a record is one specification that received at least one ballot\. “Grounded” is the share of decisive ballots that received at least one real machine signal\. “Unanimous” and “split” are shares of*all*records, so they sum to less than one wherever some record carries an abstention\.
##### Agreement between the three votes\.

[Table26](https://arxiv.org/html/2609.21190#A7.T26)reports inter\-judge agreement per property\. The headline is that agreement is high on the property that decides most verdicts and lower on the properties that almost never fail\. Faithfulness reaches a Fleissκ\\kappaof0\.770\.77in the row\-8 panel and0\.830\.83to0\.860\.86in the re\-audit; the overall verdict reaches0\.760\.76and0\.840\.84to0\.870\.87\. Completeness, in the two row\-8 panels, reaches only0\.430\.43and0\.540\.54, but its raw pairwise agreement there is94%94\\%and96%96\\%, higher than faithfulness’s89%89\\%\. The apparent contradiction is the standard prevalence effect: when95%95\\%of ratings fall on one side, chance agreement is already high andκ\\kappapunishes the remainder severely\. Both columns are therefore reported, and no claim here rests on theκ\\kappaof a property whose prevalence exceeds0\.90\.9\.

Table 26:Inter\-judge agreement across the three independent ballots, by property and panel\.κ\\kappais Fleiss’κ\\kappaandα\\alphaKrippendorff’sα\\alpha; “pairwise” is the raw share of agreeing judge pairs and “unanim\.” the share of items on which all three agree\. “Prev\.” is the share of individual ratings that voted the property satisfied; readκ\\kappaagainst it, since agreement coefficients are depressed by extreme prevalence even at very high raw agreement\. “Overall” is the ballot’s accept/reject verdict across all five properties\.The re\-audit agrees with itself substantially better than the row\-8 panel does:κ\\kappaon the overall verdict rises from0\.760\.76to0\.840\.84–0\.870\.87, and all three ballots agree on all five properties on74%74\\%–76%76\\%of specifications against67%67\\%–72%72\\%\. It also works harder, making6\.96\.9to7\.17\.1tool calls per ballot against4\.04\.0, and it abstains more often, on2\.3%2\.3\\%to3\.1%3\.1\\%of ballots against under0\.1%0\.1\\%\. It also has far more ungrounded ballots than the row\-8 panel:394394of Opus 4\.8’s5,8645\{,\}864decisive ballots and588588of GPT\-5\.5’s5,8175\{,\}817reached a verdict without a usable machine signal\. Those ballots are worse ballots\. Their mean self\-reported confidence is0\.650\.65against0\.830\.83, and they lean towards rejection, four to one\. Recomputing every record’s verdict from tool\-grounded ballots only leaves1,9431\{,\}943of2,0002\{,\}000Opus 4\.8 verdicts and1,8981\{,\}898of2,0002\{,\}000GPT\-5\.5 verdicts unchanged, flips1414and2525, and leaves2828and5555with no majority at all\. The re\-audit’s rejection rates would move by under a point and a half either way\. They are quoted as recorded\.

##### Failures concentrate on faithfulness, and the concentration is structural\.

[Table27](https://arxiv.org/html/2609.21190#A7.T27)decomposes which properties fail together\. Across the pooled row\-8 records,50%50\\%of Opus 4\.8’s specifications and38%38\\%of GPT\-5\.5’s fail at least one property\. Faithfulness is implicated in95%95\\%and97%97\\%of those failures, and it is the*only*failing property in56%56\\%and59%59\\%of them\. The converse also holds and is the more informative direction: every single specification that fails completeness also fails faithfulness, as does96%96\\%of those failing soundness and three quarters to four fifths of those failing axiom soundness or admissibility\. Only4747of Opus 4\.8’s1,0021\{,\}002failing specifications, and2424of GPT\-5\.5’s749749, fail something without failing faithfulness\.

This is not a redundant panel\. A specification can be sound, admissible, complete, and axiomatically honest and still fail faithfulness\. That happens often: conditional on passing the other four properties, faithfulness still fails on36%36\\%of Opus 4\.8’s specifications and26%26\\%of GPT\-5\.5’s\. What the co\-occurrence says is that the other four properties are close to necessary conditions for faithfulness rather than independent hurdles: a specification that mis\-states what the callees do, or that admits an implementation the reference would reject, is almost never a faithful reading of the issue either\.

Table 27:Co\-occurrence of failing properties in row 8, pooled over the four repetitions\. The upper block gives the marginals; the lower block gives, for each non\-faithfulness property, how often a specification that fails it also fails faithfulness\. “Faithfulness only” is the share of*all*audited specifications whose sole failure is faithfulness\.
##### A blind re\-audit of specifications that were never audited\.

Row 3 requires a solver to write a specification, prove an implementation against it, and submit a patch\. Its pass criterion never asks whether the specification says anything useful\. That makes row 3 a natural test of the audit itself: the specifications were written by agents that had no reason to expect them to be read, and the audit’s verdicts cannot have influenced the outcome, because the outcome was already recorded\. We re\-audited all4,0004\{,\}000of them with a stronger judge given no access to any earlier verdict\.

The re\-audit rejects63\.6%63\.6\\%of Opus 4\.8’s row\-3 specifications and57\.5%57\.5\\%of GPT\-5\.5’s\. For comparison, the row\-8 panel rejects49\.2%49\.2\\%–52\.0%52\.0\\%of Opus 4\.8’s synthesized specifications and34\.2%34\.2\\%–40\.2%40\.2\\%of GPT\-5\.5’s\. Both figures stand in sharp contrast to the near\-zero rate at which the corresponding construction check finds a defect in the released corpus \([SectionE\.3](https://arxiv.org/html/2609.21190#A5.SS3)\): the same procedure, pointed at specifications built without the construction pipeline, rejects between a third and two thirds of what it is shown\.

##### What the audit’s verdicts predict about an outcome they never see\.

The re\-audit judge sees the specification, the issue, and a prover\. It does not see the patch, the hidden tests, or whether the episode resolved the instance\.[Table28](https://arxiv.org/html/2609.21190#A7.T28)reports how much its verdicts predict about that unseen outcome\.

Table 28:Association between failing the blind re\-audit and failing to resolve the instance, over the row\-3 repetitions\. Each repetition is a2×22\\times 2table on493493–500500instances; the Mantel–Haenszel estimate combines them without assuming the four are independent samples \(the same instances recur\)\. The lower block decomposes the association by property\.The association is large and consistent: an odds ratio near1414for both models, with every individual repetition between88and2525\. Reading it in the direction the judge cannot see,94\.9%94\.9\\%of the episodes that failed to resolve had written a specification the audit rejects, against58\.1%58\.1\\%of those that resolved\. The per\-property decomposition puts almost all of the association on the same two properties that dominate the failure counts: faithfulness at an odds ratio near1313and soundness near5\.65\.6\. Admissibility and completeness are close to11and not significant for at least one model\. A judge asked whether a specification pins down the right behavior is answering a question with real consequences downstream\.

The association could in principle be produced by a single latent variable, instance difficulty, rather than by any relationship between specification quality and outcome\. Two observations argue against that reading\. First, the audit is not a proxy for whether the patch worked: among the episodes that*did*resolve, the audit still rejects58%58\\%of Opus 4\.8’s specifications and48%48\\%of GPT\-5\.5’s\. A grader that merely tracked success would not reject half of the successes\. Second, the failure profile differs between the two groups in kind and not only in degree\. Among resolved episodes, the single most common outcome is a specification that passes all five properties \(711711of1,6891\{,\}689for Opus 4\.8\); among unresolved episodes, that outcome occurs1515times out of296296, and the modal profile is a faithfulness failure with or without a soundness failure alongside it\.

[Table30](https://arxiv.org/html/2609.21190#A7.T30)makes the same point property by property, and shows that the association is not spread evenly across the panel;[Figure9](https://arxiv.org/html/2609.21190#A7.F9)draws it beside the row\-8 panel\. Two properties separate the two outcome groups sharply: soundness, which fails on9%9\\%of resolving Opus 4\.8 specifications and37%37\\%of non\-resolving ones, and faithfulness, which fails on56%56\\%against94%94\\%\. Two barely separate them at all: admissibility moves by two points and completeness by three\. Axiom soundness moves by less than a point for Opus 4\.8 and in the wrong direction\. This is what one would expect of a panel whose properties test different things: the two properties that ask whether the specification describes the required behavior track whether the episode produced that behavior, and the two that ask whether the specification is internally well\-formed do not\.

Table 29:Specification correctness against patch correctness, by verification backend \(% of specifications failing each property\)\. The five\-property audit of[Section4\.5](https://arxiv.org/html/2609.21190#S4.SS5)is re\-run on the specifications the end\-to\-end setting constructs for itself, blind to the agent’s patch and its outcome, and split by whether the episode resolved the instance \(Res\.\) or not \(Unres\.\)\.Table 30:Per\-property rejection rate of the blind row\-3 re\-audit, split by whether the episode resolved the instance, an outcome the judge never sees\. Entries are the mean over the four repetitions with the spread across them; “gap” is unresolved minus resolved in percentage points\. Recomputing every entry over tool\-grounded ballots only, and dropping the instances left without one, moves no entry by more than1\.81\.8points and changes no sign\.Figure 9:Where audited specifications fail, by property\.Left:the row\-8 panel, which scores a specification the agent wrote during its own episode\. Bars are the mean rejection rate over the four repetitions and whiskers the spread across them; the marginals are those of[Table27](https://arxiv.org/html/2609.21190#A7.T27)\.Right:the blind re\-audit of row\-3 specifications \([Table30](https://arxiv.org/html/2609.21190#A7.T30)\), with each property’s rate split by whether the accompanying patch resolved the instance, an outcome the judge never sees\. The vertical distance inside a pair is the property’s discriminative gap: soundness and faithfulness open wide, admissibility and completeness barely move\.The two models fail differently, and the difference is invisible in their nearly identical resolve rates\. GPT\-5\.5 writes specifications that fail soundness more often \(22\.5%22\.5\\%against13\.3%13\.3\\%\) and faithfulness less often \(54\.8%54\.8\\%against61\.9%61\.9\\%\); Opus 4\.8 writes specifications that misstate an axiom about a callee nearly four times as often \(7\.5%7\.5\\%against2\.0%2\.0\\%\)\. The aggregate rejection rates of the two models differ by six points, but the failures behind them are not the same failures, and a benchmark that reported only the rate would not show it\.

##### Specifications with no ballot\.

In row 8, between zero and three specifications per repetition were empty: the agent submitted a verifying witness and no specification, so no ballot could be cast\. Those instances fail the audit and are counted as failures, but flag no individual property\. In the re\-audit, the per\-property denominators are493493to500500and are stated in[Table28](https://arxiv.org/html/2609.21190#A7.T28)\.

##### Specification integrity in row 7\.

Rows 6 and 7 hand the agent a specification\. Row 7 additionally requires it to prove an implementation against that specification, which creates an obvious temptation: weaken the specification until the proof goes through\. We measured what agents actually did to the text they were given\.

Verbatim preservation is close to zero\. Comparing each submission against the exact text handed to the solver, only66to1010of500500submissions per repetition are byte\-identical for GPT\-5\.5, and00to11for Opus 4\.8\. A byte comparison counts reflowed docstrings, renamed local variables, reordered imports, and added explanatory comments as changes, so it says little about what the submission requires\.

Comparing the logical content instead gives the opposite picture\.[Table31](https://arxiv.org/html/2609.21190#A7.T31)reports a clause\-level comparison over the same submissions: every precondition, postcondition, invariant, and termination obligation in the supplied specification, matched against the submission\. Between99\.8%99\.8\\%and100\.0%100\.0\\%of the given clauses survive\. No repetition dropped a single termination obligation, the clause whose removal would most cheaply make a proof succeed\. Between00and1515clauses per7,0797\{,\}079went missing, and between00and1818were added\. In four of the eight repetitions, the number of submissions that changed a clause*and*still verified*and*still resolved is zero; in the others it is one or two\.

Table 31:What row\-7 submissions did to the specification they were handed\. “Verbatim” is byte\-identical to the supplied text\. “Clauses identical” is the share of submissions whose set of preconditions, postconditions, invariants, and termination obligations matches the supplied set exactly\. “Cosmetic only” additionally allows reflowing, comments, and renamed locals\. “Retained” is the share of the7,0797\{,\}079supplied clauses that survive\. “New helpers” counts contracted helper functions the agent added, which do not weaken anything supplied\. “Term\. dropped” counts supplied termination obligations that went missing, and “Changed & passed” is the number of submissions that altered a supplied clause and nevertheless verified and resolved\.Two consequences follow\. First, row 7 measures what it is meant to measure: agents carry the supplied specification to a verified implementation rather than negotiating it down\. Second, a verbatim\-equality test is the wrong instrument for this question, and the equivalence judge agrees\. Asked separately whether the specification was preserved, the judge’s ballots passed470470to479479of500500submissions per repetition, close to the clause\-level result and nowhere near the verbatim one\. A scoring rule built on byte equality would have rejected almost every correct submission in this setting\.

The released specifications have been revised since these runs:449449of the500500have a specification signature that differs from the run\-time text today, and447447of those differ in at least one clause\. The comparison above is against the run\-time text, which is what the solver saw\.

#### G\.4Repetitions

[Table2](https://arxiv.org/html/2609.21190#S4.T2)reports a single repetition per cell\. EveryNaginicell onSWE\-bench Verifiedwas run four times with independent sampling, apart from rows 1 and 5\. This subsection reports all four so that the main\-text number can be read against its own variability\. The other backends were run once, and their columns are left empty\.

##### The main\-text repetition is a typical one\.

[Table32](https://arxiv.org/html/2609.21190#A7.T32)gives the four rates per cell\. The largest gap between the main\-text repetition and the four\-repetition mean anywhere in the grid is2\.42\.4points, in row 8 under GPT\-5\.5, which is also the cell with the largest spread\. Every other cell is within1\.51\.5points\. In all fourteen cells, the four\-repetition mean falls inside the95%95\\%Wilson interval of the main\-text rate, and the main\-text rate falls inside the Student\-ttinterval on the mean\. The repetition\-to\-repetition standard deviation is at most1\.31\.3points in every cell except row 8 under GPT\-5\.5, where it is2\.62\.6\.

That variability should be read against the sampling uncertainty of a single500500\-instance run\. The95%95\\%Wilson half\-width atn=500n=500runs from1\.661\.66points at a rate of96\.4%96\.4\\%to4\.374\.37points at49\.2%49\.2\\%\. In every cell the sampling half\-width exceeds the repetition standard deviation by a factor of two to four\. Repeating a cell four times buys less precision than the finite corpus already costs\.[Figure10](https://arxiv.org/html/2609.21190#A7.F10)shows the two quantities on one axis\.

Table 32:Four independent repetitions of eachNaginicell onSWE\-bench Verified,n=500n=500per repetition\. The criterion is each setting’s own pass criterion, the one the main text reports\. “r1r\_\{1\}” is the repetition reported in[Table2](https://arxiv.org/html/2609.21190#S4.T2)\. “σ\\sigma” is the sample standard deviation over the four\. “Any” is the share of instances solved by at least one repetition and “all” by every repetition\. “Wilson” is the95%95\\%half\-width onr1r\_\{1\}\. Cells for the other three backends were run once and are left blank\.RepetitionSettingModelr1r\_\{1\}r2r\_\{2\}r3r\_\{3\}r4r\_\{4\}meanσ\\sigmarangeanyallWilson0 Unaided baselineOpus 4\.885\.086\.084\.484\.885\.050\.681\.689\.879\.0±3\.13\\pm 3\.13GPT\-5\.581\.281\.279\.680\.080\.500\.821\.685\.075\.6±3\.42\\pm 3\.422 End\-to\-EndOpus 4\.884\.284\.884\.685\.684\.800\.591\.490\.079\.0±3\.20\\pm 3\.20GPT\-5\.580\.879\.479\.879\.079\.750\.771\.885\.075\.0±3\.45\\pm 3\.453 Verified End\-to\-EndOpus 4\.885\.284\.683\.084\.684\.350\.942\.290\.676\.6±3\.11\\pm 3\.11GPT\-5\.579\.278\.478\.479\.678\.900\.601\.284\.272\.6±3\.55\\pm 3\.554 Localization ProvidedOpus 4\.891\.493\.091\.893\.092\.300\.821\.696\.286\.4±2\.47\\pm 2\.47GPT\-5\.585\.487\.286\.687\.086\.550\.811\.890\.681\.2±3\.10\\pm 3\.106 Specification ProvidedOpus 4\.896\.496\.496\.896\.296\.450\.250\.698\.693\.0±1\.66\\pm 1\.66GPT\-5\.595\.295\.495\.495\.495\.350\.100\.297\.293\.0±1\.90\\pm 1\.907 Verified from Spec\.Opus 4\.895\.495\.895\.495\.295\.450\.250\.698\.091\.8±1\.86\\pm 1\.86GPT\-5\.594\.095\.694\.094\.094\.400\.801\.697\.690\.0±2\.10\\pm 2\.108 Specification SynthesisOpus 4\.849\.250\.448\.050\.849\.601\.262\.872\.223\.6±4\.37\\pm 4\.37GPT\-5\.559\.865\.862\.061\.262\.202\.576\.079\.441\.4±4\.28\\pm 4\.28Rows 1 and 5, and allVelvet,Lean, andEarscells: one repetition, no spread available\.Figure 10:The repetitions of[Table32](https://arxiv.org/html/2609.21190#A7.T32), one pair of rows per setting\. The filled marker is the repetition the main text reports, the three open markers are the others, and the short rule is their mean\. The gray band behind each row is the95%95\\%Wilson interval of the reported repetition atn=500n=500, the sampling uncertainty of a single run of that size\. In thirteen of the fourteen cells all four repetitions fall inside that band: the spread between repetitions is smaller than the uncertainty any one of them already carries\. The exception is specification synthesis under GPT\-5\.5, whose best repetition \(65\.865\.8\) sits just outside the interval of the reported one\.
##### Stability is a property of the instance, not of the rate\.

The aggregate rates are stable to within a point; individual instances are not\.[Table33](https://arxiv.org/html/2609.21190#A7.T33)classifies each instance by how many of the four repetitions solved it\. In the settings where a specification is supplied,4%4\\%to6%6\\%of instances are unstable: solved by one, two, or three repetitions but not all four\. In the baseline and end\-to\-end settings the figure is9%9\\%to14%14\\%\. In row 8 it is38%38\\%to49%49\\%\.

That last number reframes what row 8 measures\. Opus 4\.8 passes the audit on49\.6%49\.6\\%of instances on average, but that average decomposes into24%24\\%of instances it passes every time,28%28\\%it fails every time, and49%49\\%where the outcome depends on the sample\. Writing a faithful specification is not a capability the model either has or lacks per instance; on half the corpus it is a coin whose bias sits somewhere in the middle\. The gap between the union over four repetitions \(72\.2%72\.2\\%\) and the intersection \(23\.6%23\.6\\%\) is48\.648\.6points, five times the corresponding gap in row 6\.

Table 33:Instance\-level stability over four repetitions, criterion = each setting’s own pass criterion\.kkis the number of repetitions that succeeded on the instance; “unstable” is1≤k≤31\\leq k\\leq 3\.HHis the Shannon entropy in bits of thekk\-histogram, a single summary of how far the cell is from deterministic\. “Any” and “all” are counts out of500500\.
##### The main\-text contrasts hold in every repetition\.

[Table34](https://arxiv.org/html/2609.21190#A7.T34)repeats the four contrasts[Section4](https://arxiv.org/html/2609.21190#S4)draws, once per repetition, with a paired test on the four differences and an exact McNemar test on the instance\-level pairing of the main\-text repetition\. All three positive contrasts have the same sign in all four repetitions and a repetition\-level standard deviation under one point\. The null contrast, adding a verifier and a specification obligation to a bare end\-to\-end setting without supplying anything, has an inconsistent sign across repetitions and a pairedppof0\.620\.62and0\.180\.18\. It is genuinely null rather than small\.

Table 34:The main\-text contrasts, per repetition\. Each row is a pair of settings compared on the same instances\. “r1r\_\{1\}” is the main\-text difference in percentage points; “mean±σ\\pm\\sigma” is over the four paired repetition\-level differences; “pairedpp” is the two\-sidedppof att\-test on those four differences \(d​f=3df=3\); “McNemarpp” is the exact two\-sided test on the main\-text repetition’s instance pairing \(n=500n=500\), withb/cb/cthe discordant counts\.
##### An integrity check across the repetitions\.

The two criteria a setting can be scored on are nested: a pass under a verified setting requires the patch to resolve the instance, so the set of passing instances must be a subset of the set of resolving ones\. Across all5656completeNaginirepetitions there is no instance that passes without resolving\. In rows 6, 2, 4, and 0 the two sets are identical by construction\. In rows 7 and 3 the difference is the verification and equivalence loss: one to five instances per repetition resolve the hidden tests but do not clear verification and equivalence, and are therefore not counted\. Row 8 does not grade the patch at all, so its resolution figures are informational; the two models make different and ungraded choices about whether to leave a patch behind, and the row\-8 resolution column is not comparable between them\.

#### G\.5Outcomes by instance property

The main text reports one gain: supplying a specification raises the resolution rate by1111to1414points\. That aggregate hides a factor of five\. This subsection asks where the gain comes from, using structural properties of the reference fix as strata\. All figures use the four repetitions of theNaginicells onSWE\-bench Verified\.

Table 35:Structural properties of the500500SWE\-bench Verifiedinstances, as measured on the reference fix and the reference bundle\. “Modeled functions” is the number of functions the specification constrains; “axioms” is the number of assumptions the bundle states about callees it does not model\.Propertymeanp25medianp75maxAdded lines in the reference fix9\.942410202Deleted lines4\.3912489Added\+\+deleted lines14\.333713232Files touched1\.2511121Hunks2\.4411245Modeled functions1\.411127Axioms0\.530018Bundles carrying at least one axiom27%––––Fail\-to\-pass tests3\.03112438Pass\-to\-pass tests120\.281950\.51112476Reference verification time \(s\)25\.8413\.918\.429\.5288\.6Issue text length \(characters\)17006441185203624770##### What the corpus looks like\.

[Table35](https://arxiv.org/html/2609.21190#A7.T35)records the distributions the strata are cut from\. Three features shape everything that follows\. The reference fixes are small: the median adds four lines and touches one file\. The specifications are local: the median bundle constrains a single function and states no axiom at all\. And two of the distributions have extreme tails: one instance carries438438fail\-to\-pass tests against a median of11, another touches2121files, another adds202202lines\. Any linear statistic on those columns is dominated by a handful of instances, so the rank\-based figures below are the ones to read\.

Table 36:Resolution rate by stratum, four\-repetition means\. “00” is the unaided baseline, “44” localization provided, “66” specification provided, “22” end\-to\-end, “88” specification synthesis \(a pass rate on the audit, not a resolution rate\)\. “Gain” is row 6 minus row 0 pooled over both models\. Strata withn<20n<20are shown but excluded from the ranking discussion; three strata belown=5n=5are omitted, so a family’s counts need not sum to500500\.Opus 4\.8GPT\-5\.5Stratumnn0462804628GainAll instances50085\.092\.396\.484\.849\.680\.586\.595\.379\.862\.2\+13\.1\+13\.1*Repository*django23186\.793\.697\.286\.050\.585\.690\.897\.284\.868\.9\+11\.0\+11\.0sympy7580\.389\.397\.082\.044\.370\.377\.795\.770\.753\.0\+21\.0\+21\.0sphinx4486\.493\.894\.984\.743\.876\.185\.292\.670\.557\.4\+12\.5\+12\.5matplotlib3480\.191\.2100\.077\.955\.976\.589\.0100\.077\.262\.5\+21\.7\+21\.7scikit\-learn3296\.196\.999\.296\.151\.689\.189\.196\.991\.457\.0\+5\.5\+5\.5astropy2273\.985\.293\.273\.935\.273\.983\.089\.875\.033\.0\+17\.6\+17\.6xarray2286\.492\.093\.288\.660\.283\.087\.593\.278\.464\.8\+8\.5\+8\.5pytest1992\.193\.494\.794\.768\.482\.984\.289\.584\.278\.9\+4\.6\+4\.6pylint1057\.575\.075\.055\.030\.045\.052\.565\.047\.545\.0\+18\.7\+18\.7requests890\.6100\.0100\.093\.850\.0100\.0100\.0100\.090\.665\.6\+4\.7\+4\.7*Added lines in the reference fix*≤2\\leq 216789\.494\.298\.890\.454\.088\.092\.499\.688\.068\.4\+10\.5\+10\.522–448487\.294\.698\.287\.556\.886\.090\.297\.386\.069\.6\+11\.2\+11\.244–101013288\.493\.698\.186\.750\.280\.787\.795\.180\.959\.5\+12\.0\+12\.0\>10\>1011773\.586\.590\.072\.637\.465\.674\.488\.262\.251\.1\+19\.6\+19\.6*Files touched*11file42988\.193\.398\.188\.152\.584\.789\.397\.584\.464\.7\+11\.4\+11\.4\>1\>1file7166\.586\.386\.365\.132\.055\.370\.182\.451\.447\.2\+23\.4\+23\.4*Modeled functions*single\-site35589\.793\.297\.689\.952\.885\.488\.496\.284\.865\.9\+9\.4\+9\.4multi\-site14573\.690\.093\.672\.441\.768\.682\.193\.367\.453\.1\+22\.3\+22\.3*Axioms*none36383\.791\.496\.182\.949\.779\.685\.895\.678\.662\.1\+14\.2\+14\.2≥1\\geq 113788\.794\.797\.490\.049\.582\.888\.594\.782\.862\.4\+10\.3\+10\.3*Fail\-to\-pass tests*1134588\.994\.298\.089\.354\.086\.791\.497\.685\.765\.3\+10\.0\+10\.0229578\.990\.595\.378\.942\.170\.878\.291\.371\.856\.8\+18\.4\+18\.433–553570\.783\.694\.366\.432\.955\.773\.692\.955\.053\.6\+30\.4\+30\.4≥6\\geq 62575\.085\.082\.071\.041\.067\.070\.083\.062\.052\.0\+11\.5\+11\.5*Reference verification time*≤13\.9\\leq 13\.9s13186\.191\.495\.686\.554\.084\.988\.495\.484\.268\.3\+10\.0\+10\.013\.913\.9–18\.418\.4s11986\.191\.294\.185\.148\.978\.285\.793\.577\.960\.1\+11\.7\+11\.718\.418\.4–29\.529\.5s12584\.093\.498\.281\.849\.079\.487\.095\.078\.059\.0\+14\.9\+14\.9\>29\.5\>29\.5s12584\.093\.297\.885\.846\.279\.285\.097\.478\.661\.0\+16\.0\+16\.0*Estimated time to fix*<15<15min19493\.295\.999\.092\.059\.090\.693\.298\.890\.671\.6\+7\.0\+7\.01515min–11h26183\.292\.097\.383\.445\.578\.685\.595\.977\.558\.1\+15\.7\+15\.711–44h4261\.379\.882\.763\.134\.551\.267\.380\.449\.445\.8\+25\.3\+25\.3
##### The specification helps most where the task is least local\.

[Table36](https://arxiv.org/html/2609.21190#A7.T36)gives every stratum\. Ranked by the pooled gain from supplying a specification, and restricted to strata with at least2020instances, the top of the list is: instances with three to five fail\-to\-pass tests \(\+30\.4\+30\.4points\), the one\-to\-four\-hour difficulty band \(\+25\.3\+25\.3\), fixes touching more than one file \(\+23\.4\+23\.4\), specifications constraining more than one function \(\+22\.3\+22\.3\), matplotlib \(\+21\.7\+21\.7\), and sympy \(\+21\.0\+21\.0\)\. The bottom is scikit\-learn \(\+5\.5\+5\.5\) and xarray \(\+8\.5\+8\.5\)\.

Every entry at the top of that list is a form of non\-locality\. A fix spread over several files, a specification that has to constrain several functions, and a bug whose symptom shows up in several tests are all cases where the hard part is working out what the code has to do rather than writing it\. A specification states exactly that, and the strata where it helps most are the strata where the statement is hardest to reconstruct from the issue text\. The two low\-gain repositories are the opposite case: scikit\-learn’s instances in this corpus are short, single\-site, and already resolved96%96\\%of the time without help, leaving little room\.

The gain also grows monotonically with three graded properties: with the size of the reference fix \(\+10\.5\+10\.5,\+11\.2\+11\.2,\+12\.0\+12\.0,\+19\.6\+19\.6across quartiles of added lines\), with how long the reference bundle takes to verify \(\+10\.0\+10\.0to\+16\.0\+16\.0\), and with the estimated time to fix \(\+7\.0\+7\.0,\+15\.7\+15\.7,\+25\.3\+25\.3\)\. The relationship with verification time is the mildest of the three and, as noted below, the only one of the three that does not reach significance\.

One stratum runs the other way\. Bundles carrying at least one axiom \(specifications that had to assume something about a callee rather than model it\) gain*less*from being supplied \(\+10\.3\+10\.3against\+14\.2\+14\.2\), and the difference is not significant\. Axioms mark a specification that could not be made fully self\-contained; they do not mark a harder task\.

##### Which stratum differences survive a test\.

Comparing raw rates across strata compares different instance sets, so we test on a per\-instance quantity instead: for each instance, the number of the four repetitions that resolved it with a specification minus the number that resolved it without, divided by four\. That is a within\-instance difference, and it can be compared across strata with a Kruskal–Wallis test\.[Table37](https://arxiv.org/html/2609.21190#A7.T37)reports the results\.

Table 37:Does the specification gain differ across the strata of a family? Kruskal–Wallis over the per\-instance gain, restricted to strata withn≥20n\\geq 20; for two\-stratum families the Mann–Whitney and Welch tests are reported as well\. “Pooled” combines both models; the per\-model columns give the same test on each model’s four repetitions separately\.Four families separate clearly: the number of functions the specification constrains, the number of files the fix touches, the number of failing tests, and the estimated time to fix\. Two do not: whether the bundle carries an axiom \(p=0\.38p=0\.38\) and how long the reference takes to verify \(p=0\.17p=0\.17\)\. The repository family is borderline pooled \(p=0\.066p=0\.066\) and significant for GPT\-5\.5 alone \(p=0\.022p=0\.022\), which is what one would expect if repository identity were mostly a proxy for the structural properties that do separate\. For the two\-stratum families the effect sizes are large and stable: the multi\-site stratum gains12\.9612\.96points more than the single\-site stratum \(Welcht=−4\.04t=\-4\.04,p=7\.6×10−5p=7\.6\\times 10^\{\-5\}\), and the multi\-file stratum11\.9911\.99points more than the single\-file stratum \(t=−2\.74t=\-2\.74,p=0\.0075p=0\.0075\)\.

##### How much of the gain is localization?

Row 6 supplies both a specification and a localization, so part of its advantage over the baseline is simply knowing where to look\. Row 4 supplies the localization alone, and the difference between rows 6 and 4 isolates the part attributable to the specification itself\. Pooled, that residual is\+6\.5\+6\.5points, about half of the\+13\.1\+13\.1total\. Reading it stratum by stratum changes the picture from the one the total gain suggests\.[Figure11](https://arxiv.org/html/2609.21190#A7.F11)plots both quantities together\.

Two of the strata with the largest total gain keep almost none of it once localization is supplied\. Fixes touching more than one file gain\+23\.4\+23\.4points over the unaided baseline, the third largest total in the corpus; over row 4 they gain\+6\.2\+6\.2, which is slightly*less*than the\+6\.5\+6\.5single\-file stratum\. Specifications constraining more than one function gain\+22\.3\+22\.3over the baseline and\+7\.4\+7\.4over row 4, against\+6\.1\+6\.1for the single\-function stratum\. For these two families \(the two the Kruskal–Wallis test separated most strongly\), most of what the specification was worth was the localization it carried\. Those tests are about the total gain, and the stratum differences do not survive the decomposition\.

Other strata do keep their advantage\. Instances with three to five failing tests retain\+15\.0\+15\.0points, the largest residual in the corpus and more than twice the pooled figure\. sympy retains\+12\.8\+12\.8and matplotlib\+9\.9\+9\.9, against\+4\.3\+4\.3for sphinx and\+3\.4\+3\.4for xarray\. The largest quartile of reference fixes retains\+8\.7\+8\.7and the longest verifying quartile\+8\.5\+8\.5\. The one\-to\-four\-hour difficulty band retains\+8\.0\+8\.0against\+4\.4\+4\.4for the sub\-fifteen\-minute band\. So the residual is largest where the*behavior*is hard to pin down \(many failing tests, a large fix, a repository of dense mathematical code\) and smallest where the task was already easy\. What a specification adds beyond a pointer is a statement of the required behavior, and it is worth most where that statement is hardest to reconstruct\.

The residual is also much more model\-dependent than the total\. Pooled it is\+4\.1\+4\.1for Opus 4\.8 and\+8\.8\+8\.8for GPT\-5\.5, and the split widens on exactly the non\-local strata: on multi\-file instances Opus 4\.8 gains nothing at all over row 4 \(\+0\.0\+0\.0\) while GPT\-5\.5 gains\+12\.3\+12\.3; on instances with six or more failing tests Opus 4\.8 does3\.03\.0points*worse*with the specification than with the localization alone, while GPT\-5\.5 does13\.013\.0points better\. Given a localization, the stronger model can often reconstruct the required behavior on its own; the weaker one cannot, and the specification is where the difference goes\. Any single\-model reading of how much a specification is worth is therefore a reading about that model\.

Figure 11:What a supplied specification is worth, stratum by stratum, pooled over both models and averaged over the four repetitions\. For each stratum one marker gives the gain of Specification Provided \(row 6\) over the unaided baseline \(row 0\), and the other gives the part of that gain which survives when the comparison is Localization Provided \(row 4\) instead; the segment between them is what the localization the specification carries was worth on its own\. The dashed rules are the pooled values,\+13\.1\+13\.1and\+6\.5\+6\.5points\. Strata with fewer than2020instances are omitted, andnnis the stratum size\. Two of the largest total gains keep the least: fixes touching more than one file gain\+23\.4\+23\.4over the baseline and\+6\.2\+6\.2over row 4, and specifications constraining more than one function gain\+22\.3\+22\.3and\+7\.4\+7\.4\. The largest residuals sit elsewhere: on instances with three to five failing tests, and in sympy and matplotlib\.
##### Correlates of baseline difficulty\.

[Table38](https://arxiv.org/html/2609.21190#A7.T38)correlates the same properties against the unaided baseline outcome\. Six properties are significant after a Holm correction for both models: the number of files touched, the estimated time to fix, added lines, added plus deleted lines, the number of modeled functions, and the number of hunks\. All six have negative sign; all six are measures of how spread out the fix is\. Three are not significant for either model: whether the bundle carries an axiom, the number of pass\-to\-pass tests, and the reference verification time\. How long the reference bundle takes to prove says nothing about how hard the instance is for an agent to fix\. The prover’s difficulty and the task’s difficulty are separate axes\.

The fail\-to\-pass count is the instructive case for methodology\. Its point\-biserial correlation with the baseline outcome is\+0\.008\+0\.008for Opus 4\.8 and\+0\.005\+0\.005for GPT\-5\.5 \(indistinguishable from zero\), while its Spearman correlation is−0\.18\-0\.18and−0\.25\-0\.25, comfortably significant\. The discrepancy is entirely due to two instances that carry438438and168168failing tests against a median of11, and that both models resolve\. A linear statistic is dominated by those two points; a rank statistic is not\. We report both columns and read the rank one\.

Table 38:Correlation of instance properties with the unaided baseline outcome,n=500n=500, main\-text repetition\.rp​br\_\{pb\}is the point\-biserial correlation with the binary outcome andρ\\rhois Spearman’s\. “Holm” is the Holm\-correctedpp\-value ofrp​br\_\{pb\}across the properties of that column\. “mean∣\\mid1” and “mean∣\\mid0” are the property’s mean on resolved and unresolved instances\. Readρ\\rhorather thanrp​br\_\{pb\}for the heavy\-tailed count properties\.Instances whose bundle carries at least one axiom are resolved at an odds ratio of1\.471\.47relative to those that do not, with a Fisherppof0\.260\.26for Opus 4\.8 and1\.131\.13withp=0\.70p=0\.70for GPT\-5\.5\. Carrying an axiom is uncorrelated with whether the instance gets fixed, in both directions, for both models\.

##### What fails in specification synthesis\.

Row 8 is the weakest cell in the benchmark\. A row\-8 pass requires that the agent’s witness verify against its own specification, that the specification not be vacuously satisfiable, and that it pass the five\-property audit\.[Table39](https://arxiv.org/html/2609.21190#A7.T39)decomposes all eight repetitions\.

Table 39:What fails in specification synthesis, per repetition,n=500n=500\. The three conjuncts of a pass are shown separately\. “Resolved” is informational: row 8 does not grade a patch, and the two models make different ungraded choices about whether to leave one behind, so this column is not comparable between models\.The prover is not the obstacle\. Across all eight repetitions, the agent’s witness verified against its own specification in497497to500500of500500episodes, and the non\-vacuity check failed once in4,0004\{,\}000\. Every remaining failure is the audit, and within the audit, faithfulness\. Agents can write specifications they can prove; what they cannot reliably write is specifications that say the right thing\.

Row 8 is also the only setting where the structural properties lose much of their grip\. Correlating them against the row\-8 outcome instead of the baseline, only the estimated time to fix survives the Holm correction \(rp​b=−0\.195r\_\{pb\}=\-0\.195,p=1\.5×10−4p=1\.5\\times 10^\{\-4\}\); added lines is marginal \(rp​b=−0\.132r\_\{pb\}=\-0\.132,p=0\.037p=0\.037\) and the axiom flag is flat \(odds ratio0\.910\.91,p=0\.69p=0\.69\)\. Whatever makes an instance hard to specify faithfully is largely not what makes it hard to fix\.

##### Structured requirements against a prover\.

TheEarsbackend replaces the prover with a structured\-requirements document and a well\-formedness check\. It is the natural control for how much of the corpus difficulty comes from formality itself, and it produces the sharpest contrast in the appendix\.

In row 2, where neither backend’s artifact is graded and the score is the hidden tests, the two backends are indistinguishable\. Over the500500instances, Opus 4\.8 resolves421421underEarsand428428underNagini, with1717instances resolved only underEarsand2424only underNagini\(exact McNemarp=0\.35p=0\.35\)\. GPT\-5\.5 resolves407407underEarsand401401underNagini, with1616against1010discordant \(p=0\.33p=0\.33\)\. Writing a structured requirements document instead of a formal specification neither helps nor hurts an agent’s ability to fix the bug\.

Row 3 is where they part\. There the artifact must survive a check, and the checks are not comparable in strength\.[Table40](https://arxiv.org/html/2609.21190#A7.T40)reports the matched comparison\. UnderNaginithe check is a prover:420420of Opus 4\.8’s500500submissions pass\. UnderEarsthe check is an adversarial counterexample audit against a formal sibling specification, and286286pass:151151instances pass underNaginiand fail underEars, against1717the other way \(p=5×10−28p=5\\times 10^\{\-28\}\)\. The audit finds a genuine violation against32%32\\%of Opus 4\.8’s resolving submissions and60%60\\%of GPT\-5\.5’s\.

Table 40:The structured\-requirements backend againstNagini, instance by instance\. Rows 2 and 3 carry no localization, so the only difference between the two episodes is the artifact the agent must author and the tool in its verify slot\. The resolution columns are like\-for\-like; the pass columns are not, because the two backends’ pass criteria differ:Naginirequires a proof,Earsrequires surviving an adversarial counterexample audit\.That gap does not show that structured requirements are a weaker way to pin down behavior than a formal specification\. The counterexample audit is defined for every backend, but in this campaign no formal submission that a prover had already accepted was put through it, so there is no measurement of what the same audit would find againstNaginisubmissions\. What the comparison establishes is narrower\. A well\-formed structured requirements document is cheap: an agent produces one on its first attempt in almost every episode, and the check costs no prover time at all\. A cheap check accepts a great deal that a targeted adversary can then refute\. The documents themselves are short:2\.32\.3requirements and8\.28\.2acceptance criteria for Opus 4\.8,2\.82\.8and10\.310\.3for GPT\-5\.5, of which essentially all are conditionally triggered\.

The cost side of the same comparison is one\-directional and large\. On matched cells, anEarsepisode uses44to66fewer agent steps and9191to166166thousand fewer input tokens than theNaginiepisode of the same model and setting, and spends zero seconds in a solver against5050to132132seconds per episode underNagini\.

#### G\.6Interaction and cost

This subsection reports what the evaluation cost\. Tokens and agent steps are the comparable measures; wall\-clock is reported for completeness but is a property of the host, since twelve episodes ran concurrently throughout\.

##### Accounting\.

An episode’s input count is the sum of the prompt tokens of its individual model calls\. Prompt caching was not in effect in any episode of any run, so there is no distinction between billed and presented input\. The verifier wall\-clock is the sum over an episode’s verify invocations of the time the verifier held the call; forEarsit is0\.00\.0by rounding, because the structural check returns in well under a tenth of a second\. Every figure below is pooled over the complete repetitions of a cell only\.

Table 41:Cost per episode by setting, model, and backend, pooled over the complete repetitions of the cell\. Input and output are means in thousands of tokens; steps is mean agent steps; wall is mean episode seconds under twelve\-way concurrency\. “Verify calls” is the mean number of verify invocations\. “Input per resolved” is the cell’s total input tokens divided by the number of instances it resolved, in thousands: the cost of an outcome rather than of an episode\.Figure 12:What each setting costs and what it buys, contract\-annotated Python backend, four repetitions pooled\.Left:pass rate against input tokens per episode, one point per setting and model, labeled by row number; up and to the left is better\.Right:the two quantities combined as input tokens per accepted submission, which is the scale on which settings of different accuracy can be compared\. Localization Provided is the cheapest setting per accepted submission for both models, and requiring a verifying artifact costs about what the same setting costs without one\.
##### Two models, similar accuracy, different cost\.

[Table41](https://arxiv.org/html/2609.21190#A7.T41)shows GPT\-5\.5 spending between1\.21\.2and2\.92\.9times as many input tokens per episode as Opus 4\.8 in every comparable cell, and between1\.11\.1and2\.02\.0times as many agent steps, at resolution rates that differ by a few points\. The single largest gap is the unaided baseline, where GPT\-5\.5 uses572572thousand input tokens per episode against200200thousand and resolves3\.83\.8points fewer instances\. Per resolved instance the ratio is3\.03\.0to11\. The one cell where the ordering reverses is row 8: GPT\-5\.5 uses239239thousand tokens against Opus 4\.8’s335335thousand and passes the audit more often\.

##### Cost tracks the formal obligation, not the score\.

Ordering theNaginicells by input tokens per episode gives the same order as ordering them by how much the setting asks the agent to construct: localization provided \(140140–322322thousand\), unaided baseline \(200200–572572\), specification synthesis \(239239–335335\), verified from specification \(354354–438438\), specification provided \(376376–443443\), end\-to\-end \(438438–697697\)\. Supplying a specification is*cheaper*than not supplying one in every comparison: row 6 costs less than row 2 for both models, by6363and255255thousand tokens per episode, while resolving1111to1515more points of the corpus\. The cost of an outcome separates even more sharply:389389thousand input tokens per resolved instance in row 6 against517517in row 2 for Opus 4\.8, and464464against874874for GPT\-5\.5\.[Figure12](https://arxiv.org/html/2609.21190#A7.F12)shows both scales\.

Adding the verification obligation on top of a supplied specification is essentially free: rows 6 and 7 differ by under6%6\\%in tokens and under3%3\\%in steps for both models\. The agent was already using the verifier in row 6, where it had no obligation to; formalizing that obligation changes the score by about a point and the cost by almost nothing\.

Row 8 is where wall\-clock and tokens disagree most, and[Table21](https://arxiv.org/html/2609.21190#A7.T21)explains why\. It is the cheapestNaginisetting in tokens for GPT\-5\.5 and the second cheapest for Opus 4\.8, but the most expensive in wall\-clock by more than a factor of two \(921921and742742seconds per episode\)\. The difference is prover time: row 8 spends139139and5353compute\-hours in the verifier across its cells, more than any other setting\.

##### The step budget rarely binds\.

Every episode had a budget of250250agent steps\. Of the campaign’s53,50053\{,\}500episodes,53,46553\{,\}465ended by submitting and2525reached the budget\. The submission rate is100\.0%100\.0\\%in9494of the107107runs and above99\.5%99\.5\\%in all but one:Velvetunder Opus 4\.8 in row 3, where2121episodes ran out of steps while still working on a proof, which costs that cell about four points of resolution\. Elsewhere the step distributions of[Table21](https://arxiv.org/html/2609.21190#A7.T21)put the9999th percentile between6363and103103, so the reported rates are not truncated by the interaction limit\.

##### Repetition\-to\-repetition cost stability\.

Within a cell, the ratio of the largest to the smallest per\-repetition mean is1\.031\.03to1\.221\.22for input tokens,1\.021\.02to1\.131\.13for output tokens, and1\.011\.01to1\.061\.06for steps\. For wall\-clock it is1\.021\.02to1\.791\.79\. Token and step costs are a stable property of a cell; wall\-clock is not, because different repetitions ran under different host load\. Any comparison in this appendix that relies on wall\-clock is either within a single run or is stated as a ratio to that run’s own total\.

##### Campaign totals\.

[Table42](https://arxiv.org/html/2609.21190#A7.T42)reports the whole evaluation:107107runs and53,50053\{,\}500episodes, which consumed22\.622\.6billion input tokens and415415million output tokens over1\.361\.36million agent steps,167,719167\{,\}719verifier invocations,5,9375\{,\}937hours of episode wall\-clock, and896896hours of verifier wall\-clock\.

Table 42:Cost of the evaluation campaign\. Wall\-clock hours are compute\-hours summed over episodes, not elapsed time\.Two ratios follow from that table\. Of the167,719167\{,\}719verifier invocations,68,79368\{,\}793returned an accepting verdict,41%41\\%overall; the per\-cell spread behind that average is the backend\-difficulty measurement of[SectionG\.2](https://arxiv.org/html/2609.21190#A7.SS2)\. And the mean prover call held the prover for19\.619\.6seconds, against25\.825\.8seconds for a reference verification on the same corpus, so an agent’s proof obligation is no more expensive to discharge than the reference one\.

### Appendix HWorked examples

The preceding sections describe the corpora in aggregate\. This section does the opposite\. It follows a small number of instances all the way down, so that a reader can see what a specification actually says, what the evidence attached to it actually establishes, and where the construction is genuinely hard\.

Two examples carry the section, and the first two subsections share one of them\.[SectionH\.1](https://arxiv.org/html/2609.21190#A8.SS1)takes a single Django issue and prints the four specifications the four backends produce for it side by side, together with the admission evidence each one carries\.[SectionH\.2](https://arxiv.org/html/2609.21190#A8.SS2)stays on that instance and reproduces the full adversarial record for one of the four specifications: twenty candidate implementations written to break it, the mechanism by which each was rejected, the one that verified, and the controls that decide whether a candidate that verifies is a specification weakness or a legitimate alternative implementation\.[SectionH\.3](https://arxiv.org/html/2609.21190#A8.SS3)then turns to a scikit\-learn issue on which two of the properties we require of a specification cannot both hold, and states what we do about it\.

The instance in the first two subsections was chosen because it is small enough to print in full and unusual in what it demands of a specification: the reference fix is a single added line, all four backends admit the instance, and the specification that line has to satisfy is nonetheless the most intricate of the four\. It demonstrates that the difficulty of specifying a change is not proportional to the size of the change\.

#### H\.1One task, four specifications

##### The task\.

django\-11179reports an inconsistency in Django’s deletion machinery\. When an object is deleted through the ORM, the deletion code clears the object’s primary\-key attribute in memory, which is how an in\-memory object records that it no longer corresponds to a stored row\. The machinery also contains a fast path for the simplest possible job: one collected object, of one model, with no dependent relations\. That path issues the delete and returns immediately, skipping the bookkeeping that clears primary keys\. Deleting a dependency\-free object therefore left it holding a stale identifier, unlike every other deletion\. The public problem statement says exactly this and no more: it names the fast path, states that the primary key is not cleared on it, and asks for the reset\.

Box H\.1: The reference fix fordjango\-11179[⬇](data:text/plain;base64,LS0tIGEvZGphbmdvL2RiL21vZGVscy9kZWxldGlvbi5weQorKysgYi9kamFuZ28vZGIvbW9kZWxzL2RlbGV0aW9uLnB5CkBAIC0yNzcsNiArMjc3LDcgQEAgZGVmIGRlbGV0ZShzZWxmKToKICAgICAgICAgICAgIGlmIHNlbGYuY2FuX2Zhc3RfZGVsZXRlKGluc3RhbmNlKToKICAgICAgICAgICAgICAgICB3aXRoIHRyYW5zYWN0aW9uLm1hcmtfZm9yX3JvbGxiYWNrX29uX2Vycm9yKCk6CiAgICAgICAgICAgICAgICAgICAgIGNvdW50ID0gc3FsLkRlbGV0ZVF1ZXJ5KG1vZGVsKS5kZWxldGVfYmF0Y2goW2luc3RhbmNlLnBrXSwgc2VsZi51c2luZykKKyAgICAgICAgICAgICAgICBzZXRhdHRyKGluc3RhbmNlLCBtb2RlbC5fbWV0YS5way5hdHRuYW1lLCBOb25lKQogICAgICAgICAgICAgICAgIHJldHVybiBjb3VudCwge21vZGVsLl9tZXRhLmxhYmVsOiBjb3VudH0=)—a/django/db/models/deletion\.py\+\+\+b/django/db/models/deletion\.py@@\-277,6\+277,7@@defdelete\(self\):ifself\.can\_fast\_delete\(instance\):withtransaction\.mark\_for\_rollback\_on\_error\(\):count=sql\.DeleteQuery\(model\)\.delete\_batch\(\[instance\.pk\],self\.using\)\+setattr\(instance,model\.\_meta\.pk\.attname,None\)returncount,\{model\.\_meta\.label:count\}

Box[H\.1](https://arxiv.org/html/2609.21190#A8.SS1.SSS0.Px1)is the whole fix: one added line, in one file, inside one branch\. The task’s hidden tests check that the attribute is cleared after a fast delete and that the fast path still reports the same counts and still performs exactly one query\.

##### What a specification for it has to pin down\.

The change is one assignment, but the behavior it establishes has several independent facets, and a specification that misses any of them is satisfied by code the tests reject\. The identifier must end up absent\. It must be the model’s real primary\-key attribute that is cleared, which need not be namedid\. The reset must happen after the delete has been issued, so that a failed delete does not leave an object claiming to be gone\. The reported counts must be unchanged\. And the reset must cost nothing: it is an in\-memory assignment, and an implementation that consulted the database to decide what to clear would pay a query the fast path does not have\.

The four backends divide these facets differently\. Two of them state only the first and the fourth; one states all of them; one states them as prose criteria and checks them by probe\. What follows is each specification as released, with its narrative comments removed and its structure otherwise intact\.

##### Contract\-annotated Python\.

TheNaginispecification is the only one of the four that models the cost of the fix, and doing so forces it to model the database handle as well\. Box[H\.1](https://arxiv.org/html/2609.21190#A8.SS1.SSS0.Px3)shows the solver\-visible view\. Three ghost resources carry the state that the contract talks about\.RemovalBudgetis the entitlement to issue one row\-removal statement against the stored data; it is held rather than read, so an implementation can spend it but can learn nothing from it\.CarriedOutRemovalandRefusedRemovalare the store’s two possible receipts for a statement it was asked to apply\. Only the removal primitive mints a receipt, and no implementation can alter the handle, the identifier, or the count recorded on one\. That primitive is the instance’s single axiom, shown earlier as Box[B\.2](https://arxiv.org/html/2609.21190#A2.SS2), and it is the code the fix leaves alone\.

Box H\.2:django\-11179under contract\-annotated Python: the solver\-visible specification \(narrative comments removed\)[⬇](data:text/plain;base64,QENvbnRyYWN0T25seQpAUHJlZGljYXRlCmRlZiBSZW1vdmFsQnVkZ2V0KGxlZGdlcjogUmVtb3ZhbExlZGdlcikgLT4gYm9vbDogcGFzcwoKQENvbnRyYWN0T25seQpAUHJlZGljYXRlCmRlZiBDYXJyaWVkT3V0UmVtb3ZhbChkYjogRGF0YWJhc2UsIGlkZW50aWZpZXI6IGludCwgY291bnQ6IGludCkgLT4gYm9vbDogcGFzcwoKQENvbnRyYWN0T25seQpAUHJlZGljYXRlCmRlZiBSZWZ1c2VkUmVtb3ZhbChkYjogRGF0YWJhc2UsIGlkZW50aWZpZXI6IGludCkgLT4gYm9vbDogcGFzcwoKZGVmIHJlbW92ZV9zaW5nbGVfaW5kZXBlbmRlbnQoaW5zdGFuY2U6IFBlcnNpc3RlbnRPYmplY3QsIGRiOiBEYXRhYmFzZSkgLT4gaW50OgogICAgUmVxdWlyZXMoTXVzdFRlcm1pbmF0ZSgyKSkKICAgIFJlcXVpcmVzKEFjYyhpbnN0YW5jZS5waykpCiAgICBSZXF1aXJlcyhpbnN0YW5jZS5wayBpcyBub3QgTm9uZSkKICAgIFJlcXVpcmVzKC0oMiAqKiA2MykgPD0gaW5zdGFuY2UucGsgYW5kIGluc3RhbmNlLnBrIDwgMiAqKiA2MykKICAgIFJlcXVpcmVzKFJlbW92YWxCdWRnZXQoTEVER0VSKSkKICAgIEVuc3VyZXMoQWNjKGluc3RhbmNlLnBrKSkKICAgIEVuc3VyZXMoQ2FycmllZE91dFJlbW92YWwoZGIsIE9sZChpbnN0YW5jZS5wayksIFJlc3VsdCgpKSkKICAgIEVuc3VyZXMoUmVzdWx0KCkgPj0gMCkKICAgIEVuc3VyZXMoaW5zdGFuY2UucGsgaXMgTm9uZSkKICAgIEV4c3VyZXMoUmVtb3ZhbFJlZnVzZWQsIEFjYyhpbnN0YW5jZS5waykKICAgICAgICAgICAgYW5kIFJlZnVzZWRSZW1vdmFsKGRiLCBPbGQoaW5zdGFuY2UucGspKQogICAgICAgICAgICBhbmQgaW5zdGFuY2UucGsgaXMgT2xkKGluc3RhbmNlLnBrKSk=)@ContractOnly@PredicatedefRemovalBudget\(ledger:RemovalLedger\)\-\>bool:pass@ContractOnly@PredicatedefCarriedOutRemoval\(db:Database,identifier:int,count:int\)\-\>bool:pass@ContractOnly@PredicatedefRefusedRemoval\(db:Database,identifier:int\)\-\>bool:passdefremove\_single\_independent\(instance:PersistentObject,db:Database\)\-\>int:Requires\(MustTerminate\(2\)\)Requires\(Acc\(instance\.pk\)\)Requires\(instance\.pkisnotNone\)Requires\(\-\(2\*\*63\)<=instance\.pkandinstance\.pk<2\*\*63\)Requires\(RemovalBudget\(LEDGER\)\)Ensures\(Acc\(instance\.pk\)\)Ensures\(CarriedOutRemoval\(db,Old\(instance\.pk\),Result\(\)\)\)Ensures\(Result\(\)\>=0\)Ensures\(instance\.pkisNone\)Exsures\(RemovalRefused,Acc\(instance\.pk\)andRefusedRemoval\(db,Old\(instance\.pk\)\)andinstance\.pkisOld\(instance\.pk\)\)

Read the postconditions as a group and each facet appears\. The returning exit must hold a carried\-out receipt naming the handle it was given, the identifier the object carried*on entry*, and exactly the count being returned: the statement was really issued, for the right row, and the number surfaced is that statement’s own report rather than a figure read off the store afterwards\. The identifier must be absent on that exit and unchanged on the other, so the object gives up its identity exactly when the row it names has been removed\. And the entitlement is stated as a precondition of the primitive and never returned, so the whole method may issue one statement and no more\. The permissionAcc\(instance\.pk\)is required on entry and re\-established on both exits\. No permission on any of the store’s own tallies is available anywhere in the contract, so reading them is impossible rather than merely discouraged\.

##### An imperative method in an embedded DSL\.

TheVelvetspecification, Box[H\.1](https://arxiv.org/html/2609.21190#A8.SS1.SSS0.Px4), takes the opposite view of what the instance is about\. The removal count is an input rather than something the method obtains, so the batch removal is not re\-derived and is observed only through that number\. What remains is the state change itself, and the method declares the identity handle mutable and pins its post\-state exactly\.

Box H\.3:django\-11179under the imperative DSL[⬇](data:text/plain;base64,c2V0X29wdGlvbiBsb29tLnNlbWFudGljcy50ZXJtaW5hdGlvbiAidG90YWwiCnNldF9vcHRpb24gbG9vbS5zZW1hbnRpY3MuY2hvaWNlICJkZW1vbmljIgoKb3BlbiBUb3RhbENvcnJlY3RuZXNzIERlbW9uaWNDaG9pY2UKCm1ldGhvZCBmYXN0RGVsZXRlIChtdXQgcGsgOiBPcHRpb24gSW50KSAocmVtb3ZlZCA6IE5hdCkgcmV0dXJuIChjb3VudCA6IE5hdCkKICBlbnN1cmVzIHBrID0gbm9uZQogIGVuc3VyZXMgY291bnQgPSByZW1vdmVkCiAgZG8KICAgIHBrIDo9IG5vbmUKICAgIHJldHVybiByZW1vdmVkCgpwcm92ZV9jb3JyZWN0IGZhc3REZWxldGUgYnkKICBsb29tX3NvbHZl)set\_optionloom\.semantics\.termination”total”set\_optionloom\.semantics\.choice”demonic”openTotalCorrectnessDemonicChoicemethodfastDelete\(mutpk:OptionInt\)\(removed:Nat\)return\(count:Nat\)ensurespk=noneensurescount=removeddopk:=nonereturnremovedprove\_correctfastDeletebyloom\_solve

Two postconditions do all the work\. The first pins the changed part of the state to its exact post\-value; the second pins the unchanged part by stating that the returned count equals the count that came in\. Without that second clause, an implementation that clears the handle and reports a different number satisfies the specification, and the task’s own tests would reject it\. The correctness obligation is handed to the solver\-backed tactic and discharged without a hand\-written proof\.

##### Pure functional Lean\.

TheLeanspecification, Box[H\.1](https://arxiv.org/html/2609.21190#A8.SS1.SSS0.Px5), is the only one that models the whole collected job rather than one object\. Records are grouped by kind; each carries its optional primary key; the outcome is the pair of the updated groups and the reported count\. The two paths differ in how they tally, so the count is pinned to a branch\-selected value, but the key\-clearing outcome is identical on both paths and is pinned structurally\.

Box H\.4:django\-11179under pure functionalLean\(definitions and theorem statement; proof omitted\)[⬇](data:text/plain;base64,c3RydWN0dXJlIEluc3RhbmNlIHdoZXJlCiAgcGsgOiBPcHRpb24gSW50CmRlcml2aW5nIERlY2lkYWJsZUVxCgphYmJyZXYgTGFiZWwgOj0gTmF0CmFiYnJldiBSZWNHcm91cCA6PSBMYWJlbCDDlyBMaXN0IEluc3RhbmNlCgpkZWYgY2xlYXJBbGxQa3MgKGRhdGEgOiBMaXN0IFJlY0dyb3VwKSA6IExpc3QgUmVjR3JvdXAgOj0KICBkYXRhLm1hcCAoZnVuIGcgPT4gKGcuMSwgZy4yLm1hcCAoZnVuIGkgPT4geyBpIHdpdGggcGsgOj0gbm9uZSB9KSkpCgpkZWYgaXNTaW5nbGVGYXN0IChkYXRhIDogTGlzdCBSZWNHcm91cCkgKGRpcmVjdGx5RGVsZXRhYmxlIDogSW5zdGFuY2Ug4oaSIEJvb2wpIDogQm9vbCA6PQogIG1hdGNoIGRhdGEgd2l0aAogIHwgWyhfLCBbaW5zdF0pXSA9PiBkaXJlY3RseURlbGV0YWJsZSBpbnN0CiAgfCBfID0+IGZhbHNlCgpkZWYgcHJvYmxlbV9zcGVjX2RlbGV0ZQogICAgKGRhdGEgOiBMaXN0IFJlY0dyb3VwKSAoZGlyZWN0bHlEZWxldGFibGUgOiBJbnN0YW5jZSDihpIgQm9vbCkKICAgIChkaXJlY3RDb3VudCBnZW5lcmFsVG90YWwgOiBJbnQpCiAgICAocmVzdWx0IDogKExpc3QgUmVjR3JvdXApIMOXIEludCkgOiBQcm9wIDo9CiAgcmVzdWx0LjEgPSBjbGVhckFsbFBrcyBkYXRhIOKIpwogIHJlc3VsdC4yID0gKGlmIGlzU2luZ2xlRmFzdCBkYXRhIGRpcmVjdGx5RGVsZXRhYmxlIHRoZW4gZGlyZWN0Q291bnQgZWxzZSBnZW5lcmFsVG90YWwpCgp0aGVvcmVtIGNvcnJlY3RfZGVsZXRlIDoKICAgIOKIgCAoZGF0YSA6IExpc3QgUmVjR3JvdXApIChkaXJlY3RseURlbGV0YWJsZSA6IEluc3RhbmNlIOKGkiBCb29sKQogICAgICAoZGlyZWN0Q291bnQgZ2VuZXJhbFRvdGFsIDogSW50KSwKICAgICAgcHJvYmxlbV9zcGVjX2RlbGV0ZSBkYXRhIGRpcmVjdGx5RGVsZXRhYmxlIGRpcmVjdENvdW50IGdlbmVyYWxUb3RhbAogICAgICAgIChpbXBsZW1lbnRhdGlvbl9kZWxldGUgZGF0YSBkaXJlY3RseURlbGV0YWJsZSBkaXJlY3RDb3VudCBnZW5lcmFsVG90YWwp)structureInstancewherepk:OptionIntderivingDecidableEqabbrevLabel:=NatabbrevRecGroup:=Label×ListInstancedefclearAllPks\(data:ListRecGroup\):ListRecGroup:=data\.map\(fung=\>\(g\.1,g\.2\.map\(funi=\>\{iwithpk:=none\}\)\)\)defisSingleFast\(data:ListRecGroup\)\(directlyDeletable:Instance→Bool\):Bool:=matchdatawith\|\[\(\_,\[inst\]\)\]=\>directlyDeletableinst\|\_=\>falsedefproblem\_spec\_delete\(data:ListRecGroup\)\(directlyDeletable:Instance→Bool\)\(directCountgeneralTotal:Int\)\(result:\(ListRecGroup\)×Int\):Prop:=result\.1=clearAllPksdata∧result\.2=\(ifisSingleFastdatadirectlyDeletablethendirectCountelsegeneralTotal\)theoremcorrect\_delete:∀\(data:ListRecGroup\)\(directlyDeletable:Instance→Bool\)\(directCountgeneralTotal:Int\),problem\_spec\_deletedatadirectlyDeletabledirectCountgeneralTotal\(implementation\_deletedatadirectlyDeletabledirectCountgeneralTotal\)

Note the universally quantifieddirectlyDeletable\. Whether a given object qualifies for the direct single\-row removal is an unchanged decision that the fix does not touch\. The specification quantifies over every such predicate instead of axiomatizing it, so the theorem holds for all of them\. That is the general pattern behind the axiom counts of[SectionB\.2](https://arxiv.org/html/2609.21190#A2.SS2): a language that can abstract over the unchanged decision does not need to assume anything about it\. The proof is an ordinary kernel\-checked term; it splits on the shape of the collected groups and on the predicate, and closes each case by simplification\.

##### Structured natural language\.

TheEarsrequirements document, excerpted in Box[H\.1](https://arxiv.org/html/2609.21190#A8.SS1.SSS0.Px6), is the only one of the four that states every facet, because prose is not constrained by what a solver can be made to accept\. It is also the only one with no verifier behind it, so what it buys is a precise statement of intent rather than a proof\.

Box H\.5:django\-11179under structured natural language \(acceptance criteria, abridged\)Requirement 1: clear the primary key on the fast\-delete path\.1\.1\.WHEN a single collected instance of a single model qualifies for fast deletion and is deleted via the collector’s fast path, THEN the deletion machinery SHALL set that instance’s primary\-key attribute toNone\.1\.2\.WHEN a dependency\-free model instance is deleted through the model’sdelete\(\)method, THEN, after the call returns, the deletion machinery SHALL leave the instance’s primary\-key attribute equal toNone\.1\.3\.The system SHALL clear the primary\-key attribute using the model’s actual primary\-key attribute name, so that models whose primary key is not literally namedidalso have the correct attribute reset\.1\.4\.WHEN the fast path clears the primary key, THEN the deletion machinery SHALL perform the reset after the database delete has been issued, so that a failure of the delete operation does not leave the instance reporting a cleared key for a row that still exists\.Requirement 2: preserve fast\-path count and return semantics\.2\.1\.WHEN a dependency\-free instance is deleted via the fast path, THEN the deletion machinery SHALL return the total number of deleted objects together with a per\-model\-label mapping of deletion counts, exactly as before the primary\-key reset was introduced\.2\.2\.The system SHALL leave the number of database operations performed by the fast path unchanged; clearing the primary key SHALL be an in\-memory attribute assignment that issues no additional query\.2\.3\.IF the collected work does not match the single\-instance, single\-model, fast\-deletable case, THEN the deletion machinery SHALL NOT take the fast path and SHALL instead follow the general deletion path\.Requirement 3\(six criteria, abridged\) preserves the general path’s key\-clearing, the queued field updates, the bulk\-deletion counts, the pre\-delete and post\-delete signal ordering, the configured on\-delete behaviors, and the fast\-deletability decision itself\.

##### What each specification pins, and what it leaves open\.

[Table43](https://arxiv.org/html/2609.21190#A8.T43)lays the four side by side\. They do not agree on how much of the change to model\. Each is required to be strong enough that the pre\-fix behavior fails it and weak enough that legitimate implementations pass, and there are many specifications between those two bounds\.

Table 43:The same task under four backends\. Each row is a facet of the behavior the reference fix establishes; each cell states how that backend’s specification pins it, or that it does not\. “Not modeled” means the facet is outside what that specification talks about, which is admissible as long as the pre\-fix behavior still fails; the discrimination check of[SectionE\.2](https://arxiv.org/html/2609.21190#A5.SS2)enforces that\.FacetNaginiVelvetLeanEarsModeled scopeone object and the handle it is removed throughthe object’s identity handle and the reported countevery object in every collected groupthe deletion path in placeCarrier of the keyheap field of optional integer type, with explicit access permissionmutable parameter of optional integer typeoptional\-integer field of a record inside labeled groupsthe model’s primary\-key attribute, by nameKey is clearedabsent on the returning exitpinned to absentevery record in every group, structurallycriteria 1\.1–1\.3Counts unchangedequal to the count the receipt namesequal to the count suppliedequal to the branch\-selected tallycriterion 2\.1Cost of the fixone removal statement, held as a consumable entitlementnot modelednot modeledcriterion 2\.2Reset happens after the deletepinned on both exits, one per receiptnot modelednot modeledcriterion 1\.4Store may refusesecond exit with its own postconditionnot modelednot modelednot statedCorrect attribute namemodeled as the named field the attribute stands fornot modelednot modeledcriterion 1\.3Unchanged callees assumed100–Obligation discharged bySMT solver, via the intermediate verifiersolver\-backed tactickernel\-checked proof termreading, plus executable probes
##### The pre\-fix twin, four times\.

Each of the three prover backends ships the same specification paired with the behavior the issue complains about, and each of those three must fail\. They fail for different reasons\. Under contract\-annotated Python the twin drops the assignment and the verifier reports that the postcondition placing the identifier absent might not hold\. Under the imperative DSL the twin returns the handle unchanged and the solver refuses the obligation directly\. Under pureLeanthe twin returns the collected groups untouched on the fast branch, and the theorem no longer closes because the structural equality with the cleared groups fails on the single\-record case\. TheEarsbundle has no verifier, so its analogue is the probe suite: the same six requirement probes are run against the repository before and after the fix, and they must separate the two trees\.

##### The evidence each bundle carries\.

[Table44](https://arxiv.org/html/2609.21190#A8.T44)reports what was actually run for this instance\. TheNaginibundle carries one axiom, so it carries an axiom probe:9393recorded observations of Django’s real deletion primitive, covering live rows, absent identifiers, and the extremes of the stated6464\-bit domain\. The other two prover bundles have no axioms and therefore no probe\. TheNaginireference body performs an assignment and returns a value it did not compute, so there are no operators to mutate and the mutation check does not apply; the burden falls entirely on the twin and on the property test suite, whose discrimination lane holds3636of its4040cases\. Under the imperative DSL the body is likewise an assignment and a return, and the differential harness has nothing to compare, so that check does not apply either\. The mutation check does apply, because the DSL body still contains a mutation to perturb\.

Table 44:Admission evidence recorded fordjango\-11179, per backend\. Property test suite cases are split into the admissibility, soundness, and discrimination lanes of[SectionB\.3](https://arxiv.org/html/2609.21190#A2.SS3)\. “Attacks” counts the candidate implementations an adversarial audit submitted against the specification and the number that broke it\. A dash marks a check that does not apply, with the reason given in the text\.The differential figure forNaginicomes from100,000100\{,\}000generated inputs\. The harness ran Django’s patched fast branch on each of them inside the task’s container, ran the modeled reference on the corresponding modeled inputs, and found no disagreement\. Of those runs,33,21733\{,\}217produced a refused removal rather than a count, which is the outcome the second exit of the contract describes\. That exit is not hypothetical: a third of the observed behavior takes it, and the postcondition on it was exercised against the real primitive\.

##### What the four\-way comparison shows\.

Three things, none of which is visible from the aggregate tables\.

First, verification cost tracks how much of the environment a specification models, not how large the code change is\. The same one\-line fix takes3\.53\.5seconds to verify in the DSL that models the object alone and19\.919\.9seconds in the contract language that models the handle, the entitlement, and both exits\. Across the corpus this is the dominant term in verification time\. The distributions in[SectionE\.2](https://arxiv.org/html/2609.21190#A5.SS2)therefore separate cleanly by backend rather than by task\.

Second, a specification that models more is harder to write and harder to game, and the adversarial effort spent on it reflects that\. The contract\-annotated specification is the only one of the four in which reading the store’s tallies is even expressible as an attack, and it is the only one whose audit needed twenty candidate implementations; the DSL specification, which says nothing about the store, needed nine\. The count of attacks is a measure of attack surface, not of specification quality\.

Third, one instance carries four different difficulties under a single reference fix\. A solver asked to satisfy the DSL specification must produce a state mutation that preserves a count\. A solver asked to satisfy the contract\-annotated one must additionally reason about a consumable resource and two exits, and must do so without reading state it has no permission to touch\. Both are legitimate formalizations of the same issue, and reporting per\-backend results separately, as[appendixG](https://arxiv.org/html/2609.21190#A7)does, is a consequence: the numbers are not measuring the same task in four notations\.

#### H\.2Twenty attacks on one specification, and how each is calibrated

A specification that verifies its own reference implementation has established almost nothing\. The property that matters is that it*rejects*the wrong implementations, and that property cannot be checked by inspection\. The attacker panel of[SectionE\.3](https://arxiv.org/html/2609.21190#A5.SS3)therefore attacks each specification directly: an agent with the specification view, the repository, and the verifier writes candidate implementations intended to satisfy the contract while doing something the issue forbids, and each candidate is run through the verifier\. A candidate that verifies is a finding, and it is then checked against the task’s tests to determine whether it is a specification weakness or a legitimate reshaping of the reference\.

This subsection reproduces the record for the contract\-annotated specification ofdjango\-11179in full\. Twenty candidates were submitted\. Nineteen reached the verifier, and none of them broke the specification; one could not be translated and is counted in neither tally\.[Table45](https://arxiv.org/html/2609.21190#A8.T45)lists all twenty, grouped by what the candidate was trying to get away with, together with the mechanism that stopped it\.

Table 45:The twenty candidate implementations submitted against the contract\-annotated specification ofdjango\-11179, grouped by intent\. Each was written to satisfy the contract while violating something the issue requires\. The right column names the mechanism that rejected it\. Candidate 16 verifies; its materialized patch also passes the task’s hidden tests, so it is a legitimate alternative implementation rather than a weakness\. Candidate 15 was rejected by the translation layer before reaching the solver and counts in neither tally\.##### A family that was once open\.

The second group of[Table45](https://arxiv.org/html/2609.21190#A8.T45)is in the suite because an earlier version of this specification failed it\. An earlier audit found two verifying wrong implementations: one read the store’s own count of carried\-out removals before issuing the delete and restored the identifier if the delete was refused, and one discarded the removal’s return value and echoed the store’s report of the last removal instead\. Both verified against the specification as it then stood, and neither is a correct program: obtaining either quantity in reality costs a query the fast path does not have\.

The repair was structural rather than a patched\-in clause\. Instead of adding postconditions to forbid the two reads, the specification was rewritten so that no access permission on any of the store’s fields reaches the modeled path at all, and the information the path legitimately has was moved into the three ghost receipts\. All four original shapes were then re\-encoded verbatim and re\-run, and all four are now rejected\. The rejection happens at the read rather than at a postcondition, because the branch that performs the read cannot be written\. A specification patched clause\-by\-clause invites the same family to reappear parameterized on the next unpinned difference; a specification from which the whole class of unpriced reads is inexpressible does not\.

##### The families are not interchangeable\.

Each group in[Table45](https://arxiv.org/html/2609.21190#A8.T45)is stopped by a different part of the specification, and removing any one part would open a family\. The permission discipline stops the second group: because no access to the store’s own fields is granted anywhere in the contract, a candidate that reads them does not fail a postcondition, it fails to compile a read\. The entitlement stops the third group: one statement is available, it is consumed by the removal primitive on either answer, and nothing mints another, so a second removal has no precondition to satisfy at its call site\. The two receipts stop the fourth: an implementation cannot mint either one, so it cannot reach an exit whose postcondition names a receipt it does not hold\. The termination measure stops the sixth, and it has to be stated at the method level to do so; loop invariants alone leave the diverging candidate verifying\.

##### Attacks that target the boundary rather than the behavior\.

Candidates 7 and 18 attack the domain rather than the postconditions\. Both work at the extremes of the representable identifier range, where a specification stated over unbounded integers and an implementation working on machine integers can come apart\. Candidate 7 spends the one available statement on the identifier immediately below the one the object carries, which is a distinct value inside the domain; candidate 18 arranges the refusing exit only at the very bottom of the range\. Both are rejected on the same grounds as their non\-boundary counterparts: the receipts name the identifier, and naming it leaves no room for a near miss\.

##### The candidate that could not be judged\.

Candidate 15 tried to forge the entitlement by constructing the ghost resource directly rather than receiving it\. The translation layer rejected the program before the solver saw it, so no verdict was produced\. We record this outcome as its own category and count it in neither tally, because a candidate that the tooling cannot express is evidence about the tooling and not about the specification\. Across the corpus these cases are rare, and each is recorded with the reason the translation failed\.

##### Calibrating an attack against the task’s own tests\.

An attack that verifies is not yet a finding\. The specification is supposed to admit every correct implementation, so a candidate that verifies may simply be a correct implementation written differently from the reference\. The two cases are distinguished by materializing the candidate as a repository patch and running it against the task’s hidden tests\. If it resolves the task, the specification correctly admitted a correct program\. If it does not, the specification admitted a program the task rejects, and that is a weakness\.

Three controls anchor the suite, and together they show the calibration working in both directions\.

##### The candidate that verifies, and should\.

Candidate 16 clears the identifier before asking the store and restores it if the store refuses\. It reads nothing it has no permission to read, holds the correct receipt on each exit, and leaves the identifier absent exactly when a removal was carried out\. It verifies, in15\.815\.8seconds\. Materialized as a patch and run against the task’s tests, it resolves the instance: the one fail\-to\-pass test passes and all4040pass\-to\-pass tests pass\. The specification admitted it because it is correct, not because the specification is weak\. The reordering is observationally invisible: the difference between clearing before and clearing after is visible only if the delete fails, and the specification’s second exit requires the identifier to be restored in exactly that case\.

##### The candidate that would have been a finding\.

Candidate 2 reads the store’s own count of carried\-out removals and returns it instead of the count the removal statement reported\. Under the specification it is rejected outright, for lack of permission on that field\. To check that the rejection was right, the candidate’s behavior was written out as a faithful repository patch and run against the tests\. It does not resolve the task\. It fails a pass\-to\-pass test that asserts the fast path performs exactly one query, because obtaining the store’s tally in reality requires a second one\. Three independent runs agree\. The specification’s refusal to grant that permission is the formal counterpart of a test the task already has\.

##### The reference itself, and the check that the proof is not vacuous\.

The positive control is the reference fix verbatim\. Its patch is byte\-identical to the reference diff of Box[H\.1](https://arxiv.org/html/2609.21190#A8.SS1.SSS0.Px1), and it both verifies and resolves\. Alongside it the bundle carries a vacuity probe: the reference implementation with an assertion of falsity placed on the returning path\. That probe must fail to verify, and it does\. Without it, a specification whose preconditions were unsatisfiable, or whose returning path was unreachable, would verify every candidate and reject none, and the entire attack suite above would be vacuous\. Every specification in the corpus carries this control\.

##### Why the calibration has to be part of the protocol\.

Without the test\-harness step, the attacker panel is a source of false findings in one direction and false confidence in the other\. An agent instructed to break a specification will produce candidates that verify, because many correct programs verify; grading those as weaknesses would make every well\-written specification look broken\. Conversely, an agent whose candidate is rejected has learned nothing about whether the rejection was justified\. Running the candidate against tests the task already ships resolves both cases against an authority neither the specification nor the agent controls\.[SectionE\.3](https://arxiv.org/html/2609.21190#A5.SS3)reports how often each outcome occurs across the corpus\.

##### What the suite costs, and why it is retained\.

Driving nineteen candidates through the verifier for this one specification costs minutes of prover time apiece, and writing them took an agent considerably longer\. The bundle therefore keeps them: each candidate is stored as source alongside the verdict it must receive and the reason, so a later audit that reconstructs the same attack reproduces the outcome without re\-deriving it\. The twenty attacks this specification survived are stated, so a reader who thinks of a twenty\-first can add it and see\.

#### H\.3When faithfulness and non\-disclosure collide

The specification view shown to a solver has to satisfy two requirements that usually sit comfortably together\. It must be faithful: strong enough that satisfying it entails the behavior the hidden tests check\. And it must not disclose the fix: it may reveal nothing about the change beyond what the public problem statement already says\. For a minority of instances these requirements are jointly unsatisfiable\.[SectionA\.4](https://arxiv.org/html/2609.21190#A1.SS4)states the disposition we adopt and reports how many instances it applies to; this subsection works through one of them in detail\.

##### The task\.

scikit\-learn\-26194concerns the thresholds returned alongside a receiver\-operating\-characteristic curve\. When the classifier’s scores are probability estimates, the routine’s leading threshold could exceed11, which is not a value any probability can take, and the public issue reports this as a bug\. The issue then proposes a remedy: clip the thresholds so that none exceeds11\. Box[H\.3](https://arxiv.org/html/2609.21190#A8.SS3.SSS0.Px1)shows the public text alongside the change the maintainers actually made and the assertions the hidden tests make\.

Box H\.6:scikit\-learn\-26194: what the public text asks for, what the fix does, and what the tests checkFrom the public problem statement\.The first threshold returned can be above11when the scores are probability estimates, which is not a meaningful probability\. The suggested fix is to clip the thresholds to a maximum of11\.The reference fix\.[⬇](data:text/plain;base64,LSAgICB0aHJlc2hvbGRzID0gbnAucl9bdGhyZXNob2xkc1swXSArIDEsIHRocmVzaG9sZHNdCisgICAgdGhyZXNob2xkcyA9IG5wLnJfW25wLmluZiwgdGhyZXNob2xkc10=)\-thresholds=np\.r\_\[thresholds\[0\]\+1,thresholds\]\+thresholds=np\.r\_\[np\.inf,thresholds\]What the hidden tests assert\.That the returned thresholds are exactly\[inf, 1\.0, 0\.7, 0\.0\]and\[inf, 1\.0, 0\.9, 0\.7, 0\.6, 0\.0\]on the two fixtures, and that the leading threshold satisfiesisinf\.

The fix does not clip\. It replaces the leading sentinel, previously the largest observed threshold plus one, with positive infinity\. The reasoning is that the leading entry is not a real threshold at all but the degenerate point at which nothing is predicted positive, and that infinity is the honest name for it\. The tests assert that value literally: two of them compare the whole threshold array against a list beginning with infinity, and one checks the leading entry with an infinity predicate\.

##### Why the two requirements cannot both hold\.

A faithful specification has to distinguish the fixed behavior from the behavior the tests reject\. The behavior the tests reject includes the clipping the issue itself proposes: clipping produces a leading threshold of1\.01\.0, which fails the infinity predicate and fails both array comparisons\. So a specification that pins only “the leading threshold is at most11” \(everything the public text supports\) is satisfied by an implementation the task fails\. To be faithful, the specification must name the infinite sentinel\.

But the discriminator is absent from the public text\. The tokensinf,isinf, andnp\.infdo not occur in the problem statement, and this instance ships no developer discussion at all, so there is no other public source for them\. What the public text does contain isclip, and a suggested non\-regression test asserting that every returned threshold is at most11or at least00\. Every real number satisfies that predicate, and the predicate mentions no sentinel at all\. Naming the sentinel in the specification view therefore tells a solver what the leading value changes to, which is the entire content of the fix\. The public text omits that value and points away from it\.

Box H\.7:scikit\-learn\-26194: the discriminating clause, under contract\-annotated Python[⬇](data:text/plain;base64,QFB1cmUKQENvbnRyYWN0T25seQpkZWYgaXNfcG9zX2luZih4OiBmbG9hdCkgLT4gYm9vbDogcGFzcwoKQFB1cmUKQENvbnRyYWN0T25seQpkZWYgcG9zX2luZigpIC0+IGZsb2F0OgogICAgRW5zdXJlcyhpc19wb3NfaW5mKFJlc3VsdCgpKSkKCmRlZiBhbmNob3JfdGhyZXNob2xkcyh0aHJlc2hvbGRzOiBMaXN0W2Zsb2F0XSkgLT4gTGlzdFtmbG9hdF06CiAgICBSZXF1aXJlcyhBY2MobGlzdF9wcmVkKHRocmVzaG9sZHMpLCAxIC8gMikpCiAgICBSZXF1aXJlcyhsZW4odGhyZXNob2xkcykgPj0gMSkKICAgIFJlcXVpcmVzKE11c3RUZXJtaW5hdGUoMikpCiAgICBFbnN1cmVzKEFjYyhsaXN0X3ByZWQodGhyZXNob2xkcyksIDEgLyAyKSkKICAgIEVuc3VyZXMoQWNjKGxpc3RfcHJlZChSZXN1bHQoKSkpKQogICAgRW5zdXJlcyhsZW4oUmVzdWx0KCkpID09IGxlbih0aHJlc2hvbGRzKSArIDEpCiAgICBFbnN1cmVzKGlzX3Bvc19pbmYoUmVzdWx0KClbMF0pKQogICAgRW5zdXJlcyhUb1NlcShSZXN1bHQoKSkuZHJvcCgxKSA9PSBUb1NlcSh0aHJlc2hvbGRzKSk=)@Pure@ContractOnlydefis\_pos\_inf\(x:float\)\-\>bool:pass@Pure@ContractOnlydefpos\_inf\(\)\-\>float:Ensures\(is\_pos\_inf\(Result\(\)\)\)defanchor\_thresholds\(thresholds:List\[float\]\)\-\>List\[float\]:Requires\(Acc\(list\_pred\(thresholds\),1/2\)\)Requires\(len\(thresholds\)\>=1\)Requires\(MustTerminate\(2\)\)Ensures\(Acc\(list\_pred\(thresholds\),1/2\)\)Ensures\(Acc\(list\_pred\(Result\(\)\)\)\)Ensures\(len\(Result\(\)\)==len\(thresholds\)\+1\)Ensures\(is\_pos\_inf\(Result\(\)\[0\]\)\)Ensures\(ToSeq\(Result\(\)\)\.drop\(1\)==ToSeq\(thresholds\)\)

Box[H\.3](https://arxiv.org/html/2609.21190#A8.SS3.SSS0.Px2)is the specification as released\. The clauseis\_pos\_inf\(Result\(\)\[0\]\)is the discriminator, and it is unavoidable: an implementation satisfying the length and tail clauses while prepending any finite value verifies and fails the task\. There is no weaker clause that separates the two, because the tests are stated as an equality against a specific value\. A specification that says “the leading value is greater than every threshold in the input” would be non\-disclosing and would still be satisfied by the pre\-fix code, which prepends the maximum plus one\. A specification that says “the leading value is not finite” has already said everything the fix says\.

##### The four ways out, and why we take the third\.

The collision admits exactly four responses, and the choice among them affects how the corpus should be read\.

Weaken the specification and keep non\-disclosure: the instance is retained, but the guarantee is void, since a verified submission need not resolve the task\. This is the one option we rule out unconditionally: it breaks the property the benchmark exists to provide\.

Strengthen the public text to mention the sentinel: this would restore consistency, at the cost of editing the task\. We do not modify the problem statements we inherit, so this is unavailable\.

Keep the instance, permit the discriminator in the specification view, and record the relaxation per instance\. This is what we do, and it is the only option that preserves both the guarantee and the task\.

Discard the instance\. The instances that exhibit the collision are not a random sample\. They are precisely the tasks where the public issue report is a poor description of the fix that was made: the reporter proposed one remedy and the maintainers chose another\. Removing them would systematically strip the corpus of the cases in which the natural\-language statement underdetermines the answer, and that underdetermination is what distinguishes a real repository issue from a competition problem\.

##### What is relaxed, and what is not\.

The relaxation is local to one property\. For this instance every other admission check holds in its ordinary form\. The specification verifies in13\.513\.5seconds, under an interpreted encoding for real arithmetic\. This bundle is one of the1111Naginibundles that need one, since the instance is about the values of floating\-point thresholds\. The pre\-fix twin fails, and it fails on the discriminating clause itself, reporting that the finite anchoring value is not the infinite sentinel; the two specifications are byte\-identical outside the implementation region, so the twin isolates that one substitution and nothing else\. The differential harness compares the modeled reference against the real patched routine on120,012120\{,\}012generated inputs with no disagreement\. All seven mutants of the reference body are killed\. The property test suite’s eight cases each produce their required verdict\. Seven adversarial candidates were submitted against the specification, and none broke it\. The two numeric primitives the specification assumes \(the infinity sentinel and the test for it\) are probed against the real library, on boundary values, the test fixture, and five thousand generated inputs\. What is relaxed is only the non\-disclosure screen, and the record states which tokens it would otherwise have flagged\.

An instance qualifies for the relaxation only if an independent adversarial analysis, run with the public text and the repository in hand, fails to find any reformulation derivable from that text which pins the tested behavior\. For this instance the analysis reconstructed the mismatch from both sides: it confirmed that the discriminating vocabulary occurs in the reference fix and the hidden test and nowhere in the solver\-visible text, and it confirmed that the remedy the public text does propose is incompatible with the behavior the tests require\. The mechanical screen, which is lexical, passes this instance: the specification view contains no diff line, no patched path, and no patched symbol\. The relaxation is recorded against the semantic collision, which no lexical screen can see\.

##### Reading the corpus in light of this\.

Two consequences follow, and both are stated in the released records\. First, the affected instances are marked, so anyone who wants a corpus in which no specification view exceeds the public text can exclude them and lose the counts reported in[SectionA\.4](https://arxiv.org/html/2609.21190#A1.SS4)\. Second, the settings that supply a specification to a solver are, on those instances, supplying more help than the problem statement contains\. The specification\-supplied rows of[appendixG](https://arxiv.org/html/2609.21190#A7)should therefore be read as an upper bound on that setting’s value\. That is also why the per\-stratum analysis there separates instances by how much the public text already determines\.

### Appendix IRepresentative evaluation episodes

The results in[appendixG](https://arxiv.org/html/2609.21190#A7)report how often a submission clears each grader\. They do not show what an episode looks like from the inside, and they do not show a submission that clears one grader and fails another with both verdicts in view\. This section prints three episodes in full, quoting the recorded interaction\.

The three were chosen to occupy three different cells of the outcome table, and two of them deliberately share an instance\.[SectionI\.1](https://arxiv.org/html/2609.21190#A9.SS1)is an episode in the Verified End\-to\-End setting that clears every grader: the agent writes a patch, writes a specification, is rejected twice by the verifier, repairs, and is accepted by the equivalence panel\.[SectionI\.2](https://arxiv.org/html/2609.21190#A9.SS2)is an episode in the same setting whose specification verifies on the first attempt and whose patch does not resolve\. That quadrant exists only because the two deliverables are graded separately, and it shows where a formal artifact can be honest and still miss\.[SectionI\.3](https://arxiv.org/html/2609.21190#A9.SS3)is a Specification Synthesis episode on the same instance as[SectionI\.2](https://arxiv.org/html/2609.21190#A9.SS2), in a different run and a different setting\. Its specification verifies and is then rejected unanimously by the audit panel on faithfulness, for exactly the reason the previous episode’s patch failed its hidden test\. The closing discussion of this appendix draws the three together\.

All three are Opus 4\.8 episodes on theNaginibackend from the first repetition, chosen so that the verifier output, the specification language, and the model are constant across the section and only the setting and the outcome vary\. The two that share an instance are independent: they come from different runs, in different settings, and neither agent could see the other’s work\.

##### What an episode record contains\.

Every episode is recorded in five parts\.

The*task view*is exactly the material the agent was given: the standing instruction that describes the environment and the ground rules, the setting instruction that states what is to be produced and in what order, the issue text, and the localization hint or the specification view in the settings that supply them\. The view is recorded as the agent saw it rather than reconstructed afterwards\.[SectionF\.1](https://arxiv.org/html/2609.21190#A6.SS1)describes what each setting supplies\.

The*interaction log*is an ordered list of events\. Each model turn carries its visible text and the action it requested\. Each shell call carries the command issued, its exit status, and the output the agent saw\. Each verifier call carries the artifact submitted, the verdict, the verifier’s own message, and the wall time the call took\. The log ends with the submission event and its note\.

The*submission*is the patch, captured as a diff of the agent’s edits against the base commit, together with the formal artifact in the settings that require one\. Nothing else the agent did to its container is carried forward\.

The*mechanical grades*record resolution under the official harness, verification under the backend’s own verifier, and the hygiene screen, each as a verdict with a short reason\.

The*adjudications*record every agentic verdict at the level of the individual ballot: for equivalence, each judge’s decision, confidence, and stated reason; for the specification audit, each judge’s five per\-property booleans, its reason, and the list of tool calls it made with the signal each returned\. The anti\-cheat screen’s result for the episode is recorded alongside\.

Counters derived from the log are stored with the record: model turns, shell calls, verifier calls and accepting verdicts, verifier wall time, episode wall time, and token consumption\.[Table16](https://arxiv.org/html/2609.21190#A6.T16)and[SectionG\.6](https://arxiv.org/html/2609.21190#A7.SS6)aggregate them\.

##### Transcript conventions\.

In this section a gray box is material given to the agent, a blue box is the agent’s own turn, a tan box is the output of a shell call, a red box is a verdict returned by the verifier or by a grader, and a green box is a summary of the record\. Everything inside a box is quoted from the record\. Elisions are marked\[\.\.\.\]; nothing is reworded\. Two things are cut throughout\. The standing instruction is identical across the episodes of a setting and is described in[SectionF\.2](https://arxiv.org/html/2609.21190#A6.SS2)rather than reprinted\. Long test output is shown as the agent saw it, which in these episodes means the last lines of the run, because the agent piped it that way\.

Table 46:The three episodes\. Wall time and token counts are the recorded totals for the episode; “verify” gives the number of verifier calls, the number that returned an accepting verdict, and the wall time spent inside the verifier\. The grader columns give the outcome of each grader that applies to the setting; – means the grader does not run in that setting, and “skipped” means it was not reached because an earlier grader had already failed\.
#### I\.1A verified end\-to\-end episode

This episode runs ondjango\-11179, the instance whose four corpus specifications are printed side by side in[SectionH\.1](https://arxiv.org/html/2609.21190#A8.SS1)\. The agent here has seen none of them: in the Verified End\-to\-End setting both the patch and the specification are constructed from the issue and the code\.

Setting instruction, row 3 \(excerpt\)Your task has TWO graded deliverables you construct from scratch \(no spec isgiven\): a Nagini MODEL and a real\-code PATCH\. You are scored on ALL of: \(i\) themodel VERIFIES; \(ii\) the patch RESOLVES the hidden tests; and \(iii\) anindependent audit confirming your model’s specification faithfully representsthe issue’s intent AND is logically equivalent to your patch\. \[\.\.\.\]\(1\) LOCATE THE REAL FIX FIRST\. \[\.\.\.\] confirmed by observing the wrongbehavior there, not guessed from the issue title\.\(2\) CONSTRUCT a Nagini specification of the CORRECTED behavior at thosesite\(s\) \[\.\.\.\] It does not need to be an exhaustive or airtight contract;it needs to genuinely represent the issue’s intent rather than a vacuousor trivially\-true property\.\(3\) Write a model implementing that spec and iterate with \`nagini\_verify\`until it VERIFIES \[\.\.\.\]\(4\) DERIVE your /testbed edit as a faithful translation of the verified model\(case for case\), at the real site\(s\) from step 1 \[\.\.\.\]\(5\) VALIDATE END\-TO\-END\. \[\.\.\.\] Then \`submit\` with model\_path set to yourverified model\.\-\-\- ISSUE \(problem statement\) \-\-\-delete\(\) on instances of models without any dependencies doesn’t clear PKs\.DescriptionDeleting any model with no dependencies not updates the PK on the model\. Itshould be set to None after \.delete\(\) call\.See Django\.db\.models\.deletion:276\-281\. Should update the model line 280\.

The issue names a file and a line range, which is unusually specific for this corpus, and the agent’s first action is to read them\.

Step 1: shell \(output abridged\)$ sed \-n ’260,330p’ /testbed/django/db/models/deletion\.pydef delete\(self\):\[\.\.\.\]\# Optimize for the case with a single obj and no dependenciesif len\(self\.data\) == 1 and len\(instances\) == 1:instance = list\(instances\)\[0\]if self\.can\_fast\_delete\(instance\):with transaction\.mark\_for\_rollback\_on\_error\(\):count = sql\.DeleteQuery\(model\)\.delete\_batch\(\[instance\.pk\], self\.using\)return count, \{model\.\_meta\.label: count\}\[\.\.\.\]for model, instances in self\.data\.items\(\):for instance in instances:setattr\(instance, model\.\_meta\.pk\.attname, None\)return sum\(deleted\_counter\.values\(\)\), dict\(deleted\_counter\)\[exit 0\]

One read is enough\. The two branches are visible in the same window: the fast path returns before the loop that clears identifiers, and the general path runs it\. The agent states the diagnosis in a single sentence and edits\.

Step 2: agentThe fast\-delete path returns early without setting the PK to None\. Let me fixit\.

The edit, applied through a scripted textual replacement, inserts one statement before the fast path’s return:

Step 2: the resulting changeif self\.can\_fast\_delete\(instance\):with transaction\.mark\_for\_rollback\_on\_error\(\):count = sql\.DeleteQuery\(model\)\.delete\_batch\(\[instance\.pk\], self\.using\)\+ setattr\(instance, model\.\_meta\.pk\.attname, None\)return count, \{model\.\_meta\.label: count\}

This is byte\-for\-byte the reference fix\. The remainder of the episode is about the other deliverable\.

Step 3: agent writes its specificationNow the Nagini model\.from typing import Optionalfrom nagini\_contracts\.contracts import \*class Instance:def \_\_init\_\_\(self\) \-\> None:self\.pk = 1 \# type: Optional\[int\]Ensures\(Acc\(self\.pk\) and self\.pk == 1\)@ContractOnlydef delete\_batch\(pk: Optional\[int\]\) \-\> int:Ensures\(Result\(\) \>= 0\)def fast\_delete\(instance: Instance\) \-\> int:Requires\(Acc\(instance\.pk\)\)\# After delete on a no\-dependency instance, the PK must be cleared\.Ensures\(Acc\(instance\.pk\)\)Ensures\(instance\.pk is None\)Ensures\(Result\(\) \>= 0\)count = delete\_batch\(instance\.pk\)instance\.pk = Nonereturn count

The removal itself is an unchanged callee and is axiomatized as one, with the weakest fact the proof needs: it returns a count that is not negative\. The obligation the issue asks for is stated directly as a postcondition on the changed operation: after it returns, the identifier is cleared\. The class exists only to give the identifier somewhere to live\.

Step 4: verifier,23\.523\.5sVerification failedErrors:cand\_model\.py:8:34:8:46: error: The precondition of \(self\.pk == 1\) might not hold\.Verification took 21\.67 seconds\.

This rejection is not about deletion\. It is about permission discipline: reading a field inside a contract requires holding access to that field, and in the constructor’s postcondition the conjunction asserts the permission and consumes it in the same expression\. The agent splits the conjunction into two clauses\.

Step 5: agent \(the changed lines\)Ensures\(Acc\(self\.pk\)\)Ensures\(self\.pk == 1\)

Step 6: verifier,24\.724\.7sVerification failedErrors:cand\_model\.py:9:17:9:29: error: The precondition of \(self\.pk == 1\) might not hold\.Verification took 22\.87 seconds\.

The same rejection, one line lower\. Splitting the clauses does not help, because each clause is checked on its own and the second one still reads a field whose permission the first one asserts\. The agent then gives up part of its claim\.

Step 7: agentThe constructor postcondition referencing self\.pk needs Acc in same Ensures\.Let me simplify—remove the constructor entirely since it’s not needed\.

What it actually writes keeps the constructor but drops the claim about the identifier’s initial value, leaving only the permission:

Step 7: the changed constructorclass Instance:def \_\_init\_\_\(self\) \-\> None:Ensures\(Acc\(self\.pk\)\)self\.pk = 1 \# type: Optional\[int\]

Step 8: verifier,21\.921\.9sVerification successfulVerification took 20\.33 seconds\.

The claim that was dropped is not part of the obligation\. Nothing in the issue depends on what the identifier held before the deletion; the obligation is that afterwards it holds nothing\. The agent’s simplification removes a decorative claim and leaves the load\-bearing one intact\. That is the distinction the anti\-fakery screen enforces, and the screen accepts this artifact\.

The last two actions are validation and submission\.

Step 9: shell$ python tests/runtests\.py delete \-\-parallel 1Creating test database for alias ’default’\.\.\.\.\.s\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-Ran 41 tests in 0\.831sOK \(skipped=1\)\[exit 0\]

Step 10: agent submitsTests pass\. The fix sets the PK to None on the fast\-delete path\.submit\(note="Fast\-delete path in Collector\.delete now clears the instance PK\(setattr to None\) before returning, matching the slow path behavior\."\)

Grading is then mechanical up to the last step\. The patch is re\-applied to a clean checkout inside the official image and resolves\. The specification and implementation are re\-checked by the verifier on the grading host and verify\. The hygiene screen passes\. Only then does the equivalence panel run\.

Grading: equivalence panel,33accept /00rejectvote 1 \(confidence 0\.95\): Spec clause ’instance\.pk is None’ matches the issue’sintent that PK be cleared after delete; model’s setting pk=None afterdelete\_batch mirrors patch’s setattr\(instance, model\.\_meta\.pk\.attname, None\)after delete\_batch\.vote 2 \(confidence 0\.95\): Spec’s Ensures\(instance\.pk is None\) faithfullycaptures the issue’s intent that PK be cleared after delete; model’sdelete\_batch\-\>set pk=None\-\>return count mirrors the patch’sdelete\_batch\-\>setattr\(None\)\-\>return count\.vote 3 \(confidence 0\.95\): Spec’s Ensures\(instance\.pk is None\) captures theissue’s intent of clearing PK after delete; model’s delete\_batch\+pk=Nonemirrors patch’s delete\_batch\+setattr\(pk,None\)\.

##### What the formal channel contributed\.

Three verifier calls,70\.170\.1seconds of the episode’s300\.8300\.8, and every one of them was about the specification language rather than about Django\. The first two rejections concern where a permission may be stated; the third call accepts\. Not one of them said anything the agent could have used to locate or improve the fix, because the fix was already correct at step 2\. In this episode the verifier functioned as a proof checker\. The episode passes because the agent’s reading of the issue was right before it started specifying\.

##### What the specification does and does not say\.

Set beside the corpus specification for the same instance in[SectionH\.1](https://arxiv.org/html/2609.21190#A8.SS1), this one is very small\. It says nothing about the count reported to the caller, nothing about the general path, and nothing about the store the removal talks to; the corpus specification prices all three, because it has to survive twenty adversarial candidates rather than one panel\. The agent’s specification is nonetheless not vacuous: the pre\-fix behavior violates it, which is the property the corpus construction requires of a specification and the property the anti\-fakery screen checks a submission for\. Small and discriminating is the common case among the accepted row\-3 submissions\.

#### I\.2A verified submission that does not resolve

The next episode is in the same setting, with the same model and backend, and its formal artifact verifies on the first attempt\. It fails, because the patch does not resolve\. This is the outcome that the two\-deliverable design exists to expose; a resolution\-only benchmark cannot see it\.

Issue \(excerpt\)SelectDateWidget can crash with OverflowError\.DescriptionGiven a relatively common view like this: \[\.\.\.\]A user can trigger a server crash, reproducible by running locally and visitinghttp://127\.0\.0\.1:8000/repro/?my\_date\_day=1&my\_date\_month=1&my\_date\_year=1234567821345678,which results in\[\.\.\.\]File "\[\.\.\.\]/django/forms/widgets\.py", line 1160, in value\_from\_datadictdate\_value = datetime\.date\(int\(y\), int\(m\), int\(d\)\)OverflowError: signed integer is greater than maximum\[\.\.\.\]Specifically, y, m and d are user controlled, and the code does this:date\_value = datetime\.date\(int\(y\), int\(m\), int\(d\)\)When large integers \(larger than sys\.maxsize\) are supplied to date’sconstructor it will throw an OverflowError \[\.\.\.\]

The agent reads the function and finds a guarded region that already handles one failure mode\.

Step 1: shell \(output abridged\)$ sed \-n ’1140,1175p’ django/forms/widgets\.pydef value\_from\_datadict\(self, data, files, name\):\[\.\.\.\]if y is not None and m is not None and d is not None:input\_format = get\_format\("DATE\_INPUT\_FORMATS"\)\[0\]input\_format = formats\.sanitize\_strftime\_format\(input\_format\)try:date\_value = datetime\.date\(int\(y\), int\(m\), int\(d\)\)except ValueError:\# Return pseudo\-ISO dates with zeros for any unselected values,\# e\.g\. ’2017\-0\-23’\.return "%s\-%s\-%s" % \(y or 0, m or 0, d or 0\)return date\_value\.strftime\(input\_format\)return data\.get\(name\)\[exit 0\]

Step 2: agentThe fix is to also catch \`OverflowError\`\. Let me make the edit\.

Step 2: the resulting changetry:date\_value = datetime\.date\(int\(y\), int\(m\), int\(d\)\)\- except ValueError:\+ except \(ValueError, OverflowError\):\# Return pseudo\-ISO dates with zeros for any unselected values,\# e\.g\. ’2017\-0\-23’\.return "%s\-%s\-%s" % \(y or 0, m or 0, d or 0\)

The specification follows, and it follows the patch closely\.

Step 3: agent writes its specification \(comments abridged\)\# Abstracts datetime\.date\(y, m, d\): \[\.\.\.\] We model both error conditions as a\# single "raises" flag\.@Pure@ContractOnlydef date\_ok\(y: int, m: int, d: int\) \-\> bool:\# True iff datetime\.date\(y, m, d\) succeeds \(no ValueError, no OverflowError\)\.\.\.\.@Pure@ContractOnlydef date\_strftime\(y: int, m: int, d: int\) \-\> int:Requires\(date\_ok\(y, m, d\)\)Ensures\(Result\(\) \>= 0\)\.\.\.\# Result encoding:\# tag 0 \-\> formatted valid date \(value = date\_strftime\(\.\.\.\)\)\# tag 1 \-\> pseudo\-ISO fallback string \(value unused\), used whenever the\# date constructor would raise \(ValueError OR OverflowError\)\.\# The KEY behavior of the fix: an out\-of\-range/too\-large value must NOT\# propagate an exception; it must fall into the fallback \(tag 1\)\.def value\_from\_datadict\(y: int, m: int, d: int\) \-\> Tuple\[int, int\]:Ensures\(date\_ok\(y, m, d\) == \(Result\(\)\[0\] == 0\)\)Ensures\(Implies\(not date\_ok\(y, m, d\), Result\(\)\[0\] == 1\)\)Ensures\(Implies\(Result\(\)\[0\] == 0, Result\(\)\[1\] == date\_strftime\(y, m, d\)\)\)if date\_ok\(y, m, d\):return \(0, date\_strftime\(y, m, d\)\)else:\# ValueError or OverflowError both handled \-\> pseudo\-ISO fallback\.return \(1, 0\)

Step 4: verifier,27\.027\.0sVerification successfulVerification took 25\.10 seconds\.

The agent then validates, and validates carefully: it runs the widget’s own test module, and it reproduces the crash from the issue\.

Steps 5 and 6: shell$ python tests/runtests\.py forms\_tests\.widget\_tests\.test\_selectdatewidgetFound 15 test\(s\)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-Ran 15 tests in 0\.145sOK\[exit 0\]$ python \- <<EOF\[\.\.\.\]w = SelectDateWidget\(\)print\(repr\(w\.value\_from\_datadict\(\{’d\_day’:’1’,’d\_month’:’1’,’d\_year’:’1234567821345678’\}, \{\}, ’d’\)\)\)EOF’1234567821345678\-1\-1’\[exit 0\]

Step 7: agent submitsNo more crash\.Let me verify the model and patch align case for case: \`date\_ok\` true \-\> tag 0\-\> strftime; false \(ValueError or OverflowError\) \-\> tag 1 \-\> fallback\. The patchcatches both \`ValueError\` and \`OverflowError\`, returning the pseudo\-ISOfallback\. Matches\.

The self\-check in that turn is correct on its own terms\. The patch and the specification do agree, case for case\. The reproduction did stop crashing\.

Gradingresolution : FAIL "patch did not resolve"verification : pass "verified"anti\-fakery : pass "model is an honest Nagini contract"equivalence : skipped \("verify/resolve/anti\-fakery not all green"\)hygiene : clean \(0 flags\)

##### What the reference fix does instead\.

The gold patch does not widen the existing handler\. It adds a second, separate one, which returns a different string\.

Box I\.1: The reference fix fordjango\-16667[⬇](data:text/plain;base64,ICAgICAgICAgICAgICAgICAjIFJldHVybiBwc2V1ZG8tSVNPIGRhdGVzIHdpdGggemVyb3MgZm9yIGFueSB1bnNlbGVjdGVkIHZhbHVlcywKICAgICAgICAgICAgICAgICAjIGUuZy4gJzIwMTctMC0yMycuCiAgICAgICAgICAgICAgICAgcmV0dXJuICIlcy0lcy0lcyIgJSAoeSBvciAwLCBtIG9yIDAsIGQgb3IgMCkKKyAgICAgICAgICAgIGV4Y2VwdCBPdmVyZmxvd0Vycm9yOgorICAgICAgICAgICAgICAgIHJldHVybiAiMC0wLTAiCiAgICAgICAgICAgICByZXR1cm4gZGF0ZV92YWx1ZS5zdHJmdGltZShpbnB1dF9mb3JtYXQp)\#Returnpseudo\-ISOdateswithzerosforanyunselectedvalues,\#e\.g\.’2017\-0\-23’\.return”%s\-%s\-%s”%\(yor0,mor0,dor0\)\+exceptOverflowError:\+return”0\-0\-0”returndate\_value\.strftime\(input\_format\)

Two of the held\-out tests turn on that distinction\. One asserts that an oversized year produces exactly the string`"0\-0\-0"`; the other requires that the same string, fed back through the field, is rejected as invalid, so that the form reports an error rather than accepting a value it cannot represent\. Widening the existing handler makes the crash stop and produces`'1234567821345678\-1\-1'`instead\. That is a pseudo\-date that echoes the attacker\-supplied year and that the field will try to parse\. The agent printed that string, read it as evidence that the crash was gone, and submitted\. It was also the evidence that the fix was wrong, and it was in the transcript\.

##### Why the specification could not have caught it\.

The specification is not vacuous\. Its abstraction of the date constructor is two\-valued: the components are accepted, or they are not\. The pre\-fix behavior, which lets the rejection escape, does not satisfy it\. It would pass the discrimination requirement that every corpus specification has to meet\. But because the abstraction has two values where the required behavior has three, both patches satisfy it: the reference fix and the agent’s widened handler map to the same tag on an oversized year, and the returned text is not modeled at all\. The information the hidden test checks was discarded when the abstraction was chosen, before a single line of proof was written\. No amount of proving recovers it\.

The corpus specification for the same instance distinguishes the two rejections and pins the sentinel exactly\.

Box I\.2: The corpus specification fordjango\-16667\(postconditions,Nagini\)[⬇](data:text/plain;base64,ZGVmIHZhbHVlX2Zyb21fY29tcG9uZW50cyhhbGxfYmxhbms6IGJvb2wsIGFsbF9wcmVzZW50OiBib29sLAogICAgICAgICAgICAgICAgICAgICAgICAgIGtpbmQ6IGludCkgLT4gTGlzdFtpbnRdOgogICAgUmVxdWlyZXMoSW1wbGllcyhhbGxfYmxhbmssIGFsbF9wcmVzZW50KSkKICAgIFJlcXVpcmVzKGtpbmQgPT0gMCBvciBraW5kID09IDEgb3Iga2luZCA9PSAyKQogICAgUmVxdWlyZXMoTXVzdFRlcm1pbmF0ZSgzKSkKICAgIEVuc3VyZXMoQWNjKGxpc3RfcHJlZChSZXN1bHQoKSkpKQogICAgRW5zdXJlcyhJbXBsaWVzKGFsbF9ibGFuaywgVG9TZXEoUmVzdWx0KCkpID09IGFic2VudF90ZXh0KCkpKQogICAgRW5zdXJlcyhJbXBsaWVzKG5vdCBhbGxfcHJlc2VudCwgVG9TZXEoUmVzdWx0KCkpID09IHJhd192YWx1ZSgpKSkKICAgICMgZXZlcnkgb3V0Y29tZSByZXR1cm5zIHRleHQ6IE5FSVRIRVIgcmVqZWN0aW9uIGVzY2FwZXMKICAgIEVuc3VyZXMoSW1wbGllcyhhbGxfcHJlc2VudCBhbmQgbm90IGFsbF9ibGFuaywgbGVuKFRvU2VxKFJlc3VsdCgpKSkgPiAwKSkKICAgICMgYSBjYXBhY2l0eS1vdmVyZmxvdyBmYWlsdXJlIHlpZWxkcyBFWEFDVExZIHRoZSAiMC0wLTAiIHNlbnRpbmVsCiAgICBFbnN1cmVzKEltcGxpZXMoYWxsX3ByZXNlbnQgYW5kIG5vdCBhbGxfYmxhbmsgYW5kIGtpbmQgPT0gMiwKICAgICAgICAgICAgICAgICAgICBUb1NlcShSZXN1bHQoKSkgPT0gemVyb196ZXJvX3plcm8oKSkpCiAgICBFbnN1cmVzKEltcGxpZXMoYWxsX3ByZXNlbnQgYW5kIG5vdCBhbGxfYmxhbmsgYW5kIGtpbmQgPT0gMCwKICAgICAgICAgICAgICAgICAgICBUb1NlcShSZXN1bHQoKSkgPT0gc3VjY2Vzc190ZXh0KCkpKQogICAgRW5zdXJlcyhJbXBsaWVzKGFsbF9wcmVzZW50IGFuZCBub3QgYWxsX2JsYW5rIGFuZCBraW5kID09IDEsCiAgICAgICAgICAgICAgICAgICAgVG9TZXEoUmVzdWx0KCkpID09IG91dF9vZl9yYW5nZV90ZXh0KCkpKQ==)defvalue\_from\_components\(all\_blank:bool,all\_present:bool,kind:int\)\-\>List\[int\]:Requires\(Implies\(all\_blank,all\_present\)\)Requires\(kind==0orkind==1orkind==2\)Requires\(MustTerminate\(3\)\)Ensures\(Acc\(list\_pred\(Result\(\)\)\)\)Ensures\(Implies\(all\_blank,ToSeq\(Result\(\)\)==absent\_text\(\)\)\)Ensures\(Implies\(notall\_present,ToSeq\(Result\(\)\)==raw\_value\(\)\)\)\#everyoutcomereturnstext:NEITHERrejectionescapesEnsures\(Implies\(all\_presentandnotall\_blank,len\(ToSeq\(Result\(\)\)\)\>0\)\)\#acapacity\-overflowfailureyieldsEXACTLYthe”0\-0\-0”sentinelEnsures\(Implies\(all\_presentandnotall\_blankandkind==2,ToSeq\(Result\(\)\)==zero\_zero\_zero\(\)\)\)Ensures\(Implies\(all\_presentandnotall\_blankandkind==0,ToSeq\(Result\(\)\)==success\_text\(\)\)\)Ensures\(Implies\(all\_presentandnotall\_blankandkind==1,ToSeq\(Result\(\)\)==out\_of\_range\_text\(\)\)\)

The construction differs in three ways, all of them consequences of the same decision\. The outcome of the date constructor is three\-valued, tagging the two rejections separately\. The result is text rather than a tag, modeled as a sequence of code points, so a postcondition can name a particular string;`zero\_zero\_zero\(\)`is the five code points of`"0\-0\-0"`\. And each branch is pinned to a distinct value, so an implementation that returns the out\-of\-range rendering on an overflow violates the contract\. The corpus specification’s own summary calls this “the discriminating obligation”, and it is the obligation the held\-out test checks\.

##### Two procedural notes\.

The equivalence panel did not run\. It is skipped whenever the mechanical graders are not all green: a judgment that a specification and a patch describe the same behavior is uninformative when the patch is known to be wrong\.[SectionF\.3](https://arxiv.org/html/2609.21190#A6.SS3)states the ordering rule\. The hygiene screen is clean and the anti\-fakery screen accepts\. Nothing about this submission is dishonest; it is a faithful proof of an insufficient claim\.

##### How common this is\.

In this run,6969of500500episodes verify and do not resolve, against429429that do both; two resolve without a verifying artifact\. Conditional on verification, the resolution rate is86\.1%86\.1\\%\. Across the four repetitions of the cell the conditional rate ranges from83\.4%83\.4\\%to86\.1%86\.1\\%\.[SectionG\.2](https://arxiv.org/html/2609.21190#A7.SS2)reports the structure of this gap across settings and backends\.

#### I\.3A synthesized specification the audit rejects

The last episode is on the same instance, in the Specification Synthesis setting, in a different run\. Here there is no patch to grade: the specification is the deliverable, any repository edit is ignored, and the grade rests on the audit panel\. The setting instruction states three demands on the specification, and it is the third that decides this episode\.

Setting instruction, row 8 \(excerpt\)Your task: from the issue and the code, CONSTRUCT a formal Nagini SPECIFICATIONof the fix the issue requires, and a small implementation that VERIFIES underit\. You are graded on the quality of the SPECIFICATION — not on editing therepository \(any repo edits are ignored\)\. \[\.\.\.\]\(2\) WRITE THE SPECIFICATION as \`Requires\`/\`Ensures\` on the function\(s\) the fixconcerns \[\.\.\.\] Make it:\- PERMISSIVE ENOUGH: the preconditions must admit every input the correctedcode legitimately handles \[\.\.\.\]\- TRUE: every postcondition must be a correct statement about the intendedbehavior — never assert something the correct fix would violate\.\- STRONG ENOUGH: the postconditions must pin the behavior tightly enoughthat ONLY a correct implementation can satisfy them\. \[\.\.\.\] After writingit, ask yourself: could an incorrect implementation still verify againstthis? If yes, strengthen it\.\(3\) For any UNCHANGED system or library function the behavior depends on, youmay add a \`@ContractOnly\` axiom \[\.\.\.\] Do NOT axiomatize the behavior thefix itself introduces \[\.\.\.\]

The agent reads the same function as before and writes a specification that goes one step further than the previous episode’s: its abstraction of the date constructor is three\-valued\.

Step 2: agent writes its specification \(comments abridged\)\# We model the date constructor as an axiomatized callee that has three\# possible outcomes encoded as an int return:\# 0 \-\> success \(a valid date\)\# 1 \-\> ValueError \(out of normal range\)\# 2 \-\> OverflowError \(integer too large\)\# This is TRUE of real datetime\.date: it never crashes other than via these\.YEAR\_MIN = 1YEAR\_MAX = 9999@Pure@ContractOnlydef date\_outcome\(y: int, m: int, d: int\) \-\> int:\# 0 success, 1 ValueError, 2 OverflowErrorEnsures\(Result\(\) \>= 0 and Result\(\) <= 2\)\# A genuinely valid date \(in\-range parts and within C long\) succeeds\.Ensures\(Implies\(y \>= YEAR\_MIN and y <= YEAR\_MAX and m \>= 1 and m <= 12and d \>= 1 and d <= 28, Result\(\) == 0\)\)\# Out\-of\-C\-long\-range years overflow\.Ensures\(Implies\(y \> YEAR\_MAX, Result\(\) \!= 0\)\)RESULT\_OK = 0RESULT\_FALLBACK = 1def value\_from\_datadict\_branch\(y: int, m: int, d: int\) \-\> int:\# POSTCONDITION pins the fix: it NEVER propagates an error; for any input\# that the date constructor rejects \(ValueError OR OverflowError\) it must\# return the fallback; only a genuinely valid date returns OK\.Ensures\(Result\(\) == RESULT\_OK or Result\(\) == RESULT\_FALLBACK\)Ensures\(Result\(\) == RESULT\_OK\) == \(date\_outcome\(y, m, d\) == 0\)Ensures\(Result\(\) == RESULT\_FALLBACK\) == \(date\_outcome\(y, m, d\) \!= 0\)oc = date\_outcome\(y, m, d\)if oc == 0:return RESULT\_OKelse:\# both ValueError \(oc==1\) and OverflowError \(oc==2\) hit this branchreturn RESULT\_FALLBACK

The distinction the axiom draws does not survive into the postconditions: the result is again two\-valued, and the two rejections are mapped to one outcome by the implementation\. Before the panel gets to say so, the verifier rejects the artifact for a different reason\.

Step 3: verifier,9\.19\.1sTranslation failedNot supported:\(Ensures\(\(Result\(\) == RESULT\_OK\)\) == \(date\_outcome\(y, m, d\) == 0\)\)

This is a language\-level refusal rather than a failed proof\. In the two lines it names, the equality sits outside the postcondition instead of inside it, so the text does not say what its author meant it to say\. The backend declines to translate the construction\. The agent recognizes the problem immediately\.

Step 4: agentThe \`Ensures\(\.\.\.\) == \(\.\.\.\)\` syntax is the issue\. Let me fix it\.Ensures\(\(Result\(\) == RESULT\_OK\) == \(date\_outcome\(y, m, d\) == 0\)\)Ensures\(\(Result\(\) == RESULT\_FALLBACK\) == \(date\_outcome\(y, m, d\) \!= 0\)\)

Step 5: verifier,16\.416\.4sVerification successfulVerification took 15\.24 seconds\.

The agent then applies the repository edit as well, unprompted, using the same widened handler as the row\-3 episode\. It is the same misreading of the fix, by the same model, in a different setting and a different run\. The setting ignores the edit, so it does not affect the grade, but the repetition is evidence that the reading is systematic rather than a sampling accident\.

Steps 6 and 7: agent submitsVerified\. The spec captures the fix: the OverflowError case \(\`oc==2\`, which thebuggy code does not catch\) must return the fallback rather than propagate\. Letme also apply the actual repo fix for completeness\.$ sed \-i ’s/ except ValueError:/ except \(ValueError, OverflowError\):/’ \\django/forms/widgets\.pysubmit\(note="Spec for SelectDateWidget\.value\_from\_datadict: thedate\-construction branch must catch both ValueError AND OverflowError,returning the pseudo\-ISO fallback rather than propagating\. \[\.\.\.\]"\)

##### The audit panel\.

Three judges vote on five properties each\. All three accept four of the five and reject the same one\. Every ballot ran the verifier on a candidate implementation and probed the axiom against the real callee inside the instance’s own image\.

Grading: specification audit,00accept /33rejectper\-property rejections \(of 3 decisive ballots\):axiom soundness 0 admissibility 0 soundness 0 completeness 0faithfulness 3tool grounding: 3/3 ballots ran the verifier; 3/3 ran an axiom probeballot 2 \(confidence 0\.85\): The probe confirmed date\_outcome is sound againstthe real datetime\.date \(unchanged callee, not a wrap\), the correct impl verifies\(sound\), and the pre\-fix buggy impl that drops the OverflowError fallback failsto verify \(complete\)\. But faithful FAILS: the candidate models the output onlyas an opaque int code and collapses ValueError and OverflowError into oneRESULT\_FALLBACK, never modeling the returned text — so it cannot pin the gold’sdiscriminating obligation that an OverflowError yields EXACTLY the ’0\-0\-0’sentinel \(vs the pseudo\-ISO fallback for ValueError\) thattest\_value\_from\_datadict asserts\. An impl returning the wrong fallback string onoverflow maps to the same RESULT\_FALLBACK and verifies, so verify does not implycorrect on the behavior the issue requires\.ballot 1 \(confidence 0\.85\): \[\.\.\.\] the branch method imposes no Requires \-\>admissible; the correct impl \(both reject\-kinds \-\> fallback\) verified \-\> sound,and the buggy port \(overflow treated as success\) failed the ’RESULT\_OK iffoutcome==0’ postcondition \-\> complete\. Faithful is FALSE: the candidatecollapses ValueError \(kind 1\) and OverflowError \(kind 2\) into a single abstractRESULT\_FALLBACK and models NO output string, leaving the gold’s centraldiscriminating obligation \[\.\.\.\] entirely unconstrained \[\.\.\.\]ballot 3 \(confidence 0\.85\): \[\.\.\.\] FAITHFUL fails: the gold pins a discriminatingobligation that OverflowError\(kind 2\) yields EXACTLY the ’0\-0\-0’ sentinel whileValueError\(kind 1\) yields the distinct pseudo\-ISO ’%s\-%s\-%s’ fallback — bothchecked by test\_value\_from\_datadict — but the candidate abstracts away alloutput strings and collapses kinds 1 and 2 into one indistinguishableRESULT\_FALLBACK \[\.\.\.\]

##### What the machine settled and what it did not\.

Soundness and completeness were decided by running the verifier: each judge ported a correct implementation into the candidate’s own signatures and confirmed it verifies, then ported the pre\-fix behavior the same way and confirmed it does not\. Axiom soundness was decided by executing a probe against the real date constructor in the instance’s image\. Four of the five properties therefore rest on machine evidence, and all four hold\.

The property that fails is the one no machine can decide, because it is not a question about the specification’s internal consistency\. It asks whether the abstraction is the right abstraction: whether what the specification chose to model includes what the issue requires\. Here it does not, and all three judges locate the omission in the same place: the returned text is not modeled, so the two rejections are indistinguishable, so an implementation that returns the wrong string on an overflow satisfies the contract\.

##### On the panel’s privilege\.

The judges see more than the agent did in[SectionI\.2](https://arxiv.org/html/2609.21190#A9.SS2)\. An audit runs after the episode has ended, and the judges are shown the issue, the reference fix, the names of the held\-out tests, and the corpus’s own artifacts for the instance\. The instruction accompanying that material states that these are references and not equivalence targets: a specification that differs from ours but carries the guarantee passes\. The ballots quoted above use that material: they name the sentinel and the test\. Given the reference behavior, the panel identifies an abstraction gap in a specification the verifier had already accepted, and attributes the failure to the right property\.[Table3](https://arxiv.org/html/2609.21190#S4.T3)reports the audit per property, and[SectionG\.3](https://arxiv.org/html/2609.21190#A7.SS3)calibrates the panel against independent re\-adjudication\.

##### How common this is\.

Faithfulness is the bottleneck of this setting\. In this run the specification passes the full audit in49\.2%49\.2\\%of episodes, and the five properties fail at3\.03\.0,5\.25\.2,11\.211\.2,4\.84\.8, and48\.648\.6percent respectively\. Faithfulness fails an order of magnitude more often than axiom soundness\. Restricting to the cleanest form of the verdict,8181of the500500episodes are rejected unanimously, on faithfulness alone, with every other property accepted by every judge; this episode is one of them\.[SectionG\.3](https://arxiv.org/html/2609.21190#A7.SS3)reports the distribution over all four repetitions and both models\.

#### I\.4What the three episodes show\.

##### The verifier is fast, decisive, and shallow\.

Across the three episodes it was called six times, accepted three, and consumed122\.6122\.6seconds of1,065\.51\{,\}065\.5seconds of episode time, or11\.5%11\.5\\%\. What it rejected was two permission\-discipline errors and one construction it declined to translate\. Not one of the six calls said anything about Django, about the issue, or about whether the fix was the right fix\. That is the correct behavior for a proof checker\. A proof\-checking channel on its own cannot carry a correctness claim about a repository\.[SectionG\.6](https://arxiv.org/html/2609.21190#A7.SS6)shows the same pattern at scale: for Opus 4\.8 the verifier’s share of episode time is2020to27%27\\%underNagini, and the calls concentrate where the artifact is the only deliverable\.

##### Verification does not entail resolution, and the gap has a shape\.

The gap is not vacuity\. All three specifications in this section are discriminating: the pre\-fix behavior violates each of them, which is the property we require of every specification in the corpus\. The gap is the choice of abstraction\. Two of the three episodes model a three\-valued behavior with a two\-valued abstraction, and once that choice is made the missing distinction cannot be recovered by proving harder\. The wrong implementation and the right one become the same object\.[SectionI\.1](https://arxiv.org/html/2609.21190#A9.SS1)passes because its small specification happens to capture the whole of a one\-line obligation, and[SectionI\.2](https://arxiv.org/html/2609.21190#A9.SS2)fails because its equally small specification captures two\-thirds of a three\-case one\.

##### The property audit is the instrument that sees the gap\.

A verifier cannot report an abstraction that is too coarse\. A coarse abstraction is not an error; it is a smaller claim, correctly proved\. The audit asks the separate question, and its per\-property verdicts say where the answer went wrong: in[SectionI\.3](https://arxiv.org/html/2609.21190#A9.SS3)four properties were discharged by machine and the fifth, faithfulness, was rejected unanimously with the omission named\. The same asymmetry holds in aggregate\. Faithfulness accounts for48\.6%48\.6\\%of failures in this run against3\.03\.0to11\.2%11\.2\\%for the other four\. We report the synthesis setting per property rather than as a single rate\.

##### Provenance of the quoted material\.

The three episodes aredjango\-11179anddjango\-16667in the Opus 4\.8Naginiruns of the Verified End\-to\-End and Specification Synthesis settings, first repetition\. Every quoted turn, verdict, ballot, and timing is read from their records, and every count in[Table46](https://arxiv.org/html/2609.21190#A9.T46)is stored with them\.

### Appendix JArtifacts, licenses, and intended use

The release has two provenances, and the terms that apply follow from which one a file belongs to\. The task data is copied unchanged from two existing benchmarks, which in turn draw on the public issue trackers and commit histories of fifteen open\-source projects\. The specifications, proofs, evidence, and admission records are new work written for this release\. This section states, for each part, what it is, where it came from, and under what terms it may be used\. It then states what a released bundle certifies and what it does not\.

##### What the release contains\.

[Table47](https://arxiv.org/html/2609.21190#A10.T47)lists the parts\. Three of them we do not redistribute: the repository snapshots, the container images, and the verifier distributions are named, pinned, and fetched, which keeps our correctness oracle identical to the upstream one\.

Table 47:The parts of the release, their provenance, and their terms\. “Not redistributed” means the release names the object and the pipeline fetches it; nothing is vendored\. The three rows in the last block are what a reader must obtain elsewhere to re\-derive a stamp\.PartWhere it comes fromTermsTask rows: problem statement, base and environment commits, reference patch, test patch, hidden test namesCopied byte\-identically fromSWE\-bench VerifiedandSWE\-bench Pro, which draw them from the projects’ issue trackers and commit historySource benchmark; quoted repository text under its project’s licenseSpecification modules and pre\-fix twinsWritten for this release from the issue text, the reference patch, and the repository behavior the patch changesMITCorrespondence and provenance mapsWritten for this release; they name repository symbols and quote short source excerptsMIT; quoted source under its project’s licenseProperty test suites, axiom probes, differential harnessesWritten for this releaseMITAdmission recordsEmitted by the construction pipeline as each check closedMITConstruction pipeline, evaluation harness, analysis and figure scriptsWritten for this releaseMITRepository snapshotsNot redistributed; a bundle names the project and the commitProject license,[Table48](https://arxiv.org/html/2609.21190#A10.T48)Per\-instance container imagesNot redistributed; pulled by name from the source benchmarks’ published registriesImage publisher’s termsVerifiers, provers, SMT solversNot redistributed; installed at the pinned revisions of[Table49](https://arxiv.org/html/2609.21190#A10.T49)[Table49](https://arxiv.org/html/2609.21190#A10.T49)
##### Task data provenance\.

Both source benchmarks are publicly distributed dataset records, and we resolve each at a fixed revision and copy the task fields verbatim\. The task\-identity check of[Table11](https://arxiv.org/html/2609.21190#A5.T11)re\-verifies byte identity for all3,0643\{,\}064bundles at release time, so any number reported onSWE\-Proofis comparable to the corresponding number upstream\. Neither of the two dataset records we resolve declares a license identifier, and we assert none on their behalf\. We redistribute the task fields because a bundle is not usable without them, and we do not relicense them; a user who needs terms for the task data should take them from the source distribution\. All of that text is derived from public project artifacts: issue reports, review comments, commits, and test files\. The projects’ own licenses are the operative ones wherever repository text is quoted\.

##### The upstream projects\.

[Table48](https://arxiv.org/html/2609.21190#A10.T48)lists the fifteen projects the two corpora draw on, with their instance counts and licenses\. Both distributions are skewed: Django alone supplies231231of the500500SWE\-bench Verifiedinstances, and the threeSWE\-bench Proprojects are split96/91/7996/91/79\. The permissive\-license families dominate theSWE\-bench Verifiedside: six projects under BSD\-3\-Clause, one under BSD\-2\-Clause, two under Apache\-2\.0, one under MIT, and Matplotlib under its own PSF\-derived license, with one copyleft exception, Pylint\.SWE\-bench Prois the opposite: all three of its projects are copyleft, two under GPL\-3\.0\-or\-later and one under AGPL\-3\.0\. Anyone redistributing modified bundles should read those three rows carefully, because the reference patches and the source excerpts quoted in aSWE\-bench Probundle’s maps carry the copyleft terms of the project they came from\. We do not restate or reinterpret any project’s license, and we do not ship a project’s source tree\. A bundle names the project and the commit, and[Table48](https://arxiv.org/html/2609.21190#A10.T48)maps the project to its terms\.

Table 48:The projects the corpora draw on, with instances per corpus and the license each project distributes itself under\. License identifiers are SPDX where one applies; Matplotlib distributes under its own PSF\-derived license, for which there is no SPDX identifier\.ProjectVerifiedProLicensedjango/django231–BSD\-3\-Clausesympy/sympy75–BSD\-3\-Clausesphinx\-doc/sphinx44–BSD\-2\-Clausematplotlib/matplotlib34–Matplotlib licensescikit\-learn/scikit\-learn32–BSD\-3\-Clauseastropy/astropy22–BSD\-3\-Clausepydata/xarray22–Apache\-2\.0pytest\-dev/pytest19–MITpylint\-dev/pylint10–GPL\-2\.0\-or\-laterpsf/requests8–Apache\-2\.0mwaskom/seaborn2–BSD\-3\-Clausepallets/flask1–BSD\-3\-Clauseansible/ansible–96GPL\-3\.0\-or\-laterinternetarchive/openlibrary–91AGPL\-3\.0qutebrowser/qutebrowser–79GPL\-3\.0\-or\-laterTotal500266
##### Container images\.

The correctness oracle runs the task’s hidden tests inside the per\-instance image the source benchmark publishes, and neither corpus uses an image we built\.SWE\-bench Verifiedimages are published by the source benchmark underswebench/sweb\.eval\.x86\_64, one per instance at thelatesttag;SWE\-bench Proimages are published underjefzda/sweap\-images, with the tag fixed per instance by the frozen task row\. We publish no image\. An image bakes a repository checkout and its whole dependency environment, so it carries the project’s license along with the terms of everything installed alongside it\.

##### Verifiers, provers, and solvers\.

A verdict is only meaningful relative to the prover that produced it, so the toolchain pins in[Table49](https://arxiv.org/html/2609.21190#A10.T49)are part of the artifact\. Both Lean\-family backends elaborate under one toolchain and import the same Mathlib revision, so a single prover configuration accounts for every Lean\-family verdict\. The table records each Lean package by revision, including the transitive ones, and the two SMT solvers by release version\.

Table 49:Verifier and prover pins\. Revisions are shown as the leading twelve hexadecimal digits; the release records them in full\. The two solver rows give release versions rather than package revisions\.ComponentRolePinTermsNaginiContract verifier for annotated Python1\.2\.0MPL\-2\.0Lean toolchainElaborator and kernelv4\.24\.0Apache\-2\.0MathlibMathematical libraryf897ebcf72cdApache\-2\.0VelvetImperative language and its logic2cf0acb04c51Apache\-2\.0LoomSolver\-backed proof automationd10340821dafApache\-2\.0batteriesStandard library extensions8da40b72feceApache\-2\.0aesopProof search725ac8cd67acApache\-2\.0QqTyped quotationsdea6a3361fa3Apache\-2\.0proofwidgetsInteractive display556caed0eadbApache\-2\.0importGraphDependency toolingd768126816beApache\-2\.0plausibleRandomized falsificationdfd06ebfe8d0Apache\-2\.0LeanSearchClientSearch client99657ad92e23Apache\-2\.0autoAutomation front end36d85bf6372fApache\-2\.0CliCommand\-line front end91c18fa62838MITz3SMT solver4\.15\.4MITcvc5SMT solver1\.3\.1BSD\-3\-Clause
##### Software the pipeline depends on\.

The construction pipeline and the evaluation harness run on Python3\.103\.10or later\. They use the source benchmark’s own package \(MIT\) for the task rows and the container conventions, the dataset and model\-hub clients \(both Apache\-2\.0\) to resolve the frozen rows, and a unified\-diff library \(MIT\) for patch handling\. Hypothesis \(MPL\-2\.0\) generates the inputs behind the differential harnesses and the axiom probes\. Model access goes through the two model providers’ Python clients \(MIT and Apache\-2\.0\) and the cloud provider’s SDK \(Apache\-2\.0\), and the figures in this appendix are drawn with Matplotlib \(its own license\) and NumPy \(BSD\-3\-Clause\)\. Container work goes through Docker; nothing in the pipeline requires a GPU\.

##### Our own terms\.

Everything we wrote is released under the MIT license: the construction pipeline, the evaluation harness, the analysis and figure scripts, and the artifacts we authored inside each bundle\. That grant does not extend to the copied task fields, to repository source quoted inside a bundle, to the container images, or to the verifier distributions; those remain under the terms listed above\.

##### Model access and what is reproducible\.

Both models were reached through their providers’ hosted APIs \([SectionF\.4](https://arxiv.org/html/2609.21190#A6.SS4)\)\. We release no model weights, and we did not fine\-tune anything\. The corpus, the harness, and the mechanical graders are reproducible; the hosted models, versioned by their providers, are not\. Every number in this appendix names the model, the backend, the setting, and the repetition it came from, and the repetition spread of[SectionG\.4](https://arxiv.org/html/2609.21190#A7.SS4)is reported next to the single\-run numbers it bounds\.

##### Personal and sensitive content\.

Task text is public issue\-tracker and repository text\. It names contributors, quotes their words, and carries their commit identifiers, and we copy it byte\-identically\. We add no personal data, we collect nothing from any person, and no field in any bundle records a demographic attribute\. The prompt an agent sees is assembled only from the frozen task row and our own instructions, so an episode record carries no information about any person that the task row did not already carry\. Instances keep their upstream identifiers, so a task withdrawn from a source benchmark is identifiable here under the same name\.

##### What a green bundle certifies\.

A green stamp carries five scoping statements\.

- •Verification is relative to the specification\.A passing verdict says the implementation satisfies the contracts that were written down\. It does not say the repository is correct, and it does not say the contracts are the right ones\.
- •The specification is partial by design\.It describes the behavior the issue is about and is silent on the rest of the function and the rest of the repository\.[SectionD\.5](https://arxiv.org/html/2609.21190#A4.SS5)states what each of the four languages can and cannot express\.
- •Resolution is relative to the hidden tests\.It is the upstream oracle, unchanged, and it is a test outcome rather than a proof\.
- •Verification and resolution are different claims\.The corpus contains instances where a submission earns one and not the other in both directions;[SectionI\.2](https://arxiv.org/html/2609.21190#A9.SS2)works one through end to end, and[SectionG\.3](https://arxiv.org/html/2609.21190#A7.SS3)gives the joint distribution\.
- •The construction\-time audit is a measurement, not a proof\.It is a panel of models running a stated attack suite \([SectionH\.2](https://arxiv.org/html/2609.21190#A8.SS2)\)\. Its calibration is measured on specifications that the construction loop did not shape, and[SectionE\.4](https://arxiv.org/html/2609.21190#A5.SS4)reports it\.

##### Intended use\.

The release is built for four uses\. The first is measuring verified program repair: an evaluation in which a submission has to satisfy a specification and resolve the issue, with the two graded separately\. The second is ablating what a specification is worth, using the nine settings of[SectionF\.1](https://arxiv.org/html/2609.21190#A6.SS1): the same instances with localization, a specification, a verification obligation, or none of them\. The third is studying specification synthesis and its failure modes, where the per\-property audit outcomes of[SectionG\.3](https://arxiv.org/html/2609.21190#A7.SS3)are more informative than a single rate\. The fourth is narrower\. Each bundle is a self\-contained verifier input with a labeled expected outcome, and the property test suites together comprise23,48223\{,\}482labeled cases across three verifiers, usable on their own as a regression suite for the verifiers themselves\.

##### Uses we advise against\.

Four, each with its reason\.

- •Training on the corpus and then reporting on it\.The corpus is small enough to memorize and public enough to be scraped\. A model trained on the bundles will score higher without being better at the task\. The specification settings are the most vulnerable, since a memorized specification is a memorized answer\. Anyone whose training data includes these bundles should say so alongside the number\.
- •Reading a pass rate as a claim about the projects\.The instances are the ones for which a formal twin could be built and admitted;[SectionA\.3](https://arxiv.org/html/2609.21190#A1.SS3)lists what that excludes and why, and the exclusions are not a random sample of maintenance work\.
- •Reusing the audit panel as a general\-purpose specification grader\.The panel’s usefulness is a measured property of the panel together with its attack suite and its retained controls\. The prompt on its own has no calibration outside the harness that measures it\.
- •Reading a verified\-but\-unresolved submission as a defect in the specification\.It is usually the opposite: a correct proof of a claim that was coarser than the issue\.[SectionI\.2](https://arxiv.org/html/2609.21190#A9.SS2)is a worked instance, and[SectionI\.3](https://arxiv.org/html/2609.21190#A9.SS3)shows the audit naming the same omission in a synthesized specification\.

##### Cost of re\-deriving the release\.

Re\-deriving the mechanical evidence is cheap and needs no model access\. Re\-verifying every released specification is2,2332\{,\}233verifier verdicts and8\.48\.4compute\-hours; re\-running the property test suites is23,48223\{,\}482cases and87\.087\.0compute\-hours; re\-running mutation adds25,45725\{,\}457generated mutants\. The contract\-annotated Python backend dominates all three, at25\.825\.8seconds per reference verification against10\.510\.5and6\.06\.0for the two Lean\-family backends on the same corpus \([SectionG\.2](https://arxiv.org/html/2609.21190#A7.SS2)\)\. The container\-side checks \(resolution, axiom probes, conformance\) are dominated by image pull and environment setup rather than by the checks themselves\. Only the two agentic admission checks and the evaluation campaign need model access; the campaign’s own cost is in[Table42](https://arxiv.org/html/2609.21190#A7.T42)\.

##### Versioning and re\-admission\.

The release is a snapshot\. Every bundle carries its own admission record naming each check, its outcome, and the verifier configuration it ran under, so a bundle can be re\-admitted from the bundle alone\. Re\-running a check changes the record, never the artifacts;[SectionE\.4](https://arxiv.org/html/2609.21190#A5.SS4)describes the case where a re\-audit overturns a recorded outcome and what happens to the bundle when it does\.