A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations

# A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations
Source: [https://arxiv.org/html/2608.18389](https://arxiv.org/html/2608.18389)
Shreya GuptaIsha ChaudharyNathaniel EnisRavi MangalGagandeep SinghCorina Pasareanu

###### Abstract

AI code agents are increasingly deployed to resolve real software issues, yet their reliability under superficial code variations remains poorly understood\. We evaluate whether coding agents that repair repository\-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form\. We introduce a random variant sampler that applies common semantics\-preserving transformations \(SPTs\)—spanning control\-flow rewrites, dead\-code injection, and identifier renaming—to produce perturbed variants\. We evaluate two agentic scaffolds \(mini\-SWE agent and OpenCode\) each backed by one of four frontier models \(Claude Opus 4\.5, Kimi K2\.5, MiniMax M2\.5, and Qwen 3\.6\-27B\) across instances drawn from SWE\-bench Verified and SWE\-bench Pro\. For each instance, the agent is run multiple times on the unperturbed and perturbed variants, yielding paired resolve\-rate estimates that isolate the perturbation effect from intrinsic stochasticity\. We find small degradation in most configurations: up to 6\.7 percentage points mean resolve\-rate drop in the most affected configurations with statistically significant degradations in 6 of 16 configurations of model, scaffold, and dataset\. Crucially, no single model ranking by robustness holds across scaffolds—Qwen is among the most robust under mini\-SWE agent on SWE\-bench Verified yet the most brittle under OpenCode—revealing a jagged robustness frontier\. The simpler scaffold \(mini\-SWE agent\) is more robust to perturbation\. Our results demonstrate that even top frontier models are susceptible to semantics\-preserving perturbations although the effect is not uniform, raising concerns about the deployment reliability of AI code agents in diverse real\-world codebases\.

Code—https://github\.com/CSU\-TrustLab/jagged\-frontier

## Introduction

In the span of a few years, AI\-powered coding tools have moved from research prototypes into mainstream software development, with developer surveys reporting that a majority of professionals now use them regularly\([31](https://arxiv.org/html/2608.18389#bib.bib37);[11](https://arxiv.org/html/2608.18389#bib.bib38);[14](https://arxiv.org/html/2608.18389#bib.bib39);[12](https://arxiv.org/html/2608.18389#bib.bib40)\)\. Their progress is charted on challenging coding benchmarks such as SWE\-bench Verified\([24](https://arxiv.org/html/2608.18389#bib.bib6)\)and SWE\-bench Pro\([7](https://arxiv.org/html/2608.18389#bib.bib5)\), against which nearly every new model and agent release is measured\. While performance numbers on these benchmarks can be instructive, they do not account for the phenomenon that neural models, in general, are known to learn shortcuts\([10](https://arxiv.org/html/2608.18389#bib.bib29)\)and therefore be susceptible to small input perturbations\. Two concerns follow\. First, if agent behavior shifts with superficial changes to the surrounding code, then benchmark numbers may overstate deployment reliability\. Second, such shifts would suggest that the underlying models rely on shallow syntactic patterns rather than an understanding of program semantics\. Prior work has evaluated the robustness of code LLMs to semantics\-preserving perturbations in single\-turn, non\-agentic settings\([38](https://arxiv.org/html/2608.18389#bib.bib13);[28](https://arxiv.org/html/2608.18389#bib.bib17);[35](https://arxiv.org/html/2608.18389#bib.bib20)\)\. However, the robustness of repository\-level code agents, which interact with the LLM over many turns, has received little systematic study\. We present, to our knowledge, the first such evaluation, and find that robustness is not a fixed property of a model but a*jagged frontier*: a model that holds up under one scaffold or codebase can be the most brittle under another, so benchmark rankings do not transfer to deployment\.

For robustness evaluation, we compare agent performance on unperturbed code repositories with*semantically equivalent*perturbed variants\. The perturbations are injected via local, semantics\-preserving transformations \(SPTs\) of the code\. We develop a library of SPTs that either mimic refactorings a developer may routinely perform or are intended to stress\-test the agent\. We use the library to build a randomized algorithm, i\.e\., a*sampler*, that draws perturbed variants of a base repository\. Our sampler randomly chooses which SPTs to apply and where to apply them\. Each variant is constructed independently, and unlike algorithms that aim to find adversarial examples\([30](https://arxiv.org/html/2608.18389#bib.bib16)\), the sampling process is not guided by feedback computed from model outcomes\. This non\-adversarial design is intentional\. Our aim, in this work, is to compute*lower bounds*for the impact of SPTs on code agents\. A feedback\-guided adversary can only do more damage\. These bounds can inform developer choices about whether and which code agents to use in deployment\. They can also help guide engineers training models and building agentic scaffolds towards robustness\-enhancing designs\.

The stochastic nature of LLM\-based agents makes the robustness evaluation challenging\. If an agent has a failing run on a perturbed variant but a passing run on the unperturbed repository, this observation is not sufficient to conclude that the perturbation was the cause; the observation might simply be an artifact of agent stochasticity\. Experiment design is further constrained by the cost of each agentic run\. Accordingly, we design the experimental methodology to isolate the perturbation effect from intrinsic run\-to\-run variability while balancing the statistical validity of our claims against the cost of the experiments\. We do this by running the agent repeatedly on both the unperturbed seed and its variants and pairing the outcomes\.

We evaluate two agentic scaffolds \(mini\-SWE agent and OpenCode\) each backed by one of four frontier models \(Claude Opus 4\.5, Kimi K2\.5, MiniMax M2\.5, and Qwen 3\.6\-27B\) across a total of 54 instances drawn from SWE\-bench Verified and SWE\-bench Pro\. Throughout, we study repository\-level issue resolution: each*task instance*pairs a repository at a base commit with an issue description, the tests an accepted patch must satisfy, and a*gold patch*, the reference edit that resolves the issue\. We measure the degradation in issue resolve rates across unperturbed and perturbed versions of a repository, as well as the change in step count and the cost of an agent run\.

Our experiments reveal the following insights: \(1\) localized perturbations applied in a simple, non\-feedback\-guided manner can cause small degradations in most configurations—up to 6\.7 percentage points in the most affected configurations and statistically significant degradations in 6 of 16 configurations; \(2\) perturbations raise the effort an agent expends \(increasing step counts and token cost by up to 9\.9% and 22\.9% respectively even in configurations where resolve rate is left largely unchanged\), so an outcome\-only view understates their impact; \(3\) comparing models within a scaffold, those such as Claude Opus 4\.5 with higher capability on unperturbed versions can be impacted more drastically by perturbations than lower\-capability models, whereas comparing across scaffolds, the simpler scaffold tends to be the more robust one; \(4\) the agentic scaffold and the benchmark shift the robustness ranking of models, so that Qwen, for instance, is among the most robust under mini\-SWE agent on SWE\-bench Verified yet the most brittle under OpenCode; and \(5\) the impact of the perturbations varies dramatically across different repositories, concentrating in a small set of instances while leaving others untouched\. Together these observations indicate that robustness is a joint property of the model, the scaffold, and the workload, not of the model alone, revealing a*jagged robustness frontier*\.

## Semantics\-Preserving Transformations

### Definition

A transformationTTis semantics\-preserving if the transformed programT⁡\(P\)T\(P\)produces the same observable behavior as the original programPPon every possible program input\. Concretely, under any input,PPandT⁡\(P\)T\(P\)must \(i\) return the same value or raise the same exception, and \(ii\) produce the same externally observable effects\. If either halts, so must the other\.

In the context of evaluating coding agents, we*operationalize*this definition through functional test\-suite equivalence\. Specifically, a transformation is considered semantics\-preserving ifT⁡\(P\)T\(P\)yields the same per\-test outcome asPPacross a project’s test\-suite\.

### Transformation Catalog

Table[1](https://arxiv.org/html/2608.18389#Sx2.T1)lists the 14 semantics\-preserving transformations \(SPTs\) we implement\. We represent this catalog using the notation𝒯\\mathcal\{T\}\. The catalog includes a mixture of SPTs that either mimic refactorings a developer might routinely perform \(e\.g\., reordering commutative operands, swap if and else branches\) or are intended to stress\-test the agent by introducing unnatural or behaviorally inert code fragments \(e\.g\., splitting string literals, dead code/method injection\)\.

Each SPT is specified by a structural pattern that needs to match for the SPT to be applicable and by a rewrite rule\. We call the locations in a file where an SPT’s pattern matches its*candidate sites*\. Two SPTs, namely Dead String Assignment and Dead Method Injection, must be bound to a target keyword, either a string literal or a method name, before rewriting\. Given such a keyword, the former injects an unread assignment of that string and the latter appends a dead method of that name\. Both plant a*decoy*: dead code that contains the keyword, so it surfaces whenever the agent searches the repository for that term\. An agent that localizes by keyword must tell the decoy apart from genuine sites of interest\. We write𝒯kw⊂𝒯\\mathcal\{T\}\_\{\\text\{kw\}\}\\subset\\mathcal\{T\}for this keyword\-bound subset of the catalog𝒯\\mathcal\{T\}\. Full implementation details of SPTs are provided in Appendix[A](https://arxiv.org/html/2608.18389#S1)\.

### Validation

We validate that our SPTs are semantics\-preserving empirically through differential testing\([19](https://arxiv.org/html/2608.18389#bib.bib4)\)against the test suites of three projects drawn from our experimental benchmark, SWE\-bench\([16](https://arxiv.org/html/2608.18389#bib.bib1)\), spanning distinct domains: SymPy \(symbolic mathematics; 12,994 tests\), sqlfluff \(SQL linting; 10,060 tests\), and xarray \(labeledNN\-dimensional arrays; 19,917 tests\)\. Each transformation is validated in isolation: for a given transformation, we apply the transformation at every applicable site, run the full test\-suite, and compare the per\-test outcome against the unmodified baseline\. Validating one transformation at a time renders any divergence in behavior attributable to a single transformation type\. Across all three projects, every test retained its outcome under all 14 transformations\. This evidence is bounded by the coverage of the underlying test\-suites and is therefore not a proof of equivalence\. As each transformation is independently validated to be preserving, we rely on this property rather than re\-verifying the test outcome of every perturbed task instance\.

Table 1:Catalog of semantics\-preserving transformations \(SPTs\)\.
### Composition and Scope

Individual transformations act on candidate sites within a single file, whereas evaluating an agent on a code repair task requires perturbing a whole repository\. We therefore apply a finite sequence⟨t1,…,tm⟩\\langle t\_\{1\},\\dots,t\_\{m\}\\rangleof SPTs to the source files, excluding test suite files, of a repository\. Eachtjt\_\{j\}is individually semantics\-preserving, and because observational equivalence is transitive, their composition is semantics\-preserving as well\. The perturbed repository is therefore observationally equivalent to the original\.

The exclusion of test suite files is what keeps such a perturbed repository usable as an evaluation target\. The oracle that defines task success is identical before and after perturbation, and no change in an agent’s score can be attributed to a moved target\.

## Sampling Semantics\-Preserving Variants

Given a seed repository associated with a task instance, the*variant sampler*\(Algorithm[1](https://arxiv.org/html/2608.18389#alg1); its subroutineAssignTargetNamesis given as Algorithm[2](https://arxiv.org/html/2608.18389#alg2)in Appendix[B](https://arxiv.org/html/2608.18389#S2)\) applies a sequence of SPTs at randomly chosen sites in randomly chosen files and returns a population of*variants*: repositories that, by the composition argument of the previous section, are semantically equivalent to the seed\. Each variant becomes an independent task instance used to evaluate the agent\. For each variant, the sampler makes four random decisions:

Algorithm 1Random Variant Sampler1:Seed repository

CseedC\_\{\\text\{seed\}\}, Sample count

NN, List of transformations

𝒯\\mathcal\{T\}, Files modified by the gold patch

ℱgold\\mathcal\{F\}\_\{\\text\{gold\}\}, Issue description

II, Number of transformations to apply

NtN\_\{t\}, Maximum number of keywords selected

NkN\_\{k\}, Fraction of candidates transformed

ϕ\\phi, Maximum files per keyword\-bound transformation

NfN\_\{f\}
2:Variant population

𝒱\\mathcal\{V\}
3:

4:procedureGenerateVariants\(

CseedC\_\{\\text\{seed\}\},

NN,

𝒯\\mathcal\{T\},

ℱgold\\mathcal\{F\}\_\{\\text\{gold\}\},

II,

NtN\_\{t\},

NkN\_\{k\},

ϕ\\phi,

NfN\_\{f\}\)

5:

𝒱←∅\\mathcal\{V\}\\leftarrow\\emptyset
6:

𝒦←None\\mathcal\{K\}\\leftarrow\\textsc\{None\}
7:for

j←1j\\leftarrow 1to

NNdo

8:

Cm​u​t←Clone​\(Cseed\)C\_\{mut\}\\leftarrow\\textsc\{Clone\}\(C\_\{\\text\{seed\}\}\)
9:

𝒯s​e​l​e​c​t​e​d←RandomSelect​\(𝒯,Nt\)\\mathcal\{T\}\_\{selected\}\\leftarrow\\textsc\{RandomSelect\}\(\\mathcal\{T\},N\_\{t\}\)
10:

pfile​\(t\)∼U⁡\(0,1\)p\_\{\\text\{file\}\}\(t\)\\sim U\(0,1\)for each

t∈𝒯s​e​l​e​c​t​e​dt\\in\\mathcal\{T\}\_\{selected\}
11:if

𝒯s​e​l​e​c​t​e​d∩𝒯kw≠∅\\mathcal\{T\}\_\{selected\}\\cap\\mathcal\{T\}\_\{\\text\{kw\}\}\\neq\\emptysetand

𝒦=None\\mathcal\{K\}=\\textsc\{None\}then

12:

𝒦←ExtractKeywords​\(I\)\\mathcal\{K\}\\leftarrow\\textsc\{ExtractKeywords\}\(I\)⊳\\trianglerightLLM call; cached

13:endif

14:

ℬ←AssignTargetNames​\(𝒯s​e​l​e​c​t​e​d,𝒦,Nk\)\\mathcal\{B\}\\leftarrow\\textsc\{AssignTargetNames\}\(\\mathcal\{T\}\_\{selected\},\\mathcal\{K\},N\_\{k\}\)
15:foreach

\(t,τ\)∈ℬ\(t,\\tau\)\\in\\mathcal\{B\}do

16:

c←0c\\leftarrow 0
17:foreach

f∈SourceFiles​\(Cmut\)f\\in\\textsc\{SourceFiles\}\(C\_\{\\text\{mut\}\}\)do

18:if

f∉ℱgoldf\\notin\\mathcal\{F\}\_\{\\text\{gold\}\}and

\(Random\>pfile​\(t\)CLOSE\\big\(\\textsc\{Random\}\>p\_\{\\text\{file\}\}\(t\)or

19:

\(t∈𝒯kwCLOSE\(t\\in\\mathcal\{T\}\_\{\\text\{kw\}\}and

OPENOPENc≥Nf\)\)c\\geq N\_\{f\}\)\\big\)then continue

20:

c←c\+1c\\leftarrow c\+1
21:

S←GetCandidates​\(f,t,τ\)S\\leftarrow\\textsc\{GetCandidates\}\(f,t,\\tau\)
22:if

S=∅S=\\emptysetthen continue

23:

S′←RandomSelect​\(S,⌈ϕ​\|S\|⌉\)S^\{\\prime\}\\leftarrow\\textsc\{RandomSelect\}\(S,\\lceil\\phi\|S\|\\rceil\)
24:

Cm​u​t​\[f\]←ApplyTransform​\(f,S′,t,τ\)C\_\{mut\}\[f\]\\leftarrow\\textsc\{ApplyTransform\}\(f,S^\{\\prime\},t,\\tau\)
25:endfor

26:endfor

27:

𝒱\.append​\(Cm​u​t\)\\mathcal\{V\}\.\\text\{append\}\(C\_\{mut\}\)
28:endfor

29:return

𝒱\\mathcal\{V\}
30:endprocedure

1. 1\.Which transformations?A subset ofNtN\_\{t\}transformations is drawn uniformly from the list of transformations𝒯\\mathcal\{T\}\.
2. 2\.Which files?For each selected transformation, an inclusion probabilitypfile∼U⁡\(0,1\)p\_\{\\text\{file\}\}\\sim U\(0,1\)is drawn and each file, excluding test suite files, is included with probabilitypfilep\_\{\\text\{file\}\}\. The files,ℱgold\\mathcal\{F\}\_\{\\text\{gold\}\}, modified by the gold patch \(i\.e\., the reference edit that resolves the issue\) are exempt from this filter and from theNfN\_\{f\}cap introduced below\. So they are always included\. These are the files an agent must locate and edit to resolve the issue\. So every variant targets the solution\-relevant region rather than targeting it by chance\.
3. 3\.Which sites?Within each included file, a fractionϕ\\phiof the candidate sites is chosen uniformly for rewriting\.
4. 4\.Which keywords?For each selected keyword\-bound transformationtt, up toNkN\_\{k\}targets are drawn uniformly from its extracted candidates\. We extract the targets for𝒯kw\\mathcal\{T\}\_\{\\text\{kw\}\}from the issue description with a single LLM call: method names it mentions for Dead Method Injection, literal strings for Dead String Assignment\.

Becausepfilep\_\{\\text\{file\}\}is resampled for every transformation, different transformations reach different portions of the repository, and the population spans from a single localized edit to pervasive, repository\-wide perturbation\.

##### Notation\.

Random∼U⁡\(0,1\)\\textsc\{Random\}\\sim U\(0,1\)is a uniform random number generator\.RandomSelect​\(X,m\)\\textsc\{RandomSelect\}\(X,m\)returnsmmelements drawn uniformly without replacement fromXX\.SourceFiles​\(C\)\\textsc\{SourceFiles\}\(C\)returns the source files ofCC, excluding test suite files\.Clone​\(C\)\\textsc\{Clone\}\(C\)copies repositoryCC\.ExtractKeywords​\(I\)\\textsc\{ExtractKeywords\}\(I\)invokes an LLM to obtain the keyword map,𝒦\\mathcal\{K\}, where𝒦\\mathcal\{K\}holds keyword candidates fort∈𝒯kwt\\in\\mathcal\{T\}\_\{\\text\{kw\}\}\.AssignTargetNames​\(⋅\)\\textsc\{AssignTargetNames\}\(\\cdot\)binds each selectedt∈𝒯kwt\\in\\mathcal\{T\}\_\{\\text\{kw\}\}to up toNkN\_\{k\}of its candidates, emitting one pair\(t,τ\)\(t,\\tau\)per keyword; every othertt, and anyttwith no candidates, is emitted as\(t,None\)\(t,\\textsc\{None\}\)\(Algorithm[2](https://arxiv.org/html/2608.18389#alg2)\)\.GetCandidates​\(f,t,τ\)\\textsc\{GetCandidates\}\(f,t,\\tau\)returns sites inffwherettcan be applied\.ApplyTransform​\(f,S′,t,τ\)\\textsc\{ApplyTransform\}\(f,S^\{\\prime\},t,\\tau\)rewritesffat every site inS′S^\{\\prime\}\.

##### Hyperparameters\.

Five hyperparameters govern the sampler\.NNis the number of variants drawn from each seed instance\.NtN\_\{t\}is the number of transformations sampled for each variant\.NkN\_\{k\}caps how many keywords each keyword\-bound transformation is bound to\.ϕ∈\(0,1\]\\phi\\in\(0,1\]is the fraction of candidate sites rewritten within each included file\. Finally,NfN\_\{f\}caps the number of files that a single keyword\-bound transformation may select\. This last bound is a practical necessity\. We observed that agents frequently detected the perturbation and reverted the repository outright \(for instance viagit reset\) when a variant contained too many decoys\. Capping the reach of the keyword\-bound transformations keeps variants within the range an agent treats as ordinary code\. The one quantity that is not fixed, the file\-inclusion probabilitypfilep\_\{\\text\{file\}\}, is drawn fromU⁡\(0,1\)U\(0,1\)for every transformation\.

The population𝒱\\mathcal\{V\}returned by Algorithm[1](https://arxiv.org/html/2608.18389#alg1)is consumed by the agent evaluation loop \(Algorithm[3](https://arxiv.org/html/2608.18389#alg3)in Appendix[C](https://arxiv.org/html/2608.18389#S3)\), which provisions an isolated environment for each variant, runs the agent, and evaluates its patch against the instance’s test oracle\.

## Experiments

### Research Questions

Our evaluation is organized around the following questions\.

RQ1To what extent do SPTs degrade agent performance, and how does this effect vary across models and scaffolds?

RQ2How much additional effort do SPTs induce on agents, controlling for task outcome?

RQ3What failure patterns emerge when agents fail on semantics\-preserving variants?

### Setup

##### Benchmarks and instance selection\.

We draw task instances from two popular repository\-level program\-repair benchmarks: SWE\-bench Verified\([24](https://arxiv.org/html/2608.18389#bib.bib6)\), a human\-validated 500\-instance subset of SWE\-bench\([16](https://arxiv.org/html/2608.18389#bib.bib1)\), and SWE\-bench Pro\([7](https://arxiv.org/html/2608.18389#bib.bib5)\), which contains 731 instances\. Constrained by a compute budget, we selected 28 task instances from SWE\-bench Verified and 26 from SWE\-bench Pro\. Our selection procedure is detailed in Appendix[C](https://arxiv.org/html/2608.18389#S3)\. Each task instance pairs a repository at a base commit withFAIL\_TO\_PASSandPASS\_TO\_PASStest sets that an accepted patch must satisfy\. TheFAIL\_TO\_PASStests are those that fail at the base commit but must pass once the issue is fixed\.PASS\_TO\_PASStests, by contrast, are those that already pass at the base commit but must remain passing to demonstrate that the patch fixes the issue without causing any regression\.

##### Agents and models\.

We evaluate two agentic scaffolds, mini\-SWE agent\([36](https://arxiv.org/html/2608.18389#bib.bib7)\)and OpenCode\([25](https://arxiv.org/html/2608.18389#bib.bib8)\), each driven by one of four backing models: Claude Opus 4\.5\([1](https://arxiv.org/html/2608.18389#bib.bib9)\), Kimi K2\.5\([32](https://arxiv.org/html/2608.18389#bib.bib10)\), MiniMax M2\.5\([21](https://arxiv.org/html/2608.18389#bib.bib11)\), and Qwen 3\.6\-27B\([27](https://arxiv.org/html/2608.18389#bib.bib12)\)\. Crossing two scaffolds, four models, and two benchmarks yields 16 configurations, each of which we evaluate independently\. Within a configuration the agent is run under a fixed setting, identical on the seed and on its variants, so that any change in outcome is attributable to the perturbation rather than to the agent\.

##### Variant generation\.

We configure the Random Variant Sampler \(Algorithm[1](https://arxiv.org/html/2608.18389#alg1)\) withN=20N=20,Nt=3N\_\{t\}=3,Nk=5N\_\{k\}=5,Nf=10N\_\{f\}=10, andϕ=0\.7\\phi=0\.7\. Under this configuration the sampler produces 20 variants per instance\. For keyword\-bound transformations, the target keywords are extracted from the issue description alone, using the same model that backs the agent under evaluation\.

##### Perturbation magnitude\.

Averaged over a task instance’s variants, the transformations target a median of 6\.9% of lines of source code on SWE\-bench Verified and 7\.7% on SWE\-bench Pro\. These percentages are lower bounds, since each transformation also inserts new code at the sites it touches\.

### Metrics

All metrics are computed per configuration, a \(scaffold, model, benchmark\) triple\. Metrics are reported both per instance and in aggregate\.

##### Resolve rate\.

A run*resolves*an instance if its patch passes both theFAIL\_TO\_PASSandPASS\_TO\_PASStest sets\. For instanceii, the baseline resolve rater0​\(i\)r\_\{0\}\(i\)is the fraction of theN=20N\{=\}20unperturbed runs that resolve it, and the perturbed resolve raterp​\(i\)r\_\{p\}\(i\)is the fraction of theN=20N\{=\}20variant runs that resolve it\. Both are proportions in\[0,1\]\[0,1\]\. We report them in percentage points\.

##### Degradation\.

Our primary metric is the per\-instance*degradation*

Δ⁡\(i\)=r0​\(i\)−rp​\(i\),\\Delta\(i\)\\;=\\;r\_\{0\}\(i\)\-r\_\{p\}\(i\),where a positive value means the agent resolves the issue less often after perturbation\. We summarize each configuration by the mean degradationΔ¯=1\|ℐ\|​∑iΔ⁡\(i\)\\bar\{\\Delta\}=\\frac\{1\}\{\|\\mathcal\{I\}\|\}\\sum\_\{i\}\\Delta\(i\)\(ℐ\\mathcal\{I\}is the set of instances\) and compareΔ¯\\bar\{\\Delta\}across configurations\.

##### Effort and cost\.

To capture overhead that resolve rate alone misses, we report the mean difference between perturbed and baseline runs in two quantities: agent steps and cost\. We uses⁡\(⋅\)s\(\\cdot\)for the number of agent steps in a run \(model turns or tool invocations\) andc⁡\(⋅\)c\(\\cdot\)for its token cost\. Lets¯0​\(i\)\\bar\{s\}\_\{0\}\(i\)ands¯p​\(i\)\\bar\{s\}\_\{p\}\(i\)be the mean step count over the baseline and perturbed runs of instanceii\. We report the relative change

δstep​\(i\)=s¯p​\(i\)−s¯0​\(i\)s¯0​\(i\)×100%,\\delta\_\{\\text\{step\}\}\(i\)\\;=\\;\\frac\{\\bar\{s\}\_\{p\}\(i\)\-\\bar\{s\}\_\{0\}\(i\)\}\{\\bar\{s\}\_\{0\}\(i\)\}\\times 100\\%,and defineδcost​\(i\)\\delta\_\{\\text\{cost\}\}\(i\)the same way fromc⁡\(⋅\)c\(\\cdot\)\. We summarize a configuration by the mean over instances,δ¯=1\|ℐ\|​∑iδ⁡\(i\)\\bar\{\\delta\}=\\frac\{1\}\{\|\\mathcal\{I\}\|\}\\sum\_\{i\}\\delta\(i\)\. When conditioning these metrics on whether the issue was resolved or not, we exclude instances where the baseline has no resolved runs\.

### Experimental Protocol

For each instance we performN=20N=20runs on the unperturbed seed and one run on each ofN=20N=20sampled variants\. Every run, unperturbed or perturbed, executes in a freshly provisioned, isolated environment, so no state carries between runs\. Concretely, for each instance we extract the seed repository, gold\-patch file list, and issue description; execute theNNunperturbed runs; generateNNvariants with Algorithm[1](https://arxiv.org/html/2608.18389#alg1); and then, for each variant, provision a fresh environment, inject the variant, run the agent, and evaluate the resulting patch against the instance’s test oracle\. Algorithm[3](https://arxiv.org/html/2608.18389#alg3)in Appendix[C](https://arxiv.org/html/2608.18389#S3)gives the full procedure\.

##### Two sources of randomness\.

A perturbed run varies for two independent reasons\.*Variant randomness*: the sampler draws transformations, files, and sites at random, so variants of the same instance differ in difficulty\. WritingpVp\_\{V\}for the agent’s resolve probability on variantVV, this is the spreadσ2=Var⁡\(pV\)\\sigma^\{2\}=\\mathrm\{Var\}\(p\_\{V\}\)\.*Agent randomness*: with the variant held fixed, a single run is aBernoulli⁡\(pV\)\\mathrm\{Bernoulli\}\(p\_\{V\}\)draw, contributing average within\-variant noiseν=𝔼⁡\[pV​\(1−pV\)\]\\nu=\\mathbb\{E\}\\\!\\left\[p\_\{V\}\(1\-p\_\{V\}\)\\right\]\. The unperturbed condition carries only the second\. The quantityrp​\(i\)r\_\{p\}\(i\)targets isμ=𝔼⁡\[pV\]\\mu=\\mathbb\{E\}\[p\_\{V\}\], the resolve probability averaged over the variant population, and degradation contrasts it with the unperturbed resolve probability\.

##### One run per variant is the efficient split\.

With a budget ofRRperturbed runs per instance, the design choice is how to split them intoNNvariants ofKKruns each \(R=N​KR=NK\)\. Averaging the per\-variant rates is unbiased forμ\\muat everyKK, since a single run already has conditional meanpVp\_\{V\}; repetition tightens each per\-variant estimate around a center that is already correct rather than moving it\. The variance does depend on the split:Var⁡\(μ^\)=\(K​σ2\+ν\)/R\\mathrm\{Var\}\(\\hat\{\\mu\}\)=\(K\\sigma^\{2\}\+\\nu\)/R, strictly increasing inKKwheneverσ2\>0\\sigma^\{2\}\>0\. Intuitively, a run spent on a fresh variant samples both sources of randomness, whereas re\-running a variant resamples only the agent\. We therefore setN=R=20N=R=20,K=1K=1\. Appendix[E](https://arxiv.org/html/2608.18389#S5)gives the derivation\.

##### Statistical protocol\.

Our inference is*fixed\-population*: the instance set of each benchmark is held fixed and all intervals quantify run\-to\-run variability on those instances, not sampling of instances from the benchmark\. We refer to the unperturbed and perturbed runs of an instance as its two*conditions*\. For the configuration\-level means \(degradation, step count, and cost\) we report95%95\\%bootstrap percentile intervals fromB=20,000B=20\{,\}000resamples, resampling runs within each instance while leaving the instance set intact\. For degradation, we redraw each instance’s per\-run outcomes in each condition from a binomial model,Binomial⁡\(n,p^\)/n\\mathrm\{Binomial\}\(n,\\hat\{p\}\)/n, withp^\\hat\{p\}the observed resolve rate over that condition’snnruns\. For step count and cost, we draw the per\-run values within each condition with replacement and recompute the instance’s relative change\. For*per\-instance*degradation, a difference of two proportions, each estimated from2020runs, we instead report95%95\\%Newcombe intervals\([23](https://arxiv.org/html/2608.18389#bib.bib42);[9](https://arxiv.org/html/2608.18389#bib.bib41)\)\.

### RQ1: Degradation of Agent Performance

![Refer to caption](https://arxiv.org/html/2608.18389v1/charts/merged_degradation_hierarchical_ci.png)Figure 1:Mean degradationΔ¯\\bar\{\\Delta\}\(percentage points\) for each model under each scaffold on both benchmarks\. Error bars are fixed\-population 95% bootstrap confidence intervals\.##### Robustness rankings do not transfer across scaffolds or benchmarks\.

Figure[1](https://arxiv.org/html/2608.18389#Sx4.F1)gives the mean degradationΔ¯\\bar\{\\Delta\}for each scaffold\-model pair on each benchmark with its fixed\-population 95% bootstrap confidence interval\. Perturbation reduces resolve rates in most configurations:Δ¯\\bar\{\\Delta\}is positive in 13 of the 16, and the interval excludes zero in 6 of the 16\. No scaffold\-model pair is significant on both benchmarks, so the effect is configuration\-dependent rather than uniform\. No single ordering of models by robustness holds across the two scaffolds, and none holds across the two benchmarks either\. On SWE\-bench Verified, Qwen is the most robust model under mini\-SWE agent, degrading only 0\.2 points, yet the most brittle under OpenCode at 5\.5 points\. MiniMax moves the other way, from 2\.7 points under mini\-SWE agent down to 0\.5 under OpenCode\. The benchmark axis is just as jagged\. Opus under mini\-SWE agent degrades 1\.8 points on SWE\-bench Verified but 6\.7 points on Pro, the largest drop in the study, while under OpenCode on Pro it is the least degraded of the four models\. Kimi stays at or below one point in three of its four cells but is the most degraded OpenCode model on Pro at 4\.2 points\. Every model is among the most robust in at least one cell and among the most brittle in another\. A mean degradation measured on one scaffold and benchmark therefore does not predict behavior on another\. A practitioner who picks a model for how it holds up under mini\-SWE agent on SWE\-bench Verified may get the opposite outcome after switching scaffold or codebase\. We call this a jagged robustness frontier: robustness is a joint property of the model, the scaffold, and the workload, not of the model alone\.

##### The simpler scaffold is consistently more robust\.

Averaged over the four models, mini\-SWE agent degrades less than OpenCode on both benchmarks: 1\.34 points against 1\.88 on SWE\-bench Verified, and 1\.88 against 3\.65 on SWE\-bench Pro\. Its absolute strength, by contrast, is benchmark\-dependent\. It resolves 89\.05% of unperturbed runs on Verified against OpenCode’s 81\.20%, but 76\.35% on Pro against OpenCode’s 79\.33%\. The robustness gap survives this capability flip, which is what makes it informative\. The simpler scaffold loses less under perturbation\.

![Refer to caption](https://arxiv.org/html/2608.18389v1/charts/cost_step_summary_swe_bench.png)

![Refer to caption](https://arxiv.org/html/2608.18389v1/charts/cost_step_summary_swe_pro.png)

Figure 2:Mean relative change in agent steps \(δ¯step\\bar\{\\delta\}\_\{\\text\{step\}\}\) and cost \(δ¯cost\\bar\{\\delta\}\_\{\\text\{cost\}\}\) after perturbation, for each scaffold–model configuration on the 28 SWE\-bench Verified instances \(left\) and the 26 SWE\-bench Pro instances \(right\), restricted to runs the agent resolved in both cases\. Positive values mean perturbed runs take more steps, or cost more, than unperturbed runs on the same instance\. Error bars are fixed\-population 95% bootstrap confidence intervals\.
##### Degradation is concentrated, not diffuse\.

Appendix[D](https://arxiv.org/html/2608.18389#S4.SSx2)reports the per\-instance degradationΔ⁡\(i\)\\Delta\(i\)with its Newcombe 95% interval for every instance\-configuration\. At the level of an individual instance the comparison is underpowered\. With 20 runs per condition, only swings of roughly 25 points or more can exclude zero\. Only 14 out of 432 instance\-configuration items exclude zero in their interval\. The mass of point estimates is shifted toward positive values\. The aggregate effect is carried by a small heavy tail of instance\-configuration items\. Three instances alone account for roughly two\-thirds of the 6\.7\-point mean degradation of mini\-SWE agent with Opus on SWE\-bench Pro\. The concentration mirrors the jaggedness we saw at the aggregate level\. No instance degrades under all eight configurations on either benchmark, and a few cells swing sharply the other way\. Brittleness attaches to particular instance\-configuration pairs rather than to instances, models, or repositories alone\.

### RQ2: Effort Induced by Perturbation

##### Perturbation inflates effort more broadly than it degrades outcomes\.

Figure[2](https://arxiv.org/html/2608.18389#Sx4.F2)reportsδ¯step\\bar\{\\delta\}\_\{\\text\{step\}\}andδ¯cost\\bar\{\\delta\}\_\{\\text\{cost\}\}for each configuration on both benchmarks, restricted to the runs the agent resolved, so that the comparison cannot be driven by long failing runs\. On SWE\-bench Verified, cost rises in all 8 configurations, from 4\.0% \(mini\-SWE agent with Opus\) to 22\.9% \(OpenCode with MiniMax\), and the interval excludes zero in every one\. On SWE\-bench Pro the increments are smaller and concentrated in the richer scaffold\. All four OpenCode configurations pay 8\.8–12\.6% more with intervals excluding zero, while the four mini\-SWE agent configurations sit near zero\. The same holds over all runs \(Appendix[D](https://arxiv.org/html/2608.18389#S4.SSx1)\): cost rises in all eight Verified configurations and seven of eight on Pro\. Because the trend is already present when only successful runs are counted, it is not an artifact of failure\. Measured by resolve rates, most configurations are impacted modestly by the perturbations, but, measured by what the agent spends, the impact is more drastic\.

##### The agent spends more per step, not only more steps\.

Steps and cost do not move together\. Cost per step rises in twelve of the sixteen configurations\. On SWE\-bench Verified, step overhead never exceeds 9\.9% while cost overhead reaches 22\.9%\. The agent is not merely taking more turns after a rewrite\. It is consuming more context on each turn, consistent with longer files and heavier reading under perturbation\. Among runs the agent resolved, input tokens per step rise in 13 of the 16 configurations\. The scaffold contrast from the degradation analysis reappears here on Pro\. Restricted to resolved instances, the four OpenCode configurations pay 4\.6–7\.2% more in cost per step, while the mini\-SWE agent configurations are flat or slightly cheaper\.

##### Opus compresses its trajectory rather than extending it\.

Opus under mini\-SWE agent on SWE\-bench Verified responds to perturbation by taking*fewer*steps\. This behavior is uniform\. For every instance, it takes fewer steps on perturbed runs \(mean−19\.8%\-19\.8\\%, range−6%\-6\\%to−39%\-39\\%\)\. But output tokens per step more than double, causing cost to rise on every instance \(median\+101%\+101\\%\), while input tokens per step fall by 17%\. Rather than taking more turns, Opus emits fewer, denser actions, for a net\+30\.4%\+30\.4\\%cost per step and\+4\.0%\+4\.0\\%total cost\. This behavior is also benchmark\-specific\. On SWE\-bench Pro the step reduction shrinks to \(−1\.29%\-1\.29\\%\)\.

### RQ3: Observed Failure Patterns

We manually analyze 20 agent trajectories across both scaffolds, both benchmarks \(14 distinct instances\), and all 4 models to check for behavioral patterns and how SPTs influence agent handling of tasks\. These tasks are selected for displaying high variance \(3\), high degradation \(3\), large increase in step count \(1\), file coverage \(2\) or cost \(1\)\. Additionally, cases with one mutant failure out of 20 \(2\) and those with high likelihood of SPT interference \(8\) are also picked, with a focus on covering a diverse set of model\-scaffold\-benchmark combinations\. Details on selection can be found in Appendix[F](https://arxiv.org/html/2608.18389#S6.SSx1)\. Here, we present 4 interesting behavior patterns that we observed\. For additional observations, refer to Appendix[F](https://arxiv.org/html/2608.18389#S6.SSx2)\.

##### Detecting and reverting SPTs\.

In the cases where there is significant overlap between the SPT edited files and the files that the agent touches, there are instances where the agent recognizes that the code is "obfuscated"\. For example, in one case, with MiniMax on Qutebrowser and mini\-SWE agent, the agent checks the commit history and spots that the transformations were added in the latest commit\. It then reverts the repository state before it makes the final changes so that all SPTs are cleared\.

##### Degradation of code localization capabilities\.

SPTs may make it harder for agents to localize relevant sections of the code, either by diluting grep results or making core files lose credibility\. For example, Opus with mini\-SWE agent on Openlibrary tries to grep for a keyword in the issue description, but ends up getting thrown off by multiple noisy dead string assignments which use the same keyword and turn up in the search\.

##### Editing/"fixing" of perturbed code\.

The agent occasionally simplifies the perturbed code during patching, for example, Opus with mini\-SWE agent on Qutebrowser mentions that it has recognized and fixed the perturbed code in multiple places\. This additional editing may also increase the risk of the patch failing due to the increased volume of changes\.

##### Corrupted patch validation\.

With Kimi on Pytest\-dev, one of the unit tests in the test suite fails\. The part of the code the unit test refers to contains a string literal splitter in one of the logs\. The agent gets confused and incorrectly attributes the failure to the split string, reasons that this was not due to an edit it made, and then submits the incorrect patch without attempting to resolve the error\.

## Related Work

##### Robustness of code language models\.

A substantial body of work shows that neural models of code are brittle to surface\-level perturbations that preserve semantics\. Identifier renaming attacks have been demonstrated against neural code models\([38](https://arxiv.org/html/2608.18389#bib.bib13)\), formalized under robustness constraints\([2](https://arxiv.org/html/2608.18389#bib.bib14)\), and scaled to black\-box settings against pretrained models like CodeBERT\([37](https://arxiv.org/html/2608.18389#bib.bib15)\)\. Structural perturbations such as dead code insertion and operator substitution have been used to attack program classifiers\([30](https://arxiv.org/html/2608.18389#bib.bib16)\)\.[28](https://arxiv.org/html/2608.18389#bib.bib17)combined renaming, dead code, and operand swapping to expose fragility across method\-name prediction and code summarization\.

On the defense side, contrastive learning on equivalent programs\([13](https://arxiv.org/html/2608.18389#bib.bib18)\)and naturalizing transformations as a pretraining objective\([3](https://arxiv.org/html/2608.18389#bib.bib19)\)have been shown to improve robustness\.

At the benchmark level, perturbations to docstrings, signatures, and syntax have been used to evaluate code generation models\([35](https://arxiv.org/html/2608.18389#bib.bib20)\), attack frameworks have been proposed across defect detection and clone detection\([15](https://arxiv.org/html/2608.18389#bib.bib21)\), and minor refactorings in prompt context have been shown to significantly alter Copilot’s completions\([18](https://arxiv.org/html/2608.18389#bib.bib22)\)\.

All of the above evaluate single\-turn model inferences on classification or short\-generation tasks where the perturbation is applied to a fixed\-length input\. Our work extends this to multi\-turn agents that localize faults, navigate entire repositories, and synthesize patches across dozens of reasoning steps\. The perturbations occur randomly over the codebase\.

##### Adversarial robustness of LLMs\.

LLMs are fragile to semantics\-preserving transformations of general natural\-language inputs\. Prompt formatting changes alone \(spacing, delimiters, option ordering\) cause up to 76\-point accuracy swings\([29](https://arxiv.org/html/2608.18389#bib.bib23)\), and systematic perturbation benchmarks confirm this across character\-, word\-, and sentence\-level attacks\([42](https://arxiv.org/html/2608.18389#bib.bib24);[33](https://arxiv.org/html/2608.18389#bib.bib33)\)\.

The phenomenon extends to structured problem solving\. GSM\-Symbolic\([22](https://arxiv.org/html/2608.18389#bib.bib25)\)and GSM\-Plus\([17](https://arxiv.org/html/2608.18389#bib.bib26)\)create variants of math benchmarks by changing surface features \(names, numbers, phrasing\) while preserving problem structure, observing up to 10% performance drops\. Similarly, reordering multiple\-choice options\([40](https://arxiv.org/html/2608.18389#bib.bib27);[6](https://arxiv.org/html/2608.18389#bib.bib31)\)or generating structurally equivalent reasoning problems\([41](https://arxiv.org/html/2608.18389#bib.bib28)\)significantly shifts model rankings\. Formal certification of LLM robustness has also been proposed\([4](https://arxiv.org/html/2608.18389#bib.bib34);[34](https://arxiv.org/html/2608.18389#bib.bib32);[5](https://arxiv.org/html/2608.18389#bib.bib36)\)\.

Geirhos et al\.\([10](https://arxiv.org/html/2608.18389#bib.bib29)\)frame these failures as*shortcut learning*, where networks exploit spurious surface statistics rather than learning the intended decision rule\. Dziri et al\.\([8](https://arxiv.org/html/2608.18389#bib.bib30)\)provide evidence that transformers linearize compositional reasoning, offering a mechanistic explanation for why surface changes can derail multi\-step tasks\.

Our findings echo this pattern in the code domain\. We show coding agents are sensitive to how the surrounding code is structured\. A further dimension absent from prior work is the scaffold interaction\. We find that robustness rankings can change when the same model is embedded in a different agent architecture\. Robustness certification for agentic systems has also been studied for simple, non\-code tool\-selection\([39](https://arxiv.org/html/2608.18389#bib.bib35)\)\.

## Conclusion

In this work, we contribute a library of local, semantics\-preserving transformations \(SPTs\) for code repositories and a randomized sampler that draws semantically equivalent variants of a repository without feedback from the agent; an experimental methodology that isolates the perturbation effect from the intrinsic stochasticity of LLM\-based agents through paired seed\-and\-variant runs while balancing statistical validity against the cost of agentic runs; and, to our knowledge, the first systematic robustness evaluation of repository\-level code agents\. Our experiments reveal a jagged robustness frontier across models, scaffolds, and repositories\.

## Generative AI Usage Statement

Generative AI tools were used in the preparation of this manuscript to improve the clarity of the writing, to assist in writing Python scripts for the experiments, and to develop the arguments in Section E\. Everything was carefully reviewed by the authors and they remain fully accountable for this work\.

## References

- Anthropic \(2025\)AnthropicIntroducing Claude Opus 4\.5\.Note:https://www\.anthropic\.com/news/claude\-opus\-4\-5System card:https://assets\.anthropic\.com/m/64823ba7485345a7/Claude\-Opus\-4\-5\-System\-Card\.pdfCited by:[Agents and models\.](https://arxiv.org/html/2608.18389#Sx4.SSx2.SSS0.Px2.p1.1)\.
- Bielik and Vechev \(2020\)P\. Bielik and M\. VechevAdversarial robustness for code\.InProceedings of the 37th International Conference on Machine Learning \(ICML\),Cited by:[Robustness of code language models\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px1.p1.1)\.
- Chakrabortyet al\.\(2022\)S\. Chakraborty, T\. Ahmed, Y\. Ding, P\. Devanbu, and B\. RayNatGen: generative pre\-training by “naturalizing” source code\.InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering \(ESEC/FSE\),Cited by:[Robustness of code language models\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px1.p2.1)\.
- Chaudharyet al\.\(2025a\)I\. Chaudhary, Q\. Hu, M\. Kumar, M\. Ziyadi, R\. Gupta, and G\. SinghCertifying counterfactual bias in LLMs\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Adversarial robustness of LLMs\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px2.p2.1)\.
- Chaudharyet al\.\(2025b\)I\. Chaudhary, V\. Jain, P\. Parhar, K\. Sachdeva, A\. Singh, S\. Ranu, and G\. SinghLumos: let there be language model system certification\.arXiv preprint arXiv:2512\.02966\.Cited by:[Adversarial robustness of LLMs\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px2.p2.1)\.
- Chaudharyet al\.\(2024\)I\. Chaudhary, V\. V\. Jain, and G\. SinghCertifying knowledge comprehension in LLMs\.arXiv preprint arXiv:2402\.15929\.Cited by:[Adversarial robustness of LLMs\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px2.p2.1)\.
- Denget al\.\(2025\)X\. Deng, J\. Da, E\. Pan, Y\. Y\. He, C\. Ide, K\. Garg, N\. Lauffer, A\. Park, N\. Pasari, C\. Rane,et al\.Swe\-bench pro: can ai agents solve long\-horizon software engineering tasks?\.arXiv preprint arXiv:2509\.16941\.Cited by:[Introduction](https://arxiv.org/html/2608.18389#Sx1.p1.1),[Benchmarks and instance selection\.](https://arxiv.org/html/2608.18389#Sx4.SSx2.SSS0.Px1.p1.1)\.
- Dziriet al\.\(2023\)N\. Dziri, X\. Lu, M\. Sclar, X\. L\. Li, L\. Jiang, B\. Y\. Lin, P\. West, C\. Bhagavatula, R\. Le Bras, J\. D\. Hwang,et al\.Faith and fate: limits of transformers on compositionality\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[Adversarial robustness of LLMs\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px2.p3.1)\.
- Fagerlandet al\.\(2015\)M\. W\. Fagerland, S\. Lydersen, and P\. LaakeRecommended confidence intervals for two independent binomial proportions\.Statistical methods in medical research24\(2\),pp\. 224–254\.Cited by:[Statistical protocol\.](https://arxiv.org/html/2608.18389#Sx4.SSx4.SSS0.Px3.p1.1)\.
- Geirhoset al\.\(2020\)R\. Geirhos, J\. Jacobsen, C\. Michaelis, R\. Zemel, W\. Brendel, M\. Bethge, and F\. A\. WichmannShortcut learning in deep neural networks\.Nature Machine Intelligence2\(11\),pp\. 665–673\.Cited by:[Introduction](https://arxiv.org/html/2608.18389#Sx1.p1.1),[Adversarial robustness of LLMs\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px2.p3.1)\.
- GitHub \(2025\)GitHubOctoverse 2025: the state of open source and rise of AI\.Note:https://github\.blog/news\-insights/octoverse/Accessed: 2026\-07\-22Cited by:[Introduction](https://arxiv.org/html/2608.18389#Sx1.p1.1)\.
- Google DORA \(2025\)Google DORA2025 dora report: state of AI\-assisted software development\.Note:https://dora\.dev/research/2025/Accessed: 2026\-07\-22Cited by:[Introduction](https://arxiv.org/html/2608.18389#Sx1.p1.1)\.
- Jainet al\.\(2021\)P\. Jain, A\. Jain, T\. Zhang, P\. Abbeel, J\. Gonzalez, and I\. StoicaContrastive code representation learning\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[Robustness of code language models\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px1.p2.1)\.
- JetBrains \(2025\)JetBrainsThe state of developer ecosystem 2025\.Note:https://devecosystem\-2025\.jetbrains\.com/Accessed: 2026\-07\-22Cited by:[Introduction](https://arxiv.org/html/2608.18389#Sx1.p1.1)\.
- Jha and Reddy \(2023\)A\. Jha and C\. K\. ReddyCodeAttack: code\-based adversarial attacks for pre\-trained programming language models\.InProceedings of the AAAI Conference on Artificial Intelligence,Cited by:[Robustness of code language models\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px1.p3.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. NarasimhanSwe\-bench: can language models resolve real\-world github issues?\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 54107–54157\.Cited by:[Validation](https://arxiv.org/html/2608.18389#Sx2.SSx3.p1.1),[Benchmarks and instance selection\.](https://arxiv.org/html/2608.18389#Sx4.SSx2.SSS0.Px1.p1.1)\.
- Liet al\.\(2024\)Q\. Li, L\. Cui, X\. Zhao, L\. Kong, and W\. BiGSM\-Plus: a comprehensive benchmark for evaluating the robustness of LLMs as mathematical problem solvers\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[Adversarial robustness of LLMs\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px2.p2.1)\.
- Mastropaoloet al\.\(2023\)A\. Mastropaolo, L\. Pascarella, E\. Guglielmi, M\. Ciniselli, S\. Scalabrino, R\. Oliveto, and G\. BavotaOn the robustness of code generation techniques: an empirical study on GitHub Copilot\.InProceedings of the 45th International Conference on Software Engineering \(ICSE\),pp\. 2149–2160\.Cited by:[Robustness of code language models\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px1.p3.1)\.
- McKeeman \(1998\)W\. M\. McKeemanDifferential testing for software\.Digital Technical Journal10\(1\),pp\. 100–107\.Cited by:[Validation](https://arxiv.org/html/2608.18389#Sx2.SSx3.p1.1)\.
- Miller \(1995\)G\. A\. MillerWordNet: a lexical database for english\.Communications of the ACM38\(11\),pp\. 39–41\.Cited by:[§A](https://arxiv.org/html/2608.18389#S1.SSx1.p8.1)\.
- MiniMax AI \(2026\)MiniMax AIMiniMax\-M2\.5\.Note:https://huggingface\.co/MiniMaxAI/MiniMax\-M2\.5Cited by:[Agents and models\.](https://arxiv.org/html/2608.18389#Sx4.SSx2.SSS0.Px2.p1.1)\.
- Mirzadehet al\.\(2024\)I\. Mirzadeh, K\. Alizadeh, H\. Shahrokhi, O\. Tuzel, S\. Bengio, and M\. FarajtabarGSM\-Symbolic: understanding the limitations of mathematical reasoning in large language models\.arXiv preprint arXiv:2410\.05229\.Cited by:[Adversarial robustness of LLMs\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px2.p2.1)\.
- Newcombe \(1998\)R\. G\. NewcombeInterval estimation for the difference between independent proportions: comparison of eleven methods\.Statistics in medicine17\(8\),pp\. 873–890\.Cited by:[Statistical protocol\.](https://arxiv.org/html/2608.18389#Sx4.SSx4.SSS0.Px3.p1.1)\.
- OpenAI \(2024\)OpenAIIntroducing SWE\-bench Verified\.Note:https://openai\.com/index/introducing\-swe\-bench\-verified/Accessed: 2026\-07\-10Cited by:[Introduction](https://arxiv.org/html/2608.18389#Sx1.p1.1),[Benchmarks and instance selection\.](https://arxiv.org/html/2608.18389#Sx4.SSx2.SSS0.Px1.p1.1)\.
- OpenCode Contributors \(2024\)OpenCode ContributorsOpenCode: an open\-source ai coding agent for the terminal\.\.Note:https://github\.com/anomalyco/opencodeAccessed: 2026\-07\-10Cited by:[Agents and models\.](https://arxiv.org/html/2608.18389#Sx4.SSx2.SSS0.Px2.p1.1)\.
- Pylint contributors \(2026\)Pylint contributorsastroid: an abstract syntax tree library for Python\.Note:https://github\.com/pylint\-dev/astroidCited by:[§A](https://arxiv.org/html/2608.18389#S1.SSx1.p7.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.6\-27B: flagship\-level coding in a 27b dense model\.External Links:[Link](https://qwen.ai/blog?id=qwen3.6-27b)Cited by:[Agents and models\.](https://arxiv.org/html/2608.18389#Sx4.SSx2.SSS0.Px2.p1.1)\.
- Ramakrishnanet al\.\(2022\)G\. Ramakrishnan, J\. Henkel, Z\. Wang, A\. Albarghouthi, S\. Jha, and T\. RepsSemantic robustness of models of source code\.InIEEE International Conference on Software Analysis, Evolution and Reengineering \(SANER\),pp\. 526–537\.Cited by:[Introduction](https://arxiv.org/html/2608.18389#Sx1.p1.1),[Robustness of code language models\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px1.p1.1)\.
- Sclaret al\.\(2024\)M\. Sclar, Y\. Choi, Y\. Tsvetkov, and A\. SuhrQuantifying language models’ sensitivity to spurious features in prompt design\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Adversarial robustness of LLMs\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px2.p1.1)\.
- Srikantet al\.\(2021\)S\. Srikant, S\. Liu, T\. Mitrovska, S\. Chang, Q\. Fan, G\. Zhang, and U\. O’ReillyGenerating adversarial computer programs using optimized obfuscations\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Introduction](https://arxiv.org/html/2608.18389#Sx1.p2.1),[Robustness of code language models\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px1.p1.1)\.
- Stack Overflow \(2025\)Stack Overflow2025 stack overflow developer survey\.Note:https://survey\.stackoverflow\.co/2025/Accessed: 2026\-07\-22Cited by:[Introduction](https://arxiv.org/html/2608.18389#Sx1.p1.1)\.
- Teamet al\.\(2026\)K\. Team, T\. Bai, Y\. Bai, Y\. Bao, S\. Cai, Y\. Cao, Y\. Charles, H\. Che, C\. Chen, G\. Chen,et al\.Kimi k2\. 5: visual agentic intelligence\.arXiv preprint arXiv:2602\.02276\.Cited by:[Agents and models\.](https://arxiv.org/html/2608.18389#Sx4.SSx2.SSS0.Px2.p1.1)\.
- Vegaet al\.\(2023\)J\. Vega, I\. Chaudhary, C\. Xu, and G\. SinghBypassing the safety training of open\-source LLMs with priming attacks\.arXiv preprint arXiv:2312\.12321\.Cited by:[Adversarial robustness of LLMs\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px2.p1.1)\.
- Wanget al\.\(2026\)C\. Wang, I\. Chaudhary, Q\. Hu, W\. Ruan, R\. Gupta, and G\. SinghHow catastrophic is your LLM? certifying risks in conversation\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Adversarial robustness of LLMs\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px2.p2.1)\.
- Wanget al\.\(2023\)S\. Wang, Z\. Li, H\. Qian, C\. Yang, Z\. Wang, M\. Shang, V\. Kumar, S\. Tan, B\. Ray, P\. Bhatia,et al\.ReCode: robustness evaluation of code generation models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[Introduction](https://arxiv.org/html/2608.18389#Sx1.p1.1),[Robustness of code language models\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px1.p3.1)\.
- Yanget al\.\(2024\)J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. R\. Narasimhan, and O\. PressSWE\-agent: agent\-computer interfaces enable automated software engineering\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2405.15793)Cited by:[Agents and models\.](https://arxiv.org/html/2608.18389#Sx4.SSx2.SSS0.Px2.p1.1)\.
- Yanget al\.\(2022\)Z\. Yang, J\. Shi, J\. He, and D\. LoNatural attack for pre\-trained models of code\.InProceedings of the 44th International Conference on Software Engineering \(ICSE\),pp\. 1482–1493\.Cited by:[Robustness of code language models\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px1.p1.1)\.
- Yefetet al\.\(2020\)N\. Yefet, U\. Alon, and E\. YahavAdversarial examples for models of code\.Proceedings of the ACM on Programming Languages \(OOPSLA\)4\.Cited by:[Introduction](https://arxiv.org/html/2608.18389#Sx1.p1.1),[Robustness of code language models\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px1.p1.1)\.
- Yeonet al\.\(2025\)J\. Yeon, I\. Chaudhary, and G\. SinghQuantitative certification of agentic tool selection\.arXiv preprint arXiv:2510\.03992\.Cited by:[Adversarial robustness of LLMs\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px2.p4.1)\.
- Zhenget al\.\(2024\)C\. Zheng, H\. Zhou, F\. Meng, J\. Zhou, and M\. HuangLarge language models are not robust multiple choice selectors\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Adversarial robustness of LLMs\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px2.p2.1)\.
- Zhuet al\.\(2024\)K\. Zhu, J\. Chen, J\. Wang, N\. Z\. Gong, D\. Yang, and X\. XieDyVal: dynamic evaluation of large language models for reasoning tasks\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Adversarial robustness of LLMs\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px2.p2.1)\.
- Zhuet al\.\(2023\)K\. Zhu, J\. Wang, J\. Zhou, Z\. Wang, H\. Chen, Y\. Wang, L\. Yang, W\. Ye, Y\. Zhang, N\. Z\. Gong, and X\. XiePromptRobust: towards evaluating the robustness of large language models on adversarial prompts\.arXiv preprint arXiv:2306\.04528\.Cited by:[Adversarial robustness of LLMs\.](https://arxiv.org/html/2608.18389#Sx5.SSx7.SSS0.Px2.p1.1)\.

## ASemantics\-preserving Transformation Implementation Details

The appendix details the implementation of each SPT\. It describes the applicability condition of each SPT that restricts the application of the SPT to the candidate sites\. Writing these conditions is where the difficulty lies\. They must hold across diverse repositories, where the same construct appears in many different implementations, so a condition has to rule out every unsafe variant rather than the common ones\. Writing an SPT for a function\-level code snippet is comparatively easy\. The snippet is short and self\-contained, its bindings and callers are visible in the code being rewritten, and the only observable behavior is the value it returns\.

None of this holds in a repository\. Names resolve through nested scopes and imports, so a builtin such asiter,next, orExceptionmay be shadowed at the rewrite site\. Types are not written down either\. Addition commutes on numbers, but not on strings, lists, or a class with a custom\_\_add\_\_\. A program can also observe its own structure\. It can do so throughlocals\(\), frame introspection, metaclasses, attribute\-dispatch hooks, and framework base classes that fix what a class namespace may contain\. Code that never runs can therefore still change behavior\. Scale makes this more difficult\. A single variant transforms thousands of sites in a repository\. So a rule that is not semantics\-preserving in a rare case will meet that case\. We therefore keep every SPT conservative\. When an applicability condition cannot be established from the code, we skip the site\.

We group the transformations of Table[1](https://arxiv.org/html/2608.18389#Sx2.T1)by the operation they perform: rewrites of existing structure and insertions of inert code\.

### Rewrites

If Else Switchernegates the condition and exchanges the two branches\. It is applied only toifstatements that have a directelseclause\. Anifwhoseelsebranch is anotherif\(i\.e\. anelif\) is not switched\. Although anelifclause carrying its ownelseis itself a valid target\. This SPT is also applied to inline ternary expressions, where anelseis syntactically required\. The condition, and in the ternary case both result expressions, are wrapped in parentheses before negation to maintain correct operator precedence\.

And Condition Splitterdecomposes a compoundifstatement whose condition is a Boolean conjunction \(A and B\) into semantically equivalent nestedifstatements\. When the statement has anelseclause, that clause is duplicated onto both levels so that failure of either conjunct reaches it\.

Comparison Swapperexchanges the operands of an asymmetric relational comparison\(<, \>, <=, \>=\)and inverts the operator to preserve semantic equivalence\. It is restricted to simple two\-operand comparisons: chained comparisons and symmetric operators are excluded\.

Double Negation Injectorwraps aniforwhilecondition innot not \(Condition\)\. It skips literalTrue/Falseconditions and conditions that are themselves a top\-level negation, so that a condition such asnot xis left alone while a compound condition that merely contains anot\(e\.g\.not a and b\) remains eligible\.

While Loop Unrollingunrolls one iteration of awhileloop: the loop body is duplicated at the head of the new body, followed by an innerwhilecarrying the original test and body, followed by an unconditionalbreak\. The outer test is therefore evaluated the same number of times as in the original loop, which matters when the test has side effects\. Only loops with an indented block body are transformed, and loops carrying anelseclause are excluded, since the injectedbreakwould suppress it\.

For Loop Rewritingrewrites aforloop into a semantically equivalentwhile Trueloop using the explicit iterator protocol\. It materializes an iterator withiter\(\.\.\.\)and advances it withnext\(\.\.\.\)inside atry/except StopIteration: breakblock\. The introduced iterator and flag names carry a random suffix so they cannot collide with existing bindings\. It also emulates thefor \.\.\. elsecontract via a sentinel flag\. Specifically, eachbreakstatement belonging to the rewritten loop—breaks inside nested loops are left untouched—is rewritten toflag = False; break, so that the flag correctly records whether the loop exited early\. If the iterable is an implicit tuple, it is explicitly parenthesized before being passed toiter\(\)to avoid the two\-argumentiter\(callable, sentinel\)misinterpretation\. The transformation is skipped when any ofiter,next, orStopIterationis shadowed by a binding in an enclosing scope\.async forloops are excluded, as their asynchronous protocol cannot be emulated with synchronousiter\(\)/next\(\)\.

Commutative Operand Permuterswaps the operands of commutative binary operations: addition, multiplication, bitwise AND, OR, and XOR\. Because commutativity holds for these operators only over specific types in Python \(e\.g\., string and list\+are order\-dependent, and a user\-defined\_\_add\_\_/\_\_radd\_\_need not commute\), the transformation performs type inference using Astroid\([26](https://arxiv.org/html/2608.18389#bib.bib2)\)before swapping\. The swap is applied only when both operands are independently inferred to lie within the allowed type set for that operator\. If inference is inconclusive or the inferred type lies outside the allowed set, the transformation is skipped\. Logicaland/orare never permuted\.

Local Variable Renamerrenames local variables to semantically related synonyms\. Candidates are the targets of plain assignment statements inside function bodies\. Module\-, class\-, lambda\- and comprehension\-level bindings are never renamed\. Candidate names are looked up in WordNet\([20](https://arxiv.org/html/2608.18389#bib.bib3)\)for a synonym\. If none is suitable, or the synonym would collide with an identifier already present in the module, a random five\-letter lowercase name is generated\. All occurrences within the scope of the binding are updated consistently, including nested scopes that do not rebind the name\. We apply a conservative exclusion list to avoid renaming anything whose identity may be externally meaningful or whose rename could not be confined to one scope: built\-ins, keywords, very short names, function names, parameters, imported and decorator names, names declaredglobalornonlocal, loop targets, and variables used in a call position, as an argument, or as part of an attribute access or the base of a subscript\. We additionally exclude variables that alias another name \(a = b\), variables assigned the result of a call, and any variable bound more than once in its scope, since control flow then determines which binding a reference observes\.

String Literal Splitterfragments a string literal into a\+\-joined concatenation at a randomly chosen split point, exploiting that Python evaluates the concatenation of literals to the original value\. Because the literal no longer appears as one contiguous token, this frustrates localization strategies that search for the whole string\. Splitting is skipped for literals inside type annotations, prefixed byte/raw/unicode strings \(escape semantics\), strings shorter than a safe split length of four characters, literals used as the base of an attribute access, and literals already part of an implicit concatenation, a non\-\+binary operation, or amatchpattern\. Formatted \(f\-\)strings are never targeted\.

### Inert Insertions

These transformations add code that would never alter program behavior\. Two of them—Dead String Assignment and Dead Method Injection—insert code that is inert but lexically salient, forcing an agent that relies on surface cues such as identifier or method names to distinguish genuine edit sites from decoys\.

If True Wrapperencloses an indented block in a guard built from a grammar of randomly generated tautologies, whose complexity is set by a depth parameter\. Because the guard always evaluates toTrue, every wrapped statement still executes\. Candidate blocks are those spanning between 5 and 100 lines, and blocks whose first statement is a docstring are skipped\.

Try Except Injectorwraps an indented block intry \.\.\. except Exception: raise\. Theexceptclause unconditionally re\-raises any exception it catches, so all exceptions are propagated to the same handler as in the unmodified code\. We use theexcept Exception: raisepattern instead ofexcept: raiseto avoid catchingBaseExceptionsubclasses such asKeyboardInterruptandSystemExit\. As with the previous transformation, candidates are blocks of 5 to 100 lines and blocks whose first statement is a docstring are skipped; injection is additionally skipped whenExceptionis shadowed by a binding at the injection site, which would otherwise make the handler itself raiseTypeError\.

Dead\-Code Injectorinserts anif Falseblock containing a dummy assignment at the head of an indented block\. The guard renders the block statically unreachable\. When the block begins with a docstring, the injection is placed after it so that\_\_doc\_\_is preserved\.

Dead String Assignmentinjects an unread assignment of the form<name\> = "<keyword\>"before selected statements inside a function body\. The keyword\-like string serves as a distractor resembling content an agent might search for, while remaining a no\-op\. The<name\>is drawn from a fixed pool, chosen so as not to shadow any identifier used in the enclosing function, and is never read\. Injection never displaces a leading docstring and is not placed after an unconditional terminator \(return/raise/break/continue\) in the same block\. Any function that reflects on its local namespace is excluded, as introducing a new binding would be observable: this coverslocals\(\), argument\-lessvars\(\),eval/execcalls that do not pass explicit globals and locals, and frame introspection viasys\.\_getframe,inspect\.currentframe,inspect\.stack, and related APIs\.

Dead Method Injectionappends a structurally complete method,def <name\>\(self, \*args, \*\*kwargs\)whose entire body is guarded byif False, to a class body, serving as a highly plausible edit decoy\. The method is callable but its body is unreachable, so it always returnsNonewithout side effects\. Crucially, appending a method is not unconditionally behavior\-preserving, because a class may observe its own attribute namespace\. The transformation is therefore skipped whenever a class controls that namespace: classes declaring a custom metaclass, classes that override attribute dispatch \(\_\_getattr\_\_,\_\_getattribute\_\_,\_\_setattr\_\_,\_\_delattr\_\_\), property\-only namespaces, sentinel classes whose body is a barepassor\.\.\., and classes whose base list mentions an attribute\-constraining framework type \(e\.g\., PydanticBaseModel,Enum,NamedTuple,TypedDict,Protocol,ABC\), matched by the simple name of the base expression\. A class is also skipped when it already declares the target name as a method or class attribute, and dunder target names are excluded outright\.

## BSampler Procedure Details

Algorithm[2](https://arxiv.org/html/2608.18389#alg2)describes the sub\-procedure invoked by the sampler described in Algorithm[1](https://arxiv.org/html/2608.18389#alg1)\.

Algorithm 2Target Assignment1:Selected transformations

𝒯selected\\mathcal\{T\}\_\{\\text\{selected\}\}, Keyword map

𝒦\\mathcal\{K\}, Maximum number of keywords to select

NkN\_\{k\}
2:Bound \(transformation, target\) pairs

ℬ\\mathcal\{B\}
3:procedureAssignTargetNames\(

𝒯s​e​l​e​c​t​e​d,𝒦,Nk\\mathcal\{T\}\_\{selected\},\\mathcal\{K\},N\_\{k\}\)

4:

ℬ←∅\\mathcal\{B\}\\leftarrow\\emptyset
5:foreachtransformation

t∈𝒯selectedt\\in\\mathcal\{T\}\_\{\\text\{selected\}\}do

6:if

t∉𝒯kwt\\notin\\mathcal\{T\}\_\{\\text\{kw\}\}or

𝒦=None\\mathcal\{K\}=\\textsc\{None\}then

7:

ℬ←ℬ∪\{\(t,None\)\}\\mathcal\{B\}\\leftarrow\\mathcal\{B\}\\cup\\\{\(t,\\textsc\{None\}\)\\\}⊳\\trianglerightno target required

8:else

9:

𝑛𝑎𝑚𝑒𝑠←𝒦⁡\[t\]\\mathit\{names\}\\leftarrow\\mathcal\{K\}\[t\]⊳\\trianglerightcandidates extracted fortt

10:if

𝑛𝑎𝑚𝑒𝑠=∅\\mathit\{names\}=\\emptysetthen

11:

ℬ←ℬ∪\{\(t,None\)\}\\mathcal\{B\}\\leftarrow\\mathcal\{B\}\\cup\\\{\(t,\\textsc\{None\}\)\\\}
12:else

13:

𝑡𝑎𝑟𝑔𝑒𝑡𝑒𝑑𝐾𝑒𝑦𝑤𝑜𝑟𝑑𝑠←RandomSelect​\(𝑛𝑎𝑚𝑒𝑠,min⁡\(Nk,\|𝑛𝑎𝑚𝑒𝑠\|\)\)\\mathit\{targetedKeywords\}\\leftarrow\\textsc\{RandomSelect\}\(\\mathit\{names\},\\min\(N\_\{k\},\|\\mathit\{names\}\|\)\)
14:foreach

τ∈𝑡𝑎𝑟𝑔𝑒𝑡𝑒𝑑𝐾𝑒𝑦𝑤𝑜𝑟𝑑𝑠\\tau\\in\\mathit\{targetedKeywords\}do

15:

ℬ←ℬ∪\{\(t,τ\)\}\\mathcal\{B\}\\leftarrow\\mathcal\{B\}\\cup\\\{\(t,\\tau\)\\\}
16:endfor

17:endif

18:endif

19:endfor

20:return

ℬ\\mathcal\{B\}
21:endprocedure

## CExperimental Procedure Details

#### Sampling Procedure

To construct our task sample, we first identified instances resolved by Claude Opus 4\.5 using official SWE\-bench Verified leaderboard results, ensuring that our sampled tasks were solvable under standard \(unperturbed\) conditions\. We then performed stratified sampling over repository–difficulty combinations, drawing one instance per stratum to ensure balanced coverage of the instance space\.

Since official instance wise results were unavailable for SWE\-bench pro, we started by sampling an equal number of instances from each of the repos, and running Claude Opus 4\.5 on the set\. Then, 26 passing instances were sampled across all repos that contained at least one valid python file to which the SPTs could be applied\.

### Full Experimental Procedure

Algorithm[3](https://arxiv.org/html/2608.18389#alg3)gives the full experimental procedure summarized in the Protocol paragraph of the experiments section\. For each instance it proceeds in four phases: seed retrieval, baseline evaluation \(NNruns on the unperturbed seed\), variant generation via the sampler of Algorithm[1](https://arxiv.org/html/2608.18389#alg1)of the variant sampler section, and perturbed evaluation \(one run per variant, each in a freshly provisioned environment\)\.

Algorithm 3Agent Robustness Experiment1:Set of Instance IDs

ℐ\\mathcal\{I\}, Number of samples

NN, Configuration

CC, Agent

𝒜\\mathcal\{A\}, Evaluator

ℰ\\mathcal\{E\}, Transformations

𝒯\\mathcal\{T\}, Sampler hyperparameters

Nt,Nk,ϕ,NfN\_\{t\},N\_\{k\},\\phi,N\_\{f\}
2:Baseline Reports

ℛ0\\mathcal\{R\}\_\{0\}, Perturbed Reports

ℛp\\mathcal\{R\}\_\{p\}
3:procedureRunExperiment\(

ℐ\\mathcal\{I\},

NN,

CC,

𝒜\\mathcal\{A\},

ℰ\\mathcal\{E\},

𝒯\\mathcal\{T\},

NtN\_\{t\},

NkN\_\{k\},

ϕ\\phi,

NfN\_\{f\}\)

4:

ℛ0←∅\\mathcal\{R\}\_\{0\}\\leftarrow\\emptyset,

ℛp←∅\\mathcal\{R\}\_\{p\}\\leftarrow\\emptyset⊳\\trianglerightInitialize report collections

5:foreachinstance

i∈ℐi\\in\\mathcal\{I\}do

6:⊳\\trianglerightPhase 1: Seed Retrieval

7:

\(Cs​e​e​d,ℱgold,d​e​s​c\)←ExtractInstance​\(i\)\(C\_\{seed\},\\mathcal\{F\}\_\{\\text\{gold\}\},desc\)\\leftarrow\\textsc\{ExtractInstance\}\(i\)
8:⊳\\trianglerightPhase 2: Baseline Evaluation

9:for

k←1k\\leftarrow 1to

NNdo

10:

Ee​v​a​l←ProvisionEnvironment​\(i\)E\_\{eval\}\\leftarrow\\textsc\{ProvisionEnvironment\}\(i\)⊳\\trianglerightunperturbed seed, no injection

11:

ℛ←RunAndEvaluate​\(𝒜,ℰ,Ee​v​a​l,i,C\)\\mathcal\{R\}\\leftarrow\\textsc\{RunAndEvaluate\}\(\\mathcal\{A\},\\mathcal\{E\},E\_\{eval\},i,C\)
12:

ℛ0←ℛ0∪ℛ\\mathcal\{R\}\_\{0\}\\leftarrow\\mathcal\{R\}\_\{0\}\\cup\\mathcal\{R\}
13:endfor

14:⊳\\trianglerightPhase 3: Variant Generation \(Algorithm[1](https://arxiv.org/html/2608.18389#alg1)\)

15:

𝒮←\\mathcal\{S\}\\leftarrowGenerateVariants\(

Cs​e​e​dC\_\{seed\},

NN,

𝒯\\mathcal\{T\},

ℱgold\\mathcal\{F\}\_\{\\text\{gold\}\},

d​e​s​cdesc,

NtN\_\{t\},

NkN\_\{k\},

ϕ\\phi,

NfN\_\{f\}\)

16:⊳\\trianglerightPhase 4: Perturbed Evaluation

17:foreachsample

s∈𝒮s\\in\\mathcal\{S\}do

18:

Ee​v​a​l←ProvisionEnvironment​\(i\)E\_\{eval\}\\leftarrow\\textsc\{ProvisionEnvironment\}\(i\)
19:

InjectCode​\(Ee​v​a​l,s\)\\textsc\{InjectCode\}\(E\_\{eval\},s\)
20:

ℛ←RunAndEvaluate​\(𝒜,ℰ,Ee​v​a​l,i,C\)\\mathcal\{R\}\\leftarrow\\textsc\{RunAndEvaluate\}\(\\mathcal\{A\},\\mathcal\{E\},E\_\{eval\},i,C\)
21:

ℛp←ℛp∪ℛ\\mathcal\{R\}\_\{p\}\\leftarrow\\mathcal\{R\}\_\{p\}\\cup\\mathcal\{R\}
22:endfor

23:endfor

24:return

\(ℛ0,ℛp\)\(\\mathcal\{R\}\_\{0\},\\mathcal\{R\}\_\{p\}\)
25:endprocedure

26:

27:procedureRunAndEvaluate\(

𝒜,ℰ,E,i,C\\mathcal\{A\},\\mathcal\{E\},E,i,C\)

28:

P​a​t​c​h←ExecuteAgent​\(𝒜,E,i,C\)Patch\\leftarrow\\textsc\{ExecuteAgent\}\(\\mathcal\{A\},E,i,C\)
29:

R​e​p​o​r​t←EvaluatePatch​\(ℰ,i,P​a​t​c​h\)Report\\leftarrow\\textsc\{EvaluatePatch\}\(\\mathcal\{E\},i,Patch\)
30:return

R​e​p​o​r​tReport
31:endprocedure

### Computational Infrastructure

All experiments were run on one lab server\. It has two AMD EPYC 9554 64\-core processors \(128 cores, 256 threads\), 768 GB RAM, 6 Nvidia RTX PRO 6000 Blackwell GPUs \(96 GB VRAM per GPU\), and a 7 TB NVMe drive\. The operating system is Ubuntu 24\.04\.4 LTS\. Each agent run executes in its own Docker container \(Docker version 29\.5\.2\)\.

We serve Qwen 3\.6\-27B on this machine with vLLM \(v0\.20\.1\) using 2 GPUs\. We access Claude Opus 4\.5\( AWS bedrock ID: anthropic\.claude\-opus\-4\-5\-20251101\-v1:0\), Kimi K2\.5 \(AWS bedrock ID: moonshotai\.kimi\-k2\.5\) and Minimax M2\.5 \(AWS bedrock ID: minimax\.minimax\-m2\.5\) using Amazon Bedrock API\. All 4 models used temperature 1\.0\. Our code is Python 3\.10, using LibCST v1\.8\.5 for SPTs, Astroid v3\.3\.0 for type inference, and NLTK v3\.6\.0 for WordNet lookups\. The scaffolds are mini\-SWE agent v1\.17\.5 and OpenCode v1\.18\.4\.

##### Cost accounting of Qwen 3\.6\.

Reported costs are token costs\. For the three models accessed through Bedrock we use the provider’s published per\-token prices\. Qwen 3\.6\-27B is served locally and to keep costs comparable across models we price its input and output tokens at the OpenRouter rates for the same model\. The same price schedule is applied to the unperturbed and perturbed runs of an instance, soδcost\\delta\_\{\\text\{cost\}\}is an internally consistent within\-instance comparison\.

### Randomness and Reproducibility

We do not fix a random seed\. Our pipeline is random in two places\. The sampler \(Algorithm[1](https://arxiv.org/html/2608.18389#alg1)\) draws the transformation subset, the file\-inclusion probabilitypfilep\_\{\\text\{file\}\}, the sites within each file, and the target keywords\. And all four models are sampled at temperature 1\.0, so a run is stochastic even on a fixed repository\. A seed fixes which 20 variants we draw, not what the agent does on them: rerunning the agent on the same variant can give a different outcome\. Reproducing our results therefore means drawing 20 fresh variants per instance at the same hyperparameters and recovering our reported effects\. The bootstrap of the statistical protocol is the exception: it is seeded\.

### Code and Instance Lists

Our code and data are publicly available athttps://github\.com/CSU\-TrustLab/jagged\-frontier\. The repository contains the full implementation of the experiment and analysis\. This includes the SPT implementations, the variant sampler of Algorithm[1](https://arxiv.org/html/2608.18389#alg1), the evaluation harness of Algorithm[3](https://arxiv.org/html/2608.18389#alg3), the analysis scripts that produce the reported statistics and figures, and an interactive dashboard for exploring the results\. The prompts used are included there as well, as is the list of the 28 SWE\-bench Verified and 26 SWE\-bench Pro instance IDs used in the study\.

### Agent Scaffold Comparison

This table[2](https://arxiv.org/html/2608.18389#S3.T2)compares between mini\-SWE agent and OpenCode scaffold\.

Table 2:Comparison of mini\-SWE and OpenCode Agent Scaffolds
### Selection of hyperparameter values

We did not conduct a search over the sampler’s hyperparameter space\. A single configuration at a single hyperparameter setting costs54×40=2,16054\\times 40=2\{,\}160agent runs\. So even a coarse grid over\(Nt,Nk,ϕ,Nf\)\(N\_\{t\},N\_\{k\},\\phi,N\_\{f\}\)would multiply the cost of the study by an order of magnitude\. We instead fixed values for each parameter\.Nt=3N\_\{t\}=3of the 14 transformations makes a variant a composition of several SPTs\.ϕ=0\.7\\phi=0\.7leaves a minority of candidate sites in a file untouched\. So a variant contains both transformed and untransformed instances of the same pattern\.Nk=5N\_\{k\}=5andNf=10N\_\{f\}=10are the only values set from observation\. Without them, agents frequently detected the perturbation and reverted the repository outright \(e\.g\. viagit reset\), which destroys the measurement rather than making it harder\.

## DAdditional Experiment Results

![Refer to caption](https://arxiv.org/html/2608.18389v1/charts/merged_model_ranking.png)Figure 3:Mean resolve rate \(%\) on unperturbed and perturbed runs for each model under each scaffold, on SWE\-bench Verified \(top\) and SWE\-bench Pro \(bottom\)\.##### Resolve rates fall, but capability rankings survive\.

Figure[3](https://arxiv.org/html/2608.18389#S4.F3)shows the mean resolve rate on unperturbed and perturbed runs for each model under each scaffold, separately on SWE\-bench Verified and SWE\-bench Pro\. In 13 of the 16 scaffold\-model\-benchmark combinations the agent resolves fewer issues after perturbation\. The three remaining combinations move slightly in the other direction, each by less than one point, within run\-to\-run noise\. Perturbation, however, barely disturbs the models’*capability*ordering\. In three of the four scaffold\-benchmark panels, the ranking of models by resolve rate is identical before and after perturbation\. In the fourth \(OpenCode on SWE\-Pro\), the only change is a swap between MiniMax and Qwen, which are separated by just 0\.2 points at baseline\. This shows that a leaderboard\-style comparison would not register the impact of SPTs\. The ranking that perturbation does scramble is the ranking by*robustness*, as we show next\.

### Effort Overhead over All Runs

Figure[4](https://arxiv.org/html/2608.18389#S4.F4)repeats the RQ2 effort analysis of the main text over*all*runs, rather than only the runs the agent resolved\. The trends match the resolved\-only figure\. On SWE\-bench Verified cost rises in all eight configurations, from 4\.5% \(mini\-SWE agent with Opus\) to 25\.4% \(OpenCode with MiniMax\), with the interval excluding zero in seven of the eight\. On SWE\-bench Pro cost rises in seven of the eight, by up to 14\.7% \(OpenCode with Kimi\), with six intervals excluding zero\. The one exception, mini\-SWE agent with Opus, is flat at−0\.1%\-0\.1\\%\. Step counts move less than cost in both benchmarks, and Opus under mini\-SWE agent on Verified again takes 19\.7% fewer steps while still costing more\. Because the two views agree, the overhead cannot be attributed to long failing runs, and we report the resolved\-only view in the main text as the more conservative one\.

![Refer to caption](https://arxiv.org/html/2608.18389v1/charts/cost_step_summary_all_swe_bench.png)

![Refer to caption](https://arxiv.org/html/2608.18389v1/charts/cost_step_summary_all_swe_pro.png)

Figure 4:Mean relative change in agent steps \(δ¯step\\bar\{\\delta\}\_\{\\text\{step\}\}\) and cost \(δ¯cost\\bar\{\\delta\}\_\{\\text\{cost\}\}\) after perturbation over all runs, for each scaffold–model configuration on the 28 SWE\-bench Verified instances \(left\) and the 26 SWE\-bench Pro instances \(right\)\. Positive values mean perturbed runs take more steps, or cost more, than unperturbed runs on the same instance\. Error bars are fixed\-population 95% bootstrap confidence intervals\.
### Per\-Instance Degradation Plots

Figures[5](https://arxiv.org/html/2608.18389#S4.F5)and[6](https://arxiv.org/html/2608.18389#S4.F6)give the per\-instance degradationΔ⁡\(i\)\\Delta\(i\)with its Newcombe 95% interval for every instance\-configuration cell\. This section reports the underlying numbers for the concentration claim of the RQ1 results\.

With 20 runs per condition, a Newcombe interval is wide by construction: even an instance resolved in every run of both conditions carries a±16\\pm 16\-point interval\. The interval excludes zero in only 7 of the 224 cells on SWE\-bench Verified and 7 of the 208 on SWE\-bench Pro\. Most intervals are therefore too wide to establish degradation for a specific instance, although their mass is shifted toward positive values\. Point estimates are positive in 65 cells against 35 negative on SWE\-bench Verified, and 79 against 44 on SWE\-bench Pro\.

The individually significant cells concentrate in the tail\. On SWE\-bench Verified,scikit\-learn\_\_scikit\-learn\-14983loses 40 points under OpenCode with Qwen,matplotlib\_\_matplotlib\-20859loses 35 under mini\-SWE agent with Qwen, andsympy\_\_sympy\-12489loses 30 under mini\-SWE agent with MiniMax\. On SWE\-bench Pro,element\-web\_dae13loses 50 points,openlibrary\_f3b2640, andopenlibrary\_a7b7d35, all under mini\-SWE agent with Opus\. These three Pro cells alone account for roughly two thirds of that configuration’s 6\.7\-point mean degradation\. Across configurations,qutebrowser\_ef5baandansible\_9142bdegrade in seven of eight, andsympy\_\_sympy\-11618in five while never improving\.

Single cells also swing the other way:mwaskom\_\_seaborn\-3069improves by 55 points under OpenCode with MiniMax and by 35 under mini\-SWE agent with Qwen, both with intervals excluding zero\. With a singlemwaskom\_\_seaborninstance in the sample, such swings should not be read as a repository\-level effect\.

![Refer to caption](https://arxiv.org/html/2608.18389v1/charts/per_instance_degradation_forest_swebench.png)Figure 5:Per\-instance degradationΔ⁡\(i\)\\Delta\(i\)for the 28 SWE\-bench Verified instances under each configuration\. Error bars are Newcombe 95% confidence intervals for the difference between the unperturbed and perturbed resolve proportions \(n=20n\{=\}20runs per condition\)\.![Refer to caption](https://arxiv.org/html/2608.18389v1/charts/per_instance_degradation_forest_swepro.png)Figure 6:Per\-instance degradationΔ⁡\(i\)\\Delta\(i\)for the 26 SWE\-bench Pro instances under each configuration\. Error bars are Newcombe 95% confidence intervals for the difference between the unperturbed and perturbed resolve proportions \(n=20n\{=\}20runs per condition\)\.

## EUnbiasedness of the Degradation Estimator

We estimate perturbed performance by running the agent*once*on each ofN=20N=20sampled variants\. Because a single run of a stochastic agent is a noisy measurement of that variant’s difficulty, it is reasonable to ask whether one run per variant is enough\. An alternative would be to draw fewer variants and run each several times, averaging away the agent’s randomness\. This appendix shows that no such averaging is needed\. Per\-instance degradation is unbiased for*any*number of runs per variant including one, and under a fixed run budget one run per variant is in fact the variance\-minimizing allocation\.

### Setup and notation

Consider, a task instanceiiand a configuration \(scaffold, model, benchmark\)\.

##### Variant sampling\.

A variantVVis drawn by the variant sampler from the population of semantics\-preserving variants of instanceii\. Each variant carries a*true resolve probability*

pV=Pr⁡\(agent resolves​V∣V\),p\_\{V\}\\;=\\;\\Pr\(\\text\{agent resolves \}V\\mid V\),\(1\)the probability that the agent resolves that specific variant, averaged over the agent’s own randomness\. BecauseVVis itself random,pVp\_\{V\}is a random variable\. It is the difficulty of a randomly chosen semantically equivalent rewrite of instanceii\.

##### Agent sampling\.

Given a fixed variantvv, a single agent run yields a resolve/no\-resolve indicator

Y∣\(V=v\)∼Bernoulli\(pv\),𝔼\[Y∣V=v\]=pv\.Y\\mid\(V=v\)\\;\\sim\\;\\mathrm\{Bernoulli\}\(p\_\{v\}\),\\qquad\\mathbb\{E\}\[Y\\mid V=v\]\\;=\\;p\_\{v\}\.\(2\)The agent’s stochasticity is conditionally independent of how the variant was chosen, given the variant\. This is enforced by the experimental protocol\. Every run executes in a freshly provisioned, isolated environment, so no state carries between runs\.

##### Estimands\.

The population quantity thatrp​\(i\)r\_\{p\}\(i\)targets is the mean per\-variant resolve probability,

μ=𝔼⁡\[pV\],\\mu\\;=\\;\\mathbb\{E\}\[p\_\{V\}\],\(3\)and its companion, the spread of difficulty across variants, is

σ2=Var⁡\(pV\)\.\\sigma^\{2\}\\;=\\;\\mathrm\{Var\}\(p\_\{V\}\)\.\(4\)Hereμ\\musummarizes*average*performance on variants, whileσ2\\sigma^\{2\}measures*consistency*— how much the agent’s success probability swings across semantically identical inputs\. Degradation isΔ⁡\(i\)=p0−μ\\Delta\(i\)=p\_\{0\}\-\\mu, wherep0p\_\{0\}is the resolve probability on the unperturbed seed\.Δ⁡\(i\)=p0−μ\\Delta\(i\)=p\_\{0\}\-\\muis estimated byΔ^​\(i\)=r0​\(i\)−rp​\(i\)\\hat\{\\Delta\}\(i\)=r\_\{0\}\(i\)\-r\_\{p\}\(i\), withrp​\(i\)=μ^r\_\{p\}\(i\)=\\hat\{\\mu\}\.

##### Data\.

We drawNNvariantsV1,…,VNV\_\{1\},\\dots,V\_\{N\}independently and run the agentKKtimes on each, recording the pass count and empirical pass rate

Xj∣\(Vj=vj\)∼Binomial\(K,pvj\),p^j=Xj/K\.X\_\{j\}\\mid\(V\_\{j\}=v\_\{j\}\)\\;\\sim\\;\\mathrm\{Binomial\}\(K,p\_\{v\_\{j\}\}\),\\qquad\\hat\{p\}\_\{j\}\\;=\\;X\_\{j\}/K\.\(5\)Our protocol isN=20N=20,K=1K=1\.

##### A useful identity\.

One identity relates the within\-variant noise to the two estimands and is used repeatedly below\. Write

ν=𝔼⁡\[pV​\(1−pV\)\]\\nu\\;=\\;\\mathbb\{E\}\\big\[p\_\{V\}\(1\-p\_\{V\}\)\\big\]\(6\)for the average within\-variant Bernoulli variance\. Expanding and applying𝔼⁡\[pV2\]=Var⁡\(pV\)\+\(𝔼⁡\[pV\]\)2=σ2\+μ2\\mathbb\{E\}\[p\_\{V\}^\{2\}\]=\\mathrm\{Var\}\(p\_\{V\}\)\+\(\\mathbb\{E\}\[p\_\{V\}\]\)^\{2\}=\\sigma^\{2\}\+\\mu^\{2\},

ν=𝔼⁡\[pV\]−𝔼⁡\[pV2\]=μ−\(σ2\+μ2\)=μ⁡\(1−μ\)−σ2,\\nu\\;=\\;\\mathbb\{E\}\[p\_\{V\}\]\-\\mathbb\{E\}\[p\_\{V\}^\{2\}\]\\;=\\;\\mu\-\(\\sigma^\{2\}\+\\mu^\{2\}\)\\;=\\;\\mu\(1\-\\mu\)\-\\sigma^\{2\},\(7\)so that

σ2\+ν=μ⁡\(1−μ\)\.\\sigma^\{2\}\+\\nu\\;=\\;\\mu\(1\-\\mu\)\.\(8\)The total variability in a single perturbed run therefore splits into a between\-variant partσ2\\sigma^\{2\}and a within\-variant partν\\nu\.

### The estimator

The estimator ofμ\\muis the average of the per\-variant rates,

μ^=1N​∑j=1Np^j\.\\hat\{\\mu\}\\;=\\;\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}\\hat\{p\}\_\{j\}\.\(9\)AtK=1K=1we havep^j=Yj∈\{0,1\}\\hat\{p\}\_\{j\}=Y\_\{j\}\\in\\\{0,1\\\}and this collapses to

μ^=\#​\{variants resolved\}N,\\hat\{\\mu\}\\;=\\;\\frac\{\\\#\\\{\\text\{variants resolved\}\\\}\}\{N\},\(10\)an ordinary binomial proportion\. We use this formula to calculate resolve rate for perturbed runs\.

### Why it is unbiased for anyKK, includingK=1K=1

The argument conditions on the variant first and averages over variants second, using the tower property𝔼\[⋅\]=𝔼\[𝔼\[⋅∣V\]\]\\mathbb\{E\}\[\\,\\cdot\\,\]=\\mathbb\{E\}\\big\[\\mathbb\{E\}\[\\,\\cdot\\mid V\]\\big\]\.

##### Step 1: collapse the agent randomness\.

Condition on thejj\-th variantVjV\_\{j\}and writeYj​1,…,Yj​KY\_\{j1\},\\dots,Y\_\{jK\}for theKKruns on it\. By Eq\. \([2](https://arxiv.org/html/2608.18389#S5.E2)\) each has conditional meanpVjp\_\{V\_\{j\}\}, so

𝔼⁡\[p^j∣Vj\]=𝔼⁡\[1K​∑k=1KYj​k\|Vj\]=1K​∑k=1K𝔼⁡\[Yj​k∣Vj\]=1K⋅K​pVj=pVj\.\\begin\{split\}\\mathbb\{E\}\[\\hat\{p\}\_\{j\}\\mid V\_\{j\}\]&\\;=\\;\\mathbb\{E\}\\Big\[\\tfrac\{1\}\{K\}\\textstyle\\sum\_\{k=1\}^\{K\}Y\_\{jk\}\\,\\Big\|\\,V\_\{j\}\\Big\]\\;=\\;\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\[Y\_\{jk\}\\mid V\_\{j\}\]\\\\ &\\;=\\;\\frac\{1\}\{K\}\\cdot K\\,p\_\{V\_\{j\}\}\\;=\\;p\_\{V\_\{j\}\}\.\\end\{split\}\(11\)The factors ofKKand1/K1/Kcancel, so this holds*regardless ofKK*\. Even a single draw,K=1K=1, is already unbiased for its ownpVjp\_\{V\_\{j\}\}\.

##### Step 2: collapse the variant randomness\.

BecauseVjV\_\{j\}is itself random, the conditional mean in Eq\. \([11](https://arxiv.org/html/2608.18389#S5.E11)\) is a random variable, and averaging it over the variant population gives, by the tower property and Eq\. \([3](https://arxiv.org/html/2608.18389#S5.E3)\),

𝔼⁡\[p^j\]=𝔼⁡\[𝔼⁡\[p^j∣Vj\]\]=𝔼⁡\[pVj\]=μ\.\\mathbb\{E\}\[\\hat\{p\}\_\{j\}\]\\;=\\;\\mathbb\{E\}\\big\[\\,\\mathbb\{E\}\[\\hat\{p\}\_\{j\}\\mid V\_\{j\}\]\\,\\big\]\\;=\\;\\mathbb\{E\}\\big\[\\,p\_\{V\_\{j\}\}\\,\\big\]\\;=\\;\\mu\.\(12\)

##### Step 3: average over variants\.

By linearity of expectation, applied to Eq\. \([9](https://arxiv.org/html/2608.18389#S5.E9)\),

𝔼⁡\[μ^\]=1N​∑j=1N𝔼⁡\[p^j\]=1N⋅N​μ=μ\.\\mathbb\{E\}\[\\hat\{\\mu\}\]\\;=\\;\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}\\mathbb\{E\}\[\\hat\{p\}\_\{j\}\]\\;=\\;\\frac\{1\}\{N\}\\cdot N\\mu\\;=\\;\\mu\.\(13\)

##### Consequence\.

The parameterKKnever appears in Eqs\. \([11](https://arxiv.org/html/2608.18389#S5.E11)\)–\([13](https://arxiv.org/html/2608.18389#S5.E13)\)\. Repetition is a variance\-reduction operation\. It tightens eachp^j\\hat\{p\}\_\{j\}around the centerpVjp\_\{V\_\{j\}\}, but that center was already correct, so tightening cannot remove a bias that was never there\. One run per variant is exactly as unbiased as a thousand\.

### Variance, and whyK=1K=1is the efficient allocation

Unbiasedness alone does not justifyK=1K=1\. An unbiased estimator can still be noisy\. We now show thatK=1K=1is not only admissible but optimal\.

##### Step 1: variance of one per\-variant rate\.

DecomposeVar⁡\(p^j\)\\mathrm\{Var\}\(\\hat\{p\}\_\{j\}\)with the law of total variance,

Var⁡\(p^j\)=Var⁡\(𝔼⁡\[p^j∣Vj\]\)\+𝔼⁡\[Var⁡\(p^j∣Vj\)\]\.\\mathrm\{Var\}\(\\hat\{p\}\_\{j\}\)\\;=\\;\\mathrm\{Var\}\\big\(\\mathbb\{E\}\[\\hat\{p\}\_\{j\}\\mid V\_\{j\}\]\\big\)\\;\+\\;\\mathbb\{E\}\\big\[\\mathrm\{Var\}\(\\hat\{p\}\_\{j\}\\mid V\_\{j\}\)\\big\]\.\(14\)The first term isVar⁡\(pVj\)=σ2\\mathrm\{Var\}\(p\_\{V\_\{j\}\}\)=\\sigma^\{2\}by Eqs\. \([11](https://arxiv.org/html/2608.18389#S5.E11)\) and \([4](https://arxiv.org/html/2608.18389#S5.E4)\)\. For the second,p^j=Xj/K\\hat\{p\}\_\{j\}=X\_\{j\}/KwithXj\|Vj∼Binomial⁡\(K,pVj\)X\_\{j\}\\mid V\_\{j\}\\sim\\mathrm\{Binomial\}\(K,p\_\{V\_\{j\}\}\), soVar⁡\(p^j∣Vj\)=pVj​\(1−pVj\)/K\\mathrm\{Var\}\(\\hat\{p\}\_\{j\}\\mid V\_\{j\}\)=p\_\{V\_\{j\}\}\(1\-p\_\{V\_\{j\}\}\)/K, whose expectation over variants isν/K\\nu/Kby Eq\. \([6](https://arxiv.org/html/2608.18389#S5.E6)\)\. Hence

Var⁡\(p^j\)=σ2\+νK\.\\mathrm\{Var\}\(\\hat\{p\}\_\{j\}\)\\;=\\;\\sigma^\{2\}\+\\frac\{\\nu\}\{K\}\.\(15\)

##### Step 2: variance of the estimator\.

The variants are drawn independently and each is run in an isolated environment, so thep^j\\hat\{p\}\_\{j\}are i\.i\.d\. and

Var⁡\(μ^\)=1N2​∑j=1NVar⁡\(p^j\)=σ2N\+νN​K=σ2N\+νR,\\mathrm\{Var\}\(\\hat\{\\mu\}\)\\;=\\;\\frac\{1\}\{N^\{2\}\}\\sum\_\{j=1\}^\{N\}\\mathrm\{Var\}\(\\hat\{p\}\_\{j\}\)\\;=\\;\\frac\{\\sigma^\{2\}\}\{N\}\+\\frac\{\\nu\}\{NK\}\\;=\\;\\frac\{\\sigma^\{2\}\}\{N\}\+\\frac\{\\nu\}\{R\},\(16\)writingR=N​KR=NKfor the total number of agent runs\. The within\-variant noise termν/R\\nu/Rdepends only on the*total*budgetRR; how those runs are distributed across variants is irrelevant to it\. The between\-variant termσ2/N\\sigma^\{2\}/Ndepends only on the*number of distinct variants*NN, andKKdoes not appear in it at all\.

##### Step 3: optimize under a fixed budget\.

Assume a fixed run budgetRRand substituteN=R/KN=R/Kinto Eq\. \([16](https://arxiv.org/html/2608.18389#S5.E16)\):

Var⁡\(μ^\)=K​σ2R\+νR=K​σ2\+νR\.\\mathrm\{Var\}\(\\hat\{\\mu\}\)\\;=\\;\\frac\{K\\sigma^\{2\}\}\{R\}\+\\frac\{\\nu\}\{R\}\\;=\\;\\frac\{K\\sigma^\{2\}\+\\nu\}\{R\}\.\(17\)This is strictly increasing inKKwheneverσ2\>0\\sigma^\{2\}\>0, and is therefore minimized atK=1K=1\. Under a fixed number of agent runs, the variance\-optimal design for estimating the mean is to spend every run on a*new*variant and run the agent once\.

##### Step 4: theK=1K=1specialization\.

AtK=1K=1the estimator’s variance simplifies\. Substituting the identity \([8](https://arxiv.org/html/2608.18389#S5.E8)\) into Eq\. \([17](https://arxiv.org/html/2608.18389#S5.E17)\) withK=1K=1andR=NR=N,

Var⁡\(μ^\)=σ2\+νN=μ⁡\(1−μ\)N\.\\mathrm\{Var\}\(\\hat\{\\mu\}\)\\;=\\;\\frac\{\\sigma^\{2\}\+\\nu\}\{N\}\\;=\\;\\frac\{\\mu\(1\-\\mu\)\}\{N\}\.\(18\)This has a direct interpretation\. EachYjY\_\{j\}is*marginally*Bernoulli⁡\(μ\)\\mathrm\{Bernoulli\}\(\\mu\)and theNNtrials are independent, since each uses a distinct variant and no two share a draw\. Soμ^\\hat\{\\mu\}is a textbook binomial proportion with standard errorμ^​\(1−μ^\)/N\\sqrt\{\\hat\{\\mu\}\(1\-\\hat\{\\mu\}\)/N\}, andrp​\(i\)r\_\{p\}\(i\)may be compared tor0​\(i\)r\_\{0\}\(i\)with a standard two\-proportion procedure\. No clustering correction is needed at the variant level precisely because every trial is a different variant\.

## FQualitative Analysis Details

### Instance Selection Procedure

12/20 of these instances are manually picked by evaluating characteristics like high variance \(3\), high degradation \(3\), large increase in step count \(1\) or file coverage \(2\) or cost \(1\), or cases with 1/20 mutant failures \(2\)\. The remaining 8/20 are picked by passing mutants that the agent had failed to resolve through an LLM prompt designed to take the SPT log as well as a condensed version of the agent trajectory as input, and predict possible SPT impact based on the SPTs that were applied, intersection of files touched by both the agent and the SPTs, and specific snippets from agent thoughts or tool outputs that have SPT markers\. This is done for 115 failing trajectories across all 16 experiments\. The resulting output is then manually evaluated to shortlist 8 trajectories for final human review\.

### AI\-Assisted Trajectory Analysis

The same prompt also extracts a sequence of events from the trajectory, citing snippets from the raw trajectory for each, while also classifying each event as one of the following: Localization \(exploring the repo to find relevant code\), Debugging \(reasoning, specifically based on observed outputs\), Planning \(forming a plan for patching\), Patching \(Implementation of changes\), Validation \(Testing of changes\), Recovery \(Backtracking on implemented changes or proposed plan\) or General \(additional events that don’t come under any of the 6 aforementioned types, for example, patch submission\)\. A final HTML is generated, containing the raw trajectory, a graph with the extracted events, including snippets from the actual trajectory, a record of files touched by both the agent and the SPTs, and script based results like git statements, Human evaluators then use this html to get a comprehensive view of the trajectory while analyzing\. This is strictly an assistive method, and all qualitative analysis examples mentioned in the paper have involved actual human review of all relevant portions of the raw trajectories\.

### Additional Qualitative Insights

##### Variability in SPT detection\.

There is high variance in whether SPTs are detected at all even in cases where the agent repeatedly accesses mutated code\. While some SPTs like double negation are easier to miss, SPTs which have been detected in other cases, like if true wrapper, are ignored in cases like Opus on Psf\-Requests with mini\-SWE agent\. In this particular case, even though the agent accesses mutated code multiple times, it never acknowledges any SPT presence\. Moreover, consistent with the qualitative results, SPT impact appears to be variable across instances, with more impact seen in repos like qutebrowser, openlibrary, ansible, pytest and sympy as compared to repos like nodeBB or astropy, based on qualitative reviews\.

##### Additional examples of agent response to SPTs\.

There are multiple other examples of how agents choose to respond to SPTs\. In Qwen on Matplotlib with OpenCode, after detecting SPTs, the agent decides to reset the repository\. However, it claims to have reset it multiple times without actually doing so, and only later ends up proceeding with it\. In another case, Opus on Qutebrowser with mini\-SWE agent, the agent notices complicated booleans introduced by the if true wrapper in the relevant code, and spends additional time in resolving the booleans into final true/false values\. It later also "fixes" them in the final patch, thereby investing significant effort in interacting with the SPTs\. In contrast, MiniMax with mini\-SWE agent on Qutebrowser chooses to avoid core files that have been transformed by SPTs, and instead tries to get information from other sources in the repository, subsequently increasing the time taken to localize the issue as well as file coverage\.

##### Input prompt impacts agent jaggedness\.

We observe that issue description quality and nature of issue may impact agent jaggedness\. We analyse Qwen, Kimi and Opus on the same NodeBB issue which exhibits high variance across various model and scaffold pairs\. The issue description contains two conflicting statements — editing the code to satisfy issue requirements causes one of the unit tests to break, and the description explicitly states not to alter any unit tests\. This leads to all three agents falling into a loop of patching, validating, and debugging, while going back to the issue description multiple times, which substantially adds to both the step count and cost\. In a passing trajectory for the same issue, the agent comes across the same conflict and fails the same test, but doesn’t backtrack on the code and therefore ends up passing\. So the conflicting nature of the issue instructions greatly contributes to jaggedness in agent performance, as it is left to its own judgement to resolve the conflict\.

##### Robustness of test suite formation varies\.

For Astropy, Kimi uses a three\-level test suite, complete with custom tests for basic functionality, project test suite, and a dedicated edge cases script\. However, in Pytest and Qutebrowser, the validation framework is brittle, with insufficient debugging and incorrect failure attribution to SPTs in the former and shifting test cases during debugging in the latter\. Similarly, for MiniMax, validation setup is comprehensive in Sympy, but has incorrect assertions in custom test scripts in Qutebrowser\.