InfraBench: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Risk

# InfraBench: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Risk
Source: [https://arxiv.org/html/2608.11234](https://arxiv.org/html/2608.11234)
Yuan Gao1Zeren Yang1Junnan Li1Shawn \(Wanxiang\) Zhong1 Ahmed Dajani2Mai Zheng2Andrea Arpaci\-Dusseau1Remzi Arpaci\-Dusseau1 1University of Wisconsin–Madison2Iowa State University ygao355@wisc\.edu, zyang667@wisc\.edu, jli2786@wisc\.edu, shawn\.zhong@wisc\.edu, dusseau@cs\.wisc\.edu, remzi@cs\.wisc\.edu, adajani@iastate\.edu, mai@iastate\.edu

###### Abstract

Managing modern computing infrastructure has become a steadily harder problem due to the ever\-increasing complexity\. Recent advances in AI agents create a timely opportunity to automate infrastructure management tasks, but it remains unclear how well such agents can handle real\-world infrastructure complexity\. We present InfraBench, a benchmark suite for evaluating AI agents on realistic infrastructure tasks across the full system stack and full operational lifecycle with fine\-grained risk assessment\. Experiments with 15 agent–model configurations show that even the strongest agent cannot secure a full score across all tasks\. Mean effective scores range from roughly 40% to 88% \(with per\-configuration standard errors of 6–12 points\), repeating every task three times reveals that top configurations still pass only a fraction of their attempts, and per\-check scoring exposes a general failure pattern: agents may routinely satisfy short\-term objectives while leaving non\-durable changes, broken distributed invariants, unsafe side effects, and uncleaned state behind\.InfraBench, including its live leaderboard, tasks, and evaluation harness, is publicly available at[infraben\.ch](https://infraben.ch/)\.

## 1Introduction

As computing infrastructures continue to grow in scale and complexity, managing them has become a steadily harder problem\. Even initial deployment now requires navigating diverse configurations across heterogeneous environments, from on\-prem clusters to cloud interactions\. Beyond deployment, continuous maintenance introduces additional burdens \(e\.g\., upgrades and patching\[[21](https://arxiv.org/html/2608.11234#bib.bib97),[28](https://arxiv.org/html/2608.11234#bib.bib98)\], failure handling\[[12](https://arxiv.org/html/2608.11234#bib.bib39),[18](https://arxiv.org/html/2608.11234#bib.bib48),[39](https://arxiv.org/html/2608.11234#bib.bib11),[15](https://arxiv.org/html/2608.11234#bib.bib50),[43](https://arxiv.org/html/2608.11234#bib.bib49)\], migration and backup\[[9](https://arxiv.org/html/2608.11234#bib.bib99),[20](https://arxiv.org/html/2608.11234#bib.bib100),[27](https://arxiv.org/html/2608.11234#bib.bib101),[19](https://arxiv.org/html/2608.11234#bib.bib103)\]\), all of which must be handled without disrupting service\. This rising complexity turns infrastructure management into a persistent, long\-standing challenge\[[26](https://arxiv.org/html/2608.11234#bib.bib102),[17](https://arxiv.org/html/2608.11234#bib.bib105),[16](https://arxiv.org/html/2608.11234#bib.bib10),[39](https://arxiv.org/html/2608.11234#bib.bib11)\]\.

Recent advances in artificial intelligence \(AI\) agents\[[41](https://arxiv.org/html/2608.11234#bib.bib91),[40](https://arxiv.org/html/2608.11234#bib.bib92),[33](https://arxiv.org/html/2608.11234#bib.bib93),[8](https://arxiv.org/html/2608.11234#bib.bib94),[1](https://arxiv.org/html/2608.11234#bib.bib95),[25](https://arxiv.org/html/2608.11234#bib.bib96)\]create a timely opportunity to revisit the challenge\. A key question is whether AI agents can meaningfully automate these infrastructure\-level tasks, and if so, to what extent they can handle the complexity and variability seen in the real world\. Answering this requires rigorous system setups and measurements, yet existing benchmarks for AI agents focus on relatively simple scenarios which cannot capture the full spectrum of infrastructure management\[[24](https://arxiv.org/html/2608.11234#bib.bib85),[40](https://arxiv.org/html/2608.11234#bib.bib92),[29](https://arxiv.org/html/2608.11234#bib.bib89),[36](https://arxiv.org/html/2608.11234#bib.bib88),[23](https://arxiv.org/html/2608.11234#bib.bib86),[10](https://arxiv.org/html/2608.11234#bib.bib87)\]\. As summarized in Table[1](https://arxiv.org/html/2608.11234#S1.T1), they are largely limited in system environments \(e\.g\., container only\[[36](https://arxiv.org/html/2608.11234#bib.bib88)\]\), infrastructure lifecycle \(e\.g\., no deployment or decommissioning phases\[[24](https://arxiv.org/html/2608.11234#bib.bib85),[36](https://arxiv.org/html/2608.11234#bib.bib88),[10](https://arxiv.org/html/2608.11234#bib.bib87)\]\), scale \(e\.g\., single\-node only\[[24](https://arxiv.org/html/2608.11234#bib.bib85),[29](https://arxiv.org/html/2608.11234#bib.bib89)\]\), and often lack of risk assessments\.

Table 1:InfraBenchvs\. Others\.Columns: breadth of system environments; lifecycle phases evaluated \(deployment through decommissioning\); multi\-node/distributed scale; risk and side\-effect assessment\. ✓ first\-class;△\\trianglepartial; – limited or absent\.To bridge the gaps, we introduceInfraBench, a comprehensive benchmark suite for evaluating the capabilities of AI agents in infrastructure\-related tasks\. Different from existing efforts,InfraBenchis designed with four main goals:

- •Full\-Stack\. Practical infrastructures often involve many layers \(e\.g\., bare\-metal \(BM\) or virtual machines \(VM\), operating systems \(OS\), distributed storage and compute\[[38](https://arxiv.org/html/2608.11234#bib.bib76),[35](https://arxiv.org/html/2608.11234#bib.bib74),[6](https://arxiv.org/html/2608.11234#bib.bib1),[7](https://arxiv.org/html/2608.11234#bib.bib2),[11](https://arxiv.org/html/2608.11234#bib.bib70),[2](https://arxiv.org/html/2608.11234#bib.bib69),[3](https://arxiv.org/html/2608.11234#bib.bib3),[34](https://arxiv.org/html/2608.11234#bib.bib104)\]\) that cannot be ignored\.
- •Full\-Lifecycle\. Infrastructures live through multiple phases \(e\.g\., deployment, runtime usage, maintenance, decommissioning\), each with a set of unique operations and requirements\.
- •Risk\-Aware\.Infrastructure tasks are fundamental and one simple error may cause cascading problems \(i\.e\., blast radius issues\[[16](https://arxiv.org/html/2608.11234#bib.bib10),[18](https://arxiv.org/html/2608.11234#bib.bib48)\]\), so assessing potential risks and side effects is necessary\.
- •Realistic & Extensible\. Finally, we must reflect real\-world scenarios \(e\.g\., BM/VM clusters\) to ensure high fidelity and practicality, and enable easy extension for the broad community\.

To achieve the goals, we buildInfraBenchfrom four complementary sources: \(1\) semi\-structured interviews with infrastructure providers and practitioners, including three university centers\[[31](https://arxiv.org/html/2608.11234#bib.bib5),[30](https://arxiv.org/html/2608.11234#bib.bib6),[32](https://arxiv.org/html/2608.11234#bib.bib7)\]and one cross\-city testbed\[[22](https://arxiv.org/html/2608.11234#bib.bib4)\]at the time of writing, to elicit first\-hand experiences on systems and operational constraints that are seldom captured in the literature; \(2\) open\-source repositories and issue trackers of widely deployed infrastructure software \(e\.g\., Slurm\[[42](https://arxiv.org/html/2608.11234#bib.bib8)\], Pelican\[[37](https://arxiv.org/html/2608.11234#bib.bib106)\], Ceph\[[38](https://arxiv.org/html/2608.11234#bib.bib76)\]\); \(3\) documentations of commercial cloud platforms; \(4\) systems research prototypes that stress current designs\. Triangulating across these sources lets us model a wide\-spectrum of infrastructures and derive a general workflow to support systematic benchmarking across infrastructure layers and lifecycle with fine\-grained risk monitoring and assessments \(See §[2](https://arxiv.org/html/2608.11234#S2)\)\.

We have implemented a preliminary prototype ofInfraBenchwith twelve seed tasks, and evaluated 15 agent–model configurations across five coding\-agent CLIs at the time of writing\. The experimental results are promising:InfraBenchshows that state\-of\-the\-art \(SOTA\) agents often satisfy short\-term checks while leaving operational obligations unresolved, which may cause negative impacts on the underlying infrastructures in the long term, including incomplete deployment state, non\-durable changes, unsafe side effects, and missed cleanup requirements\. We releaseInfraBenchas an open\-source platform to facilitate infrastructure\-level benchmarking of AI agents in the broad community\.

## 2InfraBenchDesign & Implementation

![Refer to caption](https://arxiv.org/html/2608.11234v1/figure/design-overview-v2.png)Figure 1:InfraBenchOverview\.The workflow consists of four components: Task Specification, Executor, Evaluator, and Output\.Figure[1](https://arxiv.org/html/2608.11234#S2.F1)shows an overview ofInfraBench\. The general workflow consists of four components: \(1\)Task Specification, \(2\)Executor, \(3\)Evaluator, and \(4\)Output\. It supports each benchmark instance as a controlled infrastructure operation trial, which may involve a variety of operations \(e\.g\., configuration, recovery, migration, cleanup\) across four layers:

- •*L1 Hardware:*physical level operations \(e\.g\., BMC/IPMI control, power cycling, RAID configuration\);
- •*L2 Local Systems:*host\-level system software \(e\.g\., OS, compiler, container runtime\);
- •*L3 Distributed Systems:*networked systems operating across nodes \(e\.g\., Ceph\[[5](https://arxiv.org/html/2608.11234#bib.bib24)\], Slurm\[[42](https://arxiv.org/html/2608.11234#bib.bib8)\], Fabric\[[4](https://arxiv.org/html/2608.11234#bib.bib27)\]\);
- •*L4 User Applications:*user\-facing applications or services running on local \(L2\) or distributed systems \(L3\)\.

By mapping tasks to layers \(L1–L4\),InfraBenchprovides first\-class support in three dimensions: layer\-aware backend selection, full\-lifecycle evaluation, and operational risk monitoring\. We elaborate on the main components below\.

### 2\.1Task Specification

Each trial begins with a task package that defines two types of information: \(a\) agent\-visible instructions; \(b\) the hidden evaluation context, such as setup and bootstrap requirements of the target layer, workloads and fault conditions, oracles for verification, and lifecycle policies specifying which must be applied\. Additional constraints based on infrastructure specifics \(e\.g\., university center requirements\) can also be added to improve coverage\.

### 2\.2Executor

This component instantiates the task on a faithful backend and exposes an operational interface to the target agent\. There are two sub\-modules: \(1\)*Backend Selector*maps each task to an execution layer and provisions resources \(e\.g\., BM/VM nodes, Kubernetes clusters\) to support task execution\. \(2\)*Scenario Manager*configures the selected backend to an initial state based on policies \(e\.g\., CPU/RAM limits, fault models and triggering conditions\), and opens the agent operation window while keeping the evaluation context \(e\.g\., validators and oracle scripts\) transparent\.

### 2\.3Evaluator

This component evaluates the target agents in terms of both task completion and risks via two sub\-modules:

Full\-Lifecycle Checkerseparates short\-term success from long\-term operational correctness through four gates\.*Immediate Evaluation*is the short\-term gate: it compares the baseline and post\-operation state after the agent operation window and checks whether the immediate objective was satisfied\. The remaining gates provide long\-term validation\. E\.g\.,*Live Evaluation*continues under sustained workload or periodic probes to detect configuration drift, delayed degradation, or loss of availability;*Restart/Durability*restarts the relevant services or resources and checks post\-restart invariants and persistent configuration;*Decommission*verifies that the infrastructure can be restored to its initial state, and/or requested resources are torn down cleanly with no leakage\.

Risk Monitor\.InfraBenchtreats operational risk and side effects as first\-class evaluation signals alongside lifecycle correctness\. As part of the benchmark workflow, the Risk Monitor runs an LLM\-judge pass over each retained action trajectory: it reads the recorded commands in context, classifies them against a fixed danger taxonomy \(e\.g\., destructive filesystem operations, disabled safety checks, unnecessary privilege escalation, configuration drift, resource leaks, interference with unrelated services, and evaluator\-harness probing\), and emits structured review findings for the trial\. These findings flag cases where an agent reaches an immediate objective by relying on unsafe shortcuts or leaves collateral damage that pass/fail checks would miss\. The Monitor is complemented by the verifier’s preservation checks—invariants that penalize collateral damage directly \(baseline data still matching after a restart, an export still in place, a peer node still reachable\)—so an unsafe shortcut is caught both by what the agent did and by what it left behind\. §[4\.6](https://arxiv.org/html/2608.11234#S4.SS6)reports both signals across published trials\.

### 2\.4Metrics

Each trial is scored by a task\-specific verifier that returns a rewardR∈\[0,1\]R\\in\[0,1\]; when a verifier exposesNNweighted checks,RRis the weighted fraction passed, so partial credit reflects how much of the operational obligation was met rather than a binary outcome \(Appendix[C](https://arxiv.org/html/2608.11234#A3)documents how check weights are derived and frozen\)\. We define four metrics used throughout §[4](https://arxiv.org/html/2608.11234#S4), following theInfraBenchreporting convention \(missing trials count as 0, never as excluded\):

*Mean effective score\.*For an agent–model configuration, the*effective score*on a task isRRif the trial ran, else 0; the*mean effective score*is the mean effective score over the 12 tasks, expressed as a percentage\. This is the headline summary reported for every configuration \(Table[3](https://arxiv.org/html/2608.11234#S4.T3)\)\.

*Attempt Pass@τ\\tau\.*With three independent attempts per task, Attempt Pass@τ\\tauis the fraction of individual attempts, pooled over all tasks, whose reward satisfiesR≥τR\\geq\\tau\. We reportτ∈\{1,0\.5\}\\tau\\in\\\{1,0\.5\\\}\(perfect, and substantially\-solved\); unlike SWE\-bench\-style Pass@kk, this is not an estimator of “probability at least one ofkksamples succeeds”—it is the raw share of attempts clearing the bar, so it directly measures how often a single attempt is trustworthy\.

*Best\-of\-N@τ\\tau\.*For the same three\-pass configurations, Best\-of\-N@τ\\tauis the fraction of the 12 tasks for which the best of the three attempts reachesR≥τR\\geq\\tau—an upper bound on what retrying would buy an operator willing to keep the best of three tries\.

*Mean±\\pmSEM\.*Alongside the three\-pass mean effective score, we report the standard error of the mean \(SEM\) over the 12 per\-task means,SEM=s/12\\mathrm\{SEM\}=s/\\sqrt\{12\}wheressis their sample standard deviation\. A wide SEM signals that a configuration’s mean score depends heavily on a handful of tasks rather than reflecting uniformly middling performance\. Appendix[B](https://arxiv.org/html/2608.11234#A2)restates these four metrics in compact closed form\.

### 2\.5Output

For each trial,InfraBenchreports a per\-trial score with phase\-level breakdown, risk and side\-effect records, execution timeline, trajectory artifacts, and leaderboard\-ready summaries\. The phase breakdown attributes failures to the responsible lifecycle stage or side\-effect category, rather than collapsing them into a binary pass/fail result\.

## 3Experimental Setup

### 3\.1Tasks and Testbed

The current prototype includes 12 seed tasks spanning the four infrastructure layers \(Table[2](https://arxiv.org/html/2608.11234#S3.T2); full catalog with difficulty and check counts in Appendix[A](https://arxiv.org/html/2608.11234#A1)\), drawn from the four sources described in §[1](https://arxiv.org/html/2608.11234#S1): production incident reports, open\-source issue trackers, cloud platform documentation, and systems\-research prototypes\. Each task specifies a target infrastructure layer, a fault or drift condition, and a lifecycle policy that determines which of the four Evaluator gates \(§[2](https://arxiv.org/html/2608.11234#S2)\) apply\. The set deliberately mixes recovery \(power, crash, hung\-node, RAID, WAL\), deployment \(Ceph bootstrap\), and drift\-repair \(scheduler, connection\-pool, federation\-identity\) scenarios, so no single operation type dominates\.

Depending on its layer, a task is provisioned as a Docker container \(L4\), a three\-node VM cluster over libvirt/KVM \(L2–L4\), or a three\-node bare\-metal cluster with out\-of\-band IPMI/BMC control \(L1, L3\)\. All trials run on the CloudLab Wisconsin testbed\[[13](https://arxiv.org/html/2608.11234#bib.bib107)\]on c220g1 nodes \(two 8\-core Xeon E5\-2630 v3, 128 GB RAM, dual 10 GbE\), so every agent operates against the same physical hardware class regardless of backend\. Each attempt starts from a freshly provisioned environment: the Scenario Manager \(§[2](https://arxiv.org/html/2608.11234#S2)\) rebuilds the target state—including injected faults—before the agent operation window opens, so consecutive attempts of the same task are independent\. During the window the agent holds root privileges and the same operational interfaces a human operator would use \(shell, SSH to peer nodes, service managers, and, for L1 tasks, the IPMI control plane\); verifier and oracle scripts are never exposed to the agent\.

Table 2:Preliminary tasks inInfraBench\.
### 3\.2Agents and Models

We evaluate 15 agent–model configurations spanning five coding\-agent command\-line interfaces: Claude Code, Cursor CLI, Gemini CLI, OpenCode, and Qoder CLI, paired with models from nine vendors \(Anthropic, Google, xAI, Cursor, DeepSeek, Xiaomi, Zhipu, Moonshot, and Alibaba; Table[3](https://arxiv.org/html/2608.11234#S4.T3)\)\. A configuration couples an agent CLI—which supplies the scaffolding: the tool\-use loop, context management, and shell integration—with an underlying model that does the reasoning; the two are not independent, and §[4\.1](https://arxiv.org/html/2608.11234#S4.SS1)shows the same CLI can move by more than 25 points depending on the model behind it\. Every configuration pins a fixed model checkpoint: vendor model routers, whose backend selection changes without notice, are excluded so that a reported result stays reproducible against a named model version\.

All CLIs run with vendor\-default settings and no system\-prompt customization beyond the task instruction, and no human intervenes during the operation window\. The CLI defaults—context management, tool\-call policy, and any built\-in system prompt—are part of the configuration under test rather than a nuisance variable: a configuration is the pair \(CLI version, model checkpoint\), and both are pinned and recorded, so the comparison across models within one CLI holds the scaffolding fixed\. Each attempt is bounded by a per\-task agent time budget \(typically 30 minutes\), after which the environment is frozen and handed to the verifier; the exact CLI version used by each trial is recorded in its trajectory artifact\. Each configuration is given the same agent\-visible instruction and the same operation window; the hidden evaluation context \(target layer, fault conditions, oracle scripts, lifecycle policy\) is identical across configurations for a given task, so score differences are attributable to the configuration rather than the environment\.

### 3\.3Evaluation Protocol

Each configuration runs every task three times, each attempt on a freshly provisioned environment, scored by the task’s verifier under the difficulty\-weighted rubric of §[2\.4](https://arxiv.org/html/2608.11234#S2.SS4)\(Appendix[C](https://arxiv.org/html/2608.11234#A3)\)\. One success can be luck, so we report Attempt Pass@τ\\tauand Best\-of\-N@τ\\tau\(§[2\.4](https://arxiv.org/html/2608.11234#S2.SS4)\) alongside the mean: together they separate configurations that solve a task reliably from those that solve it once\.

Attempts are attributed by phase, not by symptom\. Only a failure*before*the agent operation window opens—provisioning, environment start, or agent setup—is retried and left unscored, so testbed noise cannot penalize an agent\. Once the window opens the attempt is scored: a timeout or an abnormal exit still runs the verifier, and damage the agent itself causes—an unreachable peer, a broken route, a disabled interface—is graded by the checks it fails rather than excused as testbed noise, so an agent cannot earn a retry by breaking its own environment\. The residual case is an environment left so damaged that the verifier cannot run at all; such an attempt yields no score and is reported as uncovered rather than silently retried into a better one\. Every configuration is evaluated on the same 12 tasks under the same protocol\.

## 4Results

We report experiments withInfraBenchto understand where infrastructure agents fail, not only whether they complete a task, using the tasks, agents, and protocol of §[3](https://arxiv.org/html/2608.11234#S3)\.

### 4\.1Overall Leaderboard

Table[3](https://arxiv.org/html/2608.11234#S4.T3)presents theInfraBenchleaderboard over 15 agent–model configurations spanning five coding\-agent CLIs\. Each task is scored by a task\-specific verifier in\[0,1\]\[0,1\]with per\-check difficulty weighting \(§[2\.4](https://arxiv.org/html/2608.11234#S2.SS4)\), and we report the*mean effective score*across the 12 tasks\. Every configuration runs each task three times, so we additionally report the standard error of the mean \(SEM\) over tasks and*Attempt Pass*rates—the fraction of individual attempts that reach a perfect score \(Pass@1\) or substantially solve the task atτ=0\.5\\tau\{=\}0\.5\(Pass@0\.5\), and the fraction of tasks with a best\-of\-three perfect pass \(Best\-of\-N@1\)\. These are Attempt Pass rates, not SWE\-bench Pass@kk\. Overall, mean effective scores range from 39\.9% to 87\.7%, suggesting that the benchmark is not saturated even by the strongest agent/model\. The imperfect scores indicate that agents often satisfy visible objectives while still missing deeper operational obligations, such as durable state, distributed consistency, peer safety, and cleanup\. Repeating each task three times further exposes a reliability gap: no configuration passes every attempt, and Attempt Pass@1 sits well below the mean effective score \(e\.g\., Grok 4\.5 scores84\.384\.3yet passes only72\.7%72\.7\\%of attempts\), so a single successful run overstates real dependability\. We next examine where these losses come from: per\-problem results \(§[4\.2](https://arxiv.org/html/2608.11234#S4.SS2)\), where in the operational lifecycle agents fail \(§[4\.3](https://arxiv.org/html/2608.11234#S4.SS3)\), general failure patterns across layers \(§[4\.4](https://arxiv.org/html/2608.11234#S4.SS4)\) and the recurring failure modes they produce \(§[4\.5](https://arxiv.org/html/2608.11234#S4.SS5)\), risk and side effects \(§[4\.6](https://arxiv.org/html/2608.11234#S4.SS6)\), the cost–reliability trade\-off \(§[4\.7](https://arxiv.org/html/2608.11234#S4.SS7)\), and a case study drawn from a real incident \(§[4\.8](https://arxiv.org/html/2608.11234#S4.SS8)\)\.

Table 3:InfraBenchleaderboard\.Difficulty\-weighted mean effective score across the 12 tasks \(per\-check difficulty weighting, §[2\.4](https://arxiv.org/html/2608.11234#S2.SS4)\)\. All 15 configurations run each task three times, so we report Mean±\\pmSEM \(in points\) and Attempt Pass rates: Pass@τ\\tauis the fraction of attempts reaching scoreτ\\tauand Best\-of\-N@1 the fraction of tasks with a best\-of\-three perfect pass\. Parenthesized counts are the numerators: attempts clearing the bar for Pass@τ\\tau, and tasks \(of 12\) for Best\-of\-N@1\. Pass@0\.5 \(substantially solved\) separates clearly from Pass@1 \(perfect\) because difficulty weighting spreads partial scores; intermediate thresholds like0\.90\.9collapse onto Pass@1\. Every configuration pairs an agent CLI with a fixed model checkpoint; Pass@τ\\tau/Best\-of\-N are Attempt Pass rates over the 3\-pass trials, not SWE\-bench Pass@kk\.
### 4\.2Per\-Problem Results

Table[3](https://arxiv.org/html/2608.11234#S4.T3)summarizes each configuration with a single mean; Figure[2](https://arxiv.org/html/2608.11234#S4.F2)breaks that mean down into the 12×\\times15 grid of individual task–configuration effective scores \(exact values in Appendix[D](https://arxiv.org/html/2608.11234#A4)\), with tasks sorted by difficulty \(mean score, hardest at bottom\) and configurations sorted by overall mean \(strongest at left\)\. No task is uniformly easy or uniformly hard in a binary sense—most rows show a gradient rather than a cliff, consistent with the partial\-credit verifiers described in §[2\.4](https://arxiv.org/html/2608.11234#S2.SS4)\. IPMI Power Recovery and Cassandra CORDS are solved by nearly every configuration \(top rows, almost entirely blue\), confirming that L1 hardware control and the adapted CORDS\[[14](https://arxiv.org/html/2608.11234#bib.bib84)\]propagation check are within reach of current agents\. At the other extreme, Ceph Bootstrap and DB WAL Recovery \(bottom rows\) are red for most configurations regardless of overall strength—even the strongest configurations only partially clear Ceph Bootstrap, and only a couple \(e\.g\., Opus 4\.8, Grok 4\.5\) solve DB WAL Recovery outright \(§[4\.5](https://arxiv.org/html/2608.11234#S4.SS5)returns to why\)\. Fileserver RAID10, Ceph Pool Degraded, and Cassandra Hung Recovery show the widest per\-configuration spread—these mid\-difficulty tasks best separate configurations, since neither near\-universal success nor near\-universal failure leaves room to distinguish agents\.

![Refer to caption](https://arxiv.org/html/2608.11234v1/x1.png)Figure 2:Per\-problem effective scores\.All 12 tasks×\\timesthe 15 leaderboard agent–model configurations \(Table[3](https://arxiv.org/html/2608.11234#S4.T3)\)\. Cell color is the effective score in\[0,1\]\[0,1\]\(§[2\.4](https://arxiv.org/html/2608.11234#S2.SS4)\), red \(0\) through yellow to blue \(1\); rows are sorted by task mean \(hardest at bottom\), columns by configuration mean \(strongest at left\)\.
### 4\.3Where Agents Fail Across the Lifecycle

InfraBench’s verifiers score more than whether a fault was fixed: many scored checks specifically test whether a fix survives a restart or leaves no residue behind \(§[2](https://arxiv.org/html/2608.11234#S2)\)\. Bucketing every scored check across all recorded trials by the operational obligation it tests—*Functional*\(the immediate repair works\),*Durability*\(the fix survives a restart or re\-apply\), or*Cleanup*\(no residual or stale state remains\)—exposes a sharp lifecycle gradient \(Figure[3](https://arxiv.org/html/2608.11234#S4.F3)\)\. Functional checks pass 89\.0% of the time \(842/946\): agents are generally competent at making the immediate fault go away\. Durability checks pass 75\.0% of the time \(87/116\): most fixes survive a restart, but a meaningful minority silently revert\. Cleanup checks pass only 35\.2% of the time \(119/338\): agents routinely leave behind exactly the residue the task asks them to remove\. The gap is not uniform across tasks—on Pelican Key Mismatch, only 1 of 34 recorded cleanup checks passes \(the stale incident marker persists in almost every trial, echoed in the case study, §[4\.8](https://arxiv.org/html/2608.11234#S4.SS8)\), while on SLURM/Puppet Cascade cleanup checks pass 39% of the time \(118/304\), split across per\-nodedpkg\-distresidue and driftedMaxJobCountsettings\. Pass rates therefore degrade steeply across the operational lifecycle, dropping from 89\.0% on Functional checks to 75\.0% on Durability checks, and then to 35\.2% on Cleanup checks\. Agents behave as if the task ends when the fault disappears, but the obligations that persist afterward are where most of the score is lost\.

![Refer to caption](https://arxiv.org/html/2608.11234v1/x2.png)Figure 3:Pass rate by operational obligation\.Every scored verifier check across all recorded trials, bucketed by whether it tests the immediate repair \(Functional\), survival of a restart or re\-apply \(Durability\), or absence of residual state \(Cleanup\)\. Stage width tracks the pass rate; annotations give the percentage\-point drops along the Functional\>\>Durability\>\>Cleanup gradient\. Each check is assigned to the phase whose obligation it tests, applied uniformly over the verifier check names so that the pass\-phrased and fail\-phrased wording of one logical check land in the same phase; Appendix[E](https://arxiv.org/html/2608.11234#A5)lists the resulting assignment for every distinct check name\.
### 4\.4General Failure Patterns Across Layers

Failures form a layer\-wise gradient rather than a uniform pattern across layers\. Per\-check scoring exposes this pattern: agents often pass 60–90% of checks before stalling on one lifecycle obligation\. More specifically, we observe the following:L1tasks are consistently solved, suggesting that out\-of\-band hardware controls are within reach;L2task failures usually come from non\-durable changes: agents update the live runtime state but fail to persist the change across restart;L3accounts for most failures, as agents either stop after surface\-level cluster checks pass or lose track of global ordering in long multi\-step operations, leaving hidden quorum, replication, or configuration state inconsistent;L4failures are often functionally successful but operationally incomplete: agents make the data plane available but miss cleanup or drift obligations such as incident markers and stale configuration\.

### 4\.5Recurring Failure Modes

Orthogonal to the layer\-wise gradient, we label recurring*failure modes*by the operational obligation each violates, using verifier check\-level signatures and agent traces across all 15 leaderboard agent–model configurations\. Figure[4](https://arxiv.org/html/2608.11234#S4.F4)reports how many configurations exhibit each mode; a configuration counts as affected if any of its attempts shows it\.

![Refer to caption](https://arxiv.org/html/2608.11234v1/x3.png)Figure 4:Recurring failure modes and their prevalence\.Fraction of the 15 leaderboard agent–model configurations that exhibit each mode, colored by the operational obligation it violates\. Pervasive modes—missing cleanup, deployment residue, and tool\-destructive diagnosis—affect most configurations, including the strongest, whereas destructive side effects are rarer but more dangerous\.Two observations stand out\. First, the most damaging modes are*not*confined to weak models—they are near\-universal\.*Post\-repair cleanup*and*incomplete deployment residue*affect all 15 configurations: nearly every agent that substantively repairs the Pelican federation still leaves the stale incident marker, and none clears the Puppet/dpkg\-distresidue under/etc/slurm\.*Tool\-destructive diagnosis*\(87%\) is equally broad—on DB WAL recovery, agents open the database before preserving the write\-ahead log, so SQLite auto\-checkpoints and discards the very pages needed for recovery\. Second, the modes span the full obligation spectrum the benchmark is designed to expose: durable state \(*non\-persistent fixes*, e\.g\., updating registry/trust state in memory but never persisting it, so the fix is lost on refresh\), distributed and hidden state \(*hidden config\-DB entries*, where a per\-OSDosd\_recovery\_sleep\_hddthrottle survives visible CRUSH repairs, 80%\), and peer safety \(*correct fix, destructive side effect*\), which is rarer but severe when it occurs\.

These modes share a common shape: the agent satisfies the visible objective while violating an implicit obligation—durable state, intact peers, or a closed\-out incident record—that a pass/fail check would miss\. By making each obligation an explicit, checkable gate,InfraBenchcredits a trial for the repair it achieves and debits it for the obligation it leaves behind\.

### 4\.6Risk and Side Effects

InfraBench’s Risk Monitor \(§[2](https://arxiv.org/html/2608.11234#S2)\) includes an LLM\-judge stage over action traces: for each retained trajectory it classifies recorded commands in context against a seven\-type danger taxonomy \(destructive filesystem operations, disk/RAID/LVM operations, network disruption, safety/privilege bypass, unsafe restarts, cross\-service interference, and evaluator\-harness probing\) and emits structured review findings\. In this prototype the judge runs on archived trajectories rather than as a live feed during the operation window\. We report results for all 266 trials whose CLI records a machine\-readable action log, spanning 9 of the 15 leaderboard configurations over the Claude Code, Cursor CLI, Gemini CLI, and OpenCode backends; the remaining six \(the three Qoder CLI configurations and the earlier Gemini and Fable 5 campaigns\) predate per\-command trace capture in those adapters, so their commands were never recorded and they are reported as uncovered rather than silently dropped\. As an independent, ground\-truth cross\-check, we also use the preservation\-oriented verifier checks already scored in §[4\.3](https://arxiv.org/html/2608.11234#S4.SS3)\(Durability and Cleanup\), which penalize collateral damage directly rather than inferring it from actions\.

Figure[5](https://arxiv.org/html/2608.11234#S4.F5)summarizes the Risk Monitor findings\. Of 9,351 recorded commands, only 76 \(0\.8%\) were flagged as genuinely dangerous—agents are conservative by default, and the large majority of write operations \(cluster repairs, targeted power cycles, storage reassembly\) are legitimate, in\-scope remediation rather than collateral damage\. The flagged actions concentrate: 43 of the 266 trials contain at least one, and five of the seven danger types occur at all—no command was flagged for network disruption or interference with unrelated services\.

![Refer to caption](https://arxiv.org/html/2608.11234v1/x4.png)Figure 5:Dangerous actions identified per configuration\.Risk Monitor findings over every recorded command in the 266 trials that carry an action log, for the 9 leaderboard configurations whose CLI records one, classified against a seven\-type danger taxonomy \(five types occur at least once\)\. Counts, not rates, since totals differ by both trial count and average trial length\.Two findings stand out\. First, the danger is dominated by two patterns that are not random carelessness\. Destructive operations on live state account for 29 of the 76 flagged actions and safety\-mechanism bypasses for another 16, while*evaluator\-harness probing*—agents inspecting the hidden scoring artifacts adjacent to task\-visible logs, or in one case running the task’s own grader and reading the reward file—accounts for 17\. Probing appears in 7 of the 9 audited configurations and is overwhelmingly concentrated on DB WAL Recovery \(15 of its 17 actions\), the one task where the answer is unrecoverable from the environment—when agents cannot solve a task, they go looking for how it is graded\.

Second, the dangerous actions cluster on a few recurring patterns rather than scattering across many one\-off mistakes\. Every one of the 16 safety bypasses but one is the same move on Ceph Bootstrap: five different configurations—Claude Opus 4\.8, Grok 4\.5, Gemini 3\.6 Flash, Composer 2\.5, and MiMo V2\.5—independently unloaded or tore down mandatory access control*on every node*to get past a cephadm parsing bug, and one of them went further and bind\-mounted an empty file over the kernel’s AppArmor interface cluster\-wide\. That five independent configurations converge on the same bypass suggests a shortcut learned from common deployment guidance, not an isolated mistake\. The destructive operations concentrate on state the task was meant to preserve: on Cassandra Hung Recovery one MiMo V2\.5 trial wiped the node’s system keyspace, commitlog and caches nine separate times while trying to force it back into the ring, and on DB WAL Recovery three DeepSeek trials deleted the very write\-ahead log the task asks them to recover\. The most severe individual incidents are on Pelican Key Mismatch, where a Composer 2\.5 trial and a Grok 4\.5 trial each rewrote the live registry database back to the pre\-incident issuer key—re\-injecting the very incident the task asked them to close—and two Gemini 3\.6 Flash trials hand\-edited the registry’s trust material by SQL instead of through the service\. These incidents illustrate why risk cannot be inferred from the pass/fail outcome alone: a trial that ultimately scores well can still take an action a production operator would treat as a serious incident in its own right\.

The independent cross\-check corroborates the Risk Monitor picture at the aggregate level: the same trials that Figure[3](https://arxiv.org/html/2608.11234#S4.F3)shows failing Cleanup checks 64\.8% of the time are, by construction, the trials leaving verifiable residual state—a ground\-truth signal that does not depend on the judge taxonomy\. Appendix[F](https://arxiv.org/html/2608.11234#A6)summarizes the concrete incidents behind the claims above\.

![Refer to caption](https://arxiv.org/html/2608.11234v1/x5.png)Figure 6:Cost vs\. reliability across 3\-pass campaigns\.Estimated full three\-pass campaign cost \(USD, log scale\) against difficulty\-weighted Attempt Pass@0\.5 \(R≥0\.5R\\\!\\geq\\\!0\.5, substantially solved\)—the same reliability metric as Table[3](https://arxiv.org/html/2608.11234#S4.T3)\. Marker shape denotes the agent CLI and fill color the model provider; the dashed line traces the cost–reliability Pareto frontier, with frontier models in bold\. The costliest configurations are not the most reliable\.![Refer to caption](https://arxiv.org/html/2608.11234v1/x6.png)Figure 7:Token effort vs\. score\.Total input tokens consumed over the three\-pass campaign \(log scale\) against difficulty\-weighted mean score\. Qoder CLI is billed in credits rather than tokens, so token counts are undefined for its configurations; they appear in the cost view \(Figure[6](https://arxiv.org/html/2608.11234#S4.F6)\) and are absent only from this token axis\. The configurations most prone to redundant looping \(the Gemini Flash models\) sit far to the right without a commensurate score gain, while the Claude configurations reach comparable or higher scores at an order of magnitude fewer tokens\.
### 4\.7Cost, Token Efficiency, and Reliability

Beyond scores,InfraBenchrecords the estimated API/credit cost and the token usage of each three\-pass campaign\. Two complementary views summarize the economics: what a campaign*costs*against how reliably it solves tasks \(Figure[6](https://arxiv.org/html/2608.11234#S4.F6)\), and how many tokens it*consumes*against the score it reaches \(Figure[7](https://arxiv.org/html/2608.11234#S4.F7)\)\.

Spend vs\. reliability \(Figure[6](https://arxiv.org/html/2608.11234#S4.F6)\)\.Cost varies by more than two orders of magnitude for the*same*12 tasks—from well under $1 \(MiMo V2\.5 $0\.82, DeepSeek V4 Flash $1\.49\) to about $194 \(Gemini 3\.5 Flash\)—so the price of operating an infrastructure agent is a first\-class axis, not a rounding error\. Yet spend and reliability are only weakly coupled: the Pareto frontier is anchored by Grok 4\.5 \(Cursor CLI\), which reaches 84\.8% Attempt Pass@0\.5 at just $15\.5, and by Fable 5, the most reliable configuration \(91\.4%\) at $53\.5\. The costliest points sit*off*the frontier: the Gemini Flash and Pro configurations spend $60–$194 yet trail the frontier by 10–20 points of reliability\.

Token effort vs\. score \(Figure[7](https://arxiv.org/html/2608.11234#S4.F7)\)\.The token view explains*why*the expensive points are expensive: a cheap per\-token price does not imply a cheap campaign\. Gemini 3\.5 Flash is billed at only $1\.5/Mtok of input yet is the single most expensive configuration \($194\), because it loops for hundreds to thousands of tool\-calls on tasks it never solves—onevm\-pelicanattempt alone consumed 60M input tokens over 1,630 steps\. The token\-efficient Claude configurations reach comparable or higher scores at an order of magnitude fewer tokens, so token efficiency—not the per\-token list price—is what separates cheap campaigns from expensive ones\.

For an operator, paying more—or picking a nominally “cheap” model—does not buy dependability: model choice and agent\-level efficiency matter as much as the headline price, andInfraBenchmakes this trade\-off measurable\.

### 4\.8Case Study: An Incident at a University Center

The Pelican\[[37](https://arxiv.org/html/2608.11234#bib.bib106)\]task is derived from a real incident at a high throughput computing \(HTC\) center\. After an Origin host was rebuilt, its new identity drifted out of sync with the federation registry that authorizes it, blocking all client traffic\. A correct trial must reconcile cross\-component trust, restore the dependent cache and routing services, and close out the incident state\. The task is graded by a 15\-check verifier; no agent/model configuration clears all 15 checks across its three passes \(a single Grok 4\.5 attempt closes the incident completely, but not repeatably\), and the partial scores include three distinct failure patterns:

\(1\) Trusting a surface status signal\.Some agents read a high\-level “approved” indicator and stop, without checking the underlying key material that the indicator is supposed to summarize\. They report success while the registry still holds the stale trust record, leaving the root cause untouched\.

\(2\) Correct fix, destructive side effect\.Other agents reconcile the trust mismatch correctly but, in doing so, render a peer node unreachable, so end\-to\-end retrieval still fails\. The verifier credits the core repair and localizes the regression to the peer node, rather than scoring the trial as a non\-fix\.

\(3\) Functional fix, missing cleanup\.The strongest configurations succeeded in 14 of 15 checks: they restore federation trust and end\-to\-end data retrieval, but leave behind the marker that signals an open incident to operators\. The service is fully usable, yet by the standard of the infrastructure operator, the incident is not closed\.

This single task concretely instantiates several of the modes in Figure[4](https://arxiv.org/html/2608.11234#S4.F4): even the strongest configurations reach 14 of 15 checks on most attempts, yet no configuration closes the incident out on every attempt—a mix of the near\-universal cleanup mode with hidden\-state and destructive side\-effect modes, all within one real\-world incident\.

## 5Discussions & Future Work

The work presented in this paper suggests many opportunities for follow\-up improvements\.

Capability vs\. cost\.The same agent CLI varies by more than 25 points across underlying models \(§[4\.1](https://arxiv.org/html/2608.11234#S4.SS1)\), and §[4\.7](https://arxiv.org/html/2608.11234#S4.SS7)shows that spend and reliability are only weakly coupled—so the mean effective score alone conflates capability with cost\. We plan to add cost\-normalized metrics \(score per token, per dollar, per wall\-clock minute\) so that a leaderboard entry reports not just how well a configuration does, but how much that performance costs to obtain\.

From derived to native risk instrumentation\.§[4\.6](https://arxiv.org/html/2608.11234#S4.SS6)’s risk analysis is reconstructed post hoc from recorded action logs and preservation checks, not from a live monitor—the current prototype’s Risk Monitor \(§[2](https://arxiv.org/html/2608.11234#S2)\) observes what agents already log, rather than emitting its own events during the operation window\. Native instrumentation—hooking the unusedscenario\_eventsandperiodic\_verifierinterfaces already defined in the executor to emit risk events as they happen—would let us rank incidents by severity rather than only counting them\. Ranking matters: §[4\.6](https://arxiv.org/html/2608.11234#S4.SS6)shows that a fix that quietly reintroduces the incident it was meant to close is qualitatively worse than one that merely inspects the grading harness, but both currently register as one flagged action\.

Lifecycle\-phase bucketing is heuristic\.The Functional/Durability/Cleanup categorization in §[4\.3](https://arxiv.org/html/2608.11234#S4.SS3)is a keyword rule over verifier check names, audited by hand \(Appendix[E](https://arxiv.org/html/2608.11234#A5)\) but not part of the task specification format itself\. A cleaner design would have task authors tag each verifier check with its lifecycle phase directly, removing the need for post hoc inference as the task set grows\.

Task coverage\.The task set in the current prototype is small \(12 tasks\) and skewed toward L3 distributed systems; L1 hardware and L2 local\-systems coverage is thin by comparison\. We plan to derive more tasks with collaborators and call for the collective efforts of the community to fully realize the potential ofInfraBench\.

## Acknowledgments

The authors thank the anonymous reviewers for their invaluable feedback\. The authors also thank system administrators and practitioners at UW\-Madison’s Division of Information Technology \(DoIT\), Center for High Throughput Computing \(CHTC\), Computer Science Department IT \(CIDS IT\), and ISU’s ARA Wireless Living Lab \([arawireless\.org](https://arawireless.org/)\) for sharing their real\-world infrastructure management experiences\. This work was supported in part by National Science Foundation \(NSF\) under grants \#1943204, \#2130889, \#2402858, and \#2402859\. Any opinions, findings, and conclusions expressed in this material are those of the authors and do not necessarily reflect the views of the sponsor\.

## References

- \[1\]\(2023\)Recommending root\-cause and mitigation steps for cloud incidents using large language models\.InProceedings of the 45th International Conference on Software Engineering \(ICSE\),pp\. 1737–1749\.Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p2.1)\.
- \[2\]Amazon Web Services\(2026\)AWS lambda\.Amazon\.com, Inc\.\.Note:Accessed: May 20, 2026External Links:[Link](https://aws.amazon.com/lambda/)Cited by:[1st item](https://arxiv.org/html/2608.11234#S1.I1.i1.p1.1)\.
- \[3\]E\. Androulaki, A\. Barger, V\. Bortnikov, C\. Cachin, K\. Christidis, A\. De Caro, D\. Enyeart, C\. Ferris, G\. Laventman, Y\. Manevich, S\. Muralidharan, C\. Murthy, B\. Nguyen, M\. Sethi, G\. Singh, K\. Smith, A\. Sorniotti, C\. Stathakopoulou, M\. Vukolić, S\. W\. Cocco, and J\. Yellick\(2018\)Hyperledger fabric: a distributed operating system for permissioned blockchains\.InProceedings of the Thirteenth EuroSys Conference \(EuroSys\),EuroSys ’18\.External Links:[Link](https://doi.org/10.1145/3190508.3190538),[Document](https://dx.doi.org/10.1145/3190508.3190538)Cited by:[1st item](https://arxiv.org/html/2608.11234#S1.I1.i1.p1.1)\.
- \[4\]E\. Androulaki, A\. Barger, V\. Bortnikov, C\. Cachin, K\. Christidis, A\. De Caro, D\. Enyeart, C\. Ferris, G\. Laventman, Y\. Manevich,et al\.\(2018\)Hyperledger fabric: a distributed operating system for permissioned blockchains\.InProceedings of the thirteenth EuroSys conference,pp\. 1–15\.Cited by:[3rd item](https://arxiv.org/html/2608.11234#S2.I1.i3.p1.1)\.
- \[5\]CephCEPHFS QUOTAS\.Note:Accessed: June 5, 2025[https://docs\.ceph\.com/en/latest/cephfs/quota/](https://docs.ceph.com/en/latest/cephfs/quota/)Cited by:[3rd item](https://arxiv.org/html/2608.11234#S2.I1.i3.p1.1)\.
- \[6\]J\. A\. Chacko, R\. Mayer, and H\. Jacobsen\(2021\)Why do my blockchain transactions fail? a study of hyperledger fabric\.InProceedings of the 2021 international conference on management of data,pp\. 221–234\.Cited by:[1st item](https://arxiv.org/html/2608.11234#S1.I1.i1.p1.1)\.
- \[7\]J\. A\. Chacko, R\. Mayer, and H\. Jacobsen\(2023\)How to optimize my blockchain? a multi\-level recommendation approach\.Proceedings of the ACM on Management of Data1\(1\),pp\. 1–27\.Cited by:[1st item](https://arxiv.org/html/2608.11234#S1.I1.i1.p1.1)\.
- \[8\]Y\. Chen, M\. Shetty, G\. Somashekar, M\. Ma, Y\. Simmhan, J\. Mace, C\. Bansal, S\. Rajmohan, and D\. Zhang\(2025\)AIOpsLab: a holistic framework to evaluate AI agents for enabling autonomous clouds\.InProceedings of Machine Learning and Systems \(MLSys\),Cited by:[Table 1](https://arxiv.org/html/2608.11234#S1.T1.6.6.2),[§1](https://arxiv.org/html/2608.11234#S1.p2.1)\.
- \[9\]C\. Clark, K\. Fraser, S\. Hand, J\. G\. Hansen, E\. Jul, C\. Limpach, I\. Pratt, and A\. Warfield\(2005\)Live migration of virtual machines\.InProceedings of the 2nd Symposium on Networked Systems Design & Implementation \(NSDI\),pp\. 273–286\.Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p1.1)\.
- \[10\]J\. Clark, Y\. Su, S\. M\. R\. Pial, Y\. Tian, L\. Gniedziejko, H\. Jacobsen, Y\. Chen, and T\. Xu\(2026\)SREGym: a live benchmark for AI SRE agents with high\-fidelity failure scenarios\.arXiv preprint arXiv:2605\.07161\.Cited by:[Table 1](https://arxiv.org/html/2608.11234#S1.T1.2.2.3),[§1](https://arxiv.org/html/2608.11234#S1.p2.1)\.
- \[11\]J\. Dean and S\. Ghemawat\(2008\)MapReduce: simplified data processing on large clusters\.Communications of the ACM51\(1\),pp\. 107–113\.Cited by:[1st item](https://arxiv.org/html/2608.11234#S1.I1.i1.p1.1)\.
- \[12\]M\. Du, F\. Li, G\. Zheng, and V\. Srikumar\(2017\)Deeplog: anomaly detection and diagnosis from system logs through deep learning\.InProceedings of the 2017 ACM SIGSAC conference on computer and communications security,pp\. 1285–1298\.Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p1.1)\.
- \[13\]D\. Duplyakin, R\. Ricci, A\. Maricq, G\. Wong, J\. Duerig, E\. Eide, L\. Stoller, M\. Hibler, D\. Johnson, K\. Webb, A\. Akella, K\. Wang, G\. Ricart, L\. Landweber, C\. Elliott, M\. Zink, E\. Cecchet, S\. Kar, and P\. Mishra\(2019\)The design and operation of CloudLab\.InProceedings of the 2019 USENIX Annual Technical Conference \(USENIX ATC ’19\),pp\. 1–14\.Cited by:[§3\.1](https://arxiv.org/html/2608.11234#S3.SS1.p2.1)\.
- \[14\]A\. Ganesan, R\. Alagappan, A\. C\. Arpaci\-Dusseau, and R\. H\. Arpaci\-Dusseau\(2017\)Redundancy does not imply fault tolerance: analysis of distributed storage reactions to file\-system faults\.ACM Transactions on Storage \(TOS\)13\(3\),pp\. 1–33\.Cited by:[Table 2](https://arxiv.org/html/2608.11234#S3.T2.1.6.5.4.1.1),[§4\.2](https://arxiv.org/html/2608.11234#S4.SS2.p1.1)\.
- \[15\]O\. R\. Gatla, M\. Zheng, M\. Hameed, V\. Dubeyko, A\. Manzanares, F\. Blagojevic, C\. Guyot, and R\. Mateescu\(2018\)Towards robust file system checkers\.ACM Transactions on Storage \(TOS\)14\(4\),pp\. 1–25\.Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p1.1)\.
- \[16\]H\. S\. Gunawi, M\. Hao, R\. O\. Suminto, A\. Laksono, A\. D\. Satria, J\. Adityatama, and K\. J\. Eliazar\(2016\)Why does the cloud stop computing? lessons from hundreds of service outages\.InProceedings of the Seventh ACM Symposium on Cloud Computing,pp\. 1–16\.Cited by:[3rd item](https://arxiv.org/html/2608.11234#S1.I1.i3.p1.1),[§1](https://arxiv.org/html/2608.11234#S1.p1.1)\.
- \[17\]H\. S\. Gunawi, M\. Hao, T\. Leesatapornwongsa, T\. Patana\-anake, T\. Do, J\. Adityatama, K\. J\. Eliazar, A\. Laksono, J\. F\. Lukman, V\. Martin, and A\. D\. Satria\(2014\)What bugs live in the cloud? a study of 3000\+ issues in cloud systems\.InProceedings of the ACM Symposium on Cloud Computing \(SoCC\),pp\. 1–14\.Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p1.1)\.
- \[18\]R\. Han, O\. R\. Gatla, M\. Zheng, J\. Cao, D\. Zhang, D\. Dai, Y\. Chen, and J\. Cook\(2022\)A study of failure recovery and logging of high\-performance parallel file systems\.ACM Transactions on Storage \(TOS\)18\(2\),pp\. 1–44\.Cited by:[3rd item](https://arxiv.org/html/2608.11234#S1.I1.i3.p1.1),[§1](https://arxiv.org/html/2608.11234#S1.p1.1)\.
- \[19\]R\. Han, C\. Shi, T\. Mahmud, Z\. Yang, V\. Esaulov, L\. Wan, Y\. Chen, J\. Wayda, M\. Wolf, and M\. Zheng\(2024\)Revisiting erasure codes: a configuration perspective\.InProceedings of the 16th ACM Workshop on Hot Topics in Storage and File Systems,pp\. 93–100\.Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p1.1)\.
- \[20\]M\. R\. Hines and K\. Gopalan\(2009\)Post\-copy based live virtual machine migration using adaptive pre\-paging and dynamic self\-ballooning\.InProceedings of the 2009 ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments \(VEE\),pp\. 51–60\.Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p1.1)\.
- \[21\]P\. Hosek and C\. Cadar\(2013\)Safe software updates via multi\-version execution\.In2013 35th International Conference on Software Engineering \(ICSE\),pp\. 612–621\.Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p1.1)\.
- \[22\]T\. U\. Islam, J\. O\. Boateng, M\. Nadim, G\. Zu, M\. Shahid, X\. Li, T\. Zhang, S\. Reddy, W\. Xu, A\. Atalar,et al\.\(2025\)Design and implementation of ara wireless living lab for rural broadband and applications\.Computer Networks263,pp\. 111188\.Note:[https://arawireless\.org/](https://arawireless.org/)Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p5.1)\.
- \[23\]S\. Jha, R\. Arora, Y\. Watanabe, T\. Yanagawa, Y\. Chen, J\. Clark, B\. Bhavya,et al\.\(2025\)ITBench: evaluating AI agents across diverse real\-world IT automation tasks\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\),Cited by:[Table 1](https://arxiv.org/html/2608.11234#S1.T1.5.5.4),[§1](https://arxiv.org/html/2608.11234#S1.p2.1)\.
- \[24\]C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan\(2024\)SWE\-bench: can language models resolve real\-world GitHub issues?\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[Table 1](https://arxiv.org/html/2608.11234#S1.T1.8.10.1.1),[§1](https://arxiv.org/html/2608.11234#S1.p2.1)\.
- \[25\]P\. Jin, S\. Zhang, M\. Ma, H\. Li, Y\. Kang, L\. Li, Y\. Liu, B\. Qiao, C\. Zhang, P\. Zhao, S\. He, F\. Sarro, Y\. Dang, S\. Rajmohan, Q\. Lin, and D\. Zhang\(2023\)Assess and summarize: improve outage understanding with large language models\.InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering \(ESEC/FSE\),Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p2.1)\.
- \[26\]C\. Lou, P\. Huang, and S\. Smith\(2020\)Understanding, detecting and localizing partial failures in large system software\.In17th USENIX Symposium on Networked Systems Design and Implementation \(NSDI\),pp\. 559–574\.Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p1.1)\.
- \[27\]A\. Mashtizadeh, E\. Celebi, T\. Garfinkel, and M\. Cai\(2011\)The design and evolution of live storage migration in VMware ESX\.In2011 USENIX Annual Technical Conference \(USENIX ATC\),Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p1.1)\.
- \[28\]B\. Maurer\(2015\)Fail at scale: reliability in the face of rapid change\.Communications of the ACM58\(11\),pp\. 44–49\.Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p1.1)\.
- \[29\]M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi,et al\.\(2026\)Terminal\-Bench: benchmarking agents on hard, realistic tasks in command line interfaces\.InThe Fourteenth International Conference on Learning Representations \(ICLR\),Cited by:[Table 1](https://arxiv.org/html/2608.11234#S1.T1.8.8.2),[§1](https://arxiv.org/html/2608.11234#S1.p2.1),[Table 2](https://arxiv.org/html/2608.11234#S3.T2.1.11.10.4.1.1)\.
- \[30\]U\. of Wisconsin\-Madison\(2026\)Center for High Throughput Computing \(CHTC\), UW\-Madison\.Note:Accessed: June 15, 2026External Links:[Link](https://chtc.wisc.edu/)Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p5.1)\.
- \[31\]U\. of Wisconsin\-Madison\(2026\)Division of Information Technology \(DoIT\), UW\-Madison\.Note:Accessed: June 15, 2026External Links:[Link](https://it.wisc.edu/)Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p5.1)\.
- \[32\]U\. of Wisconsin\-Madison\(2026\)IT of Computer Sciences Department at UW\-Madison \(CIDS\-IT\), UW\-Madison\.Note:Accessed: June 15, 2026External Links:[Link](https://www.cs.wisc.edu/)Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p5.1)\.
- \[33\]M\. Shetty, Y\. Chen, G\. Somashekar, M\. Ma, Y\. Simmhan, X\. Zhang, J\. Mace, D\. Vandevoorde, P\. Las\-Casas, S\. M\. Gupta, S\. Nath, C\. Bansal, and S\. Rajmohan\(2024\)Building AI agents for autonomous clouds: challenges and design principles\.arXiv preprint arXiv:2407\.12165\.Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p2.1)\.
- \[34\]C\. Shi, A\. Manschula, T\. Mahmud, Z\. Yang, M\. Zheng, Y\. Chen, J\. Wayda, M\. Wolf, and B\. Bang\(2025\)Revisiting computational storage for data integrity and security\.arXiv preprint arXiv:2504\.15293\.Cited by:[1st item](https://arxiv.org/html/2608.11234#S1.I1.i1.p1.1)\.
- \[35\]K\. Shvachko, H\. Kuang, S\. Radia, and R\. Chansler\(2010\)The hadoop distributed file system\.In2010 IEEE 26th symposium on mass storage systems and technologies \(MSST\),pp\. 1–10\.Cited by:[1st item](https://arxiv.org/html/2608.11234#S1.I1.i1.p1.1)\.
- \[36\]Y\. Tang, K\. Zhu, B\. Ruan, C\. Zhang, M\. Yang, H\. Li, S\. Guo, T\. Shi, Z\. Li, C\. Kruegel, G\. Vigna, D\. Song, W\. Y\. Wang, L\. Wang, Y\. Ding, Z\. Liang, and W\. Guo\(2026\)DevOps\-Gym: benchmarking AI agents in software DevOps cycle\.arXiv preprint arXiv:2601\.20882\.Cited by:[Table 1](https://arxiv.org/html/2608.11234#S1.T1.7.7.2),[§1](https://arxiv.org/html/2608.11234#S1.p2.1)\.
- \[37\]The Pelican Platform Team\(2024\)The Pelican platform: a data federation platform powering the open science data federation\.Note:[https://pelicanplatform\.org/](https://pelicanplatform.org/)Center for High Throughput Computing, University of Wisconsin–Madison and Morgridge Institute for ResearchCited by:[§1](https://arxiv.org/html/2608.11234#S1.p5.1),[§4\.8](https://arxiv.org/html/2608.11234#S4.SS8.p1.1)\.
- \[38\]S\. Weil, S\. A\. Brandt, E\. L\. Miller, D\. D\. Long, and C\. Maltzahn\(2006\)Ceph: a scalable, high\-performance distributed file system\.InProceedings of the 7th Conference on Operating Systems Design and Implementation \(OSDI’06\),pp\. 307–320\.Cited by:[1st item](https://arxiv.org/html/2608.11234#S1.I1.i1.p1.1),[§1](https://arxiv.org/html/2608.11234#S1.p5.1)\.
- \[39\]E\. Xu, M\. Zheng, F\. Qin, Y\. Xu, and J\. Wu\(2019\)Lessons and actions: what we learned from 10k\{\\\{ssd\-related\}\\\}storage system failures\.In2019 USENIX Annual Technical Conference \(USENIX ATC 19\),pp\. 961–976\.Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p1.1)\.
- \[40\]J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press\(2024\)SWE\-agent: agent\-computer interfaces enable automated software engineering\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p2.1)\.
- \[41\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao\(2023\)ReAct: synergizing reasoning and acting in language models\.InThe Eleventh International Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p2.1)\.
- \[42\]A\. B\. Yoo, M\. A\. Jette, and M\. Grondona\(2003\)Slurm: simple linux utility for resource management\.InWorkshop on job scheduling strategies for parallel processing,pp\. 44–60\.Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p5.1),[3rd item](https://arxiv.org/html/2608.11234#S2.I1.i3.p1.1)\.
- \[43\]M\. Zheng, D\. Zhang, and A\. Dajani\(2026\)On fault tolerance of data storage systems: a holistic perspective\.Fault Tolerance in Modern Engineering Systems\.Cited by:[§1](https://arxiv.org/html/2608.11234#S1.p1.1)\.

## Appendix ATask Catalog

Table[4](https://arxiv.org/html/2608.11234#A1.T4)gives the full per\-task metadata underlying Table[2](https://arxiv.org/html/2608.11234#S3.T2): difficulty label and verifier check count, both taken from the same catalog that drives scoring \(§[2\.4](https://arxiv.org/html/2608.11234#S2.SS4)\)\.

Table 4:Full task catalog\.Difficulty and verifier check counts are assigned per task independent of any agent’s performance on it\.
## Appendix BMetric Definitions, Restated

This appendix restates the metrics of §[2\.4](https://arxiv.org/html/2608.11234#S2.SS4)in compact form for reference\. LetRc,t,p∈\[0,1\]R\_\{c,t,p\}\\in\[0,1\]be the reward of configurationccon tasktt, passpp\(each configuration has up to three passes\)\.

Mean effective score​\(c\)\\displaystyle\\text\{Mean effective score\}\(c\)=10012​∑t=112R¯c,t,\\displaystyle=\\frac\{100\}\{12\}\\sum\_\{t=1\}^\{12\}\\overline\{R\}\_\{c,t\},R¯c,t=meanp​\(Rc,t,p\)\\displaystyle\\overline\{R\}\_\{c,t\}=\\text\{mean\}\_\{p\}\(R\_\{c,t,p\}\)SEM​\(c\)\\displaystyle\\text\{SEM\}\(c\)=sc12,\\displaystyle=\\frac\{s\_\{c\}\}\{\\sqrt\{12\}\},sc=sample std\. of​\{R¯c,t\}t=112\\displaystyle s\_\{c\}=\\text\{sample std\.\\ of \}\\\{\\overline\{R\}\_\{c,t\}\\\}\_\{t=1\}^\{12\}Attempt Pass@​τ​\(c\)\\displaystyle\\text\{Attempt Pass@\}\\tau\(c\)=100nc​∑t,p𝟏​\[Rc,t,p≥τ\],\\displaystyle=\\frac\{100\}\{n\_\{c\}\}\\sum\_\{t,p\}\\mathbf\{1\}\[R\_\{c,t,p\}\\geq\\tau\],nc=total attempts for​c\\displaystyle n\_\{c\}=\\text\{total attempts for \}cBest\-of\-N@​τ​\(c\)\\displaystyle\\text\{Best\-of\-N@\}\\tau\(c\)=10012​∑t=112𝟏​\[maxp⁡Rc,t,p≥τ\]\\displaystyle=\\frac\{100\}\{12\}\\sum\_\{t=1\}^\{12\}\\mathbf\{1\}\\big\[\\max\_\{p\}R\_\{c,t,p\}\\geq\\tau\\big\]We reportτ∈\{1,0\.5\}\\tau\\in\\\{1,0\.5\\\}in the leaderboard \(Table[3](https://arxiv.org/html/2608.11234#S4.T3)\) and useτ=0\.5\\tau=0\.5for the cost–reliability figure \(§[4\.7](https://arxiv.org/html/2608.11234#S4.SS7)\)\. Under per\-check difficulty weighting few attempts score in\[0\.7,1\)\[0\.7,1\), so intermediate thresholds like0\.90\.9collapse onto Pass@1;τ=0\.5\\tau=0\.5instead captures attempts that substantially solve a task \(its easy checks\) without clearing the hardest, which is where partial credit concentrates\. Attempt Pass@τ\\taupools over both tasks and passes; it is not an estimate of the probability that at least one of several independent samples succeeds, unlike SWE\-bench\-style Pass@kk\.

## Appendix CDifficulty\-Weighted Check Scoring

This appendix documents how per\-check difficulty weights are derived, applied, and frozen; the rewardRRused throughout §[2\.4](https://arxiv.org/html/2608.11234#S2.SS4)and §[4](https://arxiv.org/html/2608.11234#S4)is computed under these weights\.

*Weight derivation\.*For every task whose verifier exposes individually scored checks \(10 of the 12 tasks; the remaining two use graders that emit only an aggregate pass/fail reward and stay binary\), we compute each checkcc’s empirical pass ratepcp\_\{c\}over all recorded attempts of the evaluated configurations and assign

wc=\(1−pc\)\+0\.1\.w\_\{c\}\\;=\\;\(1\-p\_\{c\}\)\\;\+\\;0\.1\.A check that nearly every attempt passes carries little discriminative signal and receives a weight near the0\.10\.1floor; a check that no attempt passes keeps the maximum weight1\.11\.1\. The floor keeps every satisfied obligation worth a nonzero amount, so a trial is still credited for routine repairs rather than scored only on the hardest check\.

*Application\.*A trial’s reward is the weighted fraction of scored checks passed,R=∑c∈passedwc/∑cwcR=\\sum\_\{c\\,\\in\\,\\mathrm\{passed\}\}w\_\{c\}\\,/\\,\\sum\_\{c\}w\_\{c\}; unscored \(informational\) checks are excluded\. This deflates near\-miss scores dominated by easy checks—passing 7 of 8 checks but missing the hardest one drops from7/8=0\.8757/8=0\.875under uniform weighting to≈0\.53\{\\approx\}0\.53—while configurations that clear rarely\-passed checks gain, which is what widens the separation reported in §[4](https://arxiv.org/html/2608.11234#S4)\.

*Freezing and auditability\.*Weights were computed once, over the complete campaign population reported in this paper, and are then frozen as per\-task sidecar files shipped with the released task packages\. Publishing additional configurations does not change published scores; any future re\-derivation of weights is a versioned, announced re\-scoring event\. Becausepcp\_\{c\}is estimated from the evaluated population, the weights are population\-dependent by construction; freezing them converts the measure into a fixed, auditable rubric\.

## Appendix DPer\-Problem Detailed Results

Table[5](https://arxiv.org/html/2608.11234#A4.T5)gives the exact effective score \(§[2\.4](https://arxiv.org/html/2608.11234#S2.SS4)\) underlying every cell of Figure[2](https://arxiv.org/html/2608.11234#S4.F2), to two decimal places\. Relative to the figure, the table is transposed \(configurations as rows, tasks as columns\) so it fits the page width; configurations are ordered by mean score \(strongest at top\) and tasks by difficulty \(hardest at right\)\.

Table 5:Full per\-problem effective\-score table\.Exact values underlying Figure[2](https://arxiv.org/html/2608.11234#S4.F2), transposed so tasks are columns \(hardest at right\) and the 15 configurations are rows \(strongest at top\)\.
## Appendix ELifecycle\-Phase Check Mapping

Figure[3](https://arxiv.org/html/2608.11234#S4.F3)buckets every scored verifier check into Functional, Durability, or Cleanup by a deterministic rule over the check name, derived from the obligation each check tests \(§[4\.3](https://arxiv.org/html/2608.11234#S4.SS3)\): a name mentioning a restart, reboot, or re\-apply is Durability; a name mentioning a residue marker, packaging leftover, or drifted setting is Cleanup; everything else is Functional\. The rule maps both the pass\-phrased and fail\-phrased wording of the same logical check to the same bucket\. Below is the complete list of distinct scored check names observed across all recorded trials, grouped by the bucket they were assigned to, for audit\.

Functional\(29 distinct check names\):6\-way concurrent srun workload failed baseline;6\-way concurrent srun workload passes baseline;Origin is still using the rebuilt host key, not the pre\-incident stale key;Registry approval JWKS does not match the active Origin issuer JWKS;Registry namespace approval matches the active Origin issuer JWKS;a dm\-delay target is still in place under an array member;all\_nodes\_un;all\_three\_nodes\_un;client could not retrieve the object through the Pelican federation path;client retrieved the object through the Pelican federation path;could not read namespace approval state;data\_consistent;ipmi\_power\_on;namespace approval exists for /syscraft/public;nic\_restored;no dm\-delay target remains under any disk\[0\-3\];no obvious plain HTTP/file\-server bypass is listening on node2;node1\_removed\_from\_ring;quorum\_read\_node0;quorum\_read\_node2;recent Cache logs do not show namespace key\-mismatch failures;recent Origin logs do not show namespace key\-mismatch failures;recently\_rebooted;repair\_completed;retrieved checksum does not match expected value;retrieved object checksum matches expected hidden value;retrieved object content differs from expected dataset;retrieved object content matches expected dataset;ssh\_reachable\.

Durability\(6 distinct check names\):concurrent workload broke after compute reboot \(drop\-in came back\);concurrent workload broke after puppet apply \(manifest still enforces bad config\);concurrent workload still passes after a forced puppet apply;concurrent workload still passes after compute reboots;dm\-delay target reappeared after reboot \- assemble script was not fixed;no dm\-delay target reappeared after reboot\.

Cleanup\(18 distinct check names\):/etc/slurm still has \*\.dpkg\-dist files on node0;/etc/slurm still has \*\.dpkg\-dist files on node1;/etc/slurm still has \*\.dpkg\-dist files on node2;MaxJobCount is sane on node0 \(10000\);MaxJobCount is sane on node0 \(unset\);MaxJobCount is sane on node1 \(10000\);MaxJobCount is sane on node1 \(unset\);MaxJobCount is sane on node2 \(10000\);MaxJobCount is sane on node2 \(unset\);MaxJobCount is still 2 on node0;MaxJobCount is still 2 on node1;MaxJobCount is still 2 on node2;federation incident log has no remaining key\-mismatch marker;slurmd maintenance cgroup drop\-in is gone or harmless on node1;slurmd maintenance cgroup drop\-in is gone or harmless on node2;slurmd maintenance cgroup drop\-in still starves the daemon on node1;slurmd maintenance cgroup drop\-in still starves the daemon on node2;stale namespace key\-mismatch incident marker is still present\.

## Appendix FRisk Evidence and Failure\-Mode Examples

This appendix summarizes Risk Monitor review findings that support the claims in §[4\.5](https://arxiv.org/html/2608.11234#S4.SS5)and §[4\.6](https://arxiv.org/html/2608.11234#S4.SS6), without reproducing full command transcripts\.

*Evaluator\-harness probing \(§[4\.6](https://arxiv.org/html/2608.11234#S4.SS6)\)\.*17 flagged actions across 7 of the 9 audited configurations are agents inspecting the grading environment—listing/logs/verifier, enumerating tests inside the verifier’s virtualenv, or searching the filesystem for scripts named after grading\. The most direct case is a Grok 4\.5 trial on Cassandra Hung Recovery that ran the task’s owntests/test\.shand then readreward\.txtandreward\.json\.

*Access\-control bypass on Ceph Bootstrap\.*15 flagged actions across five configurations \(Claude Opus 4\.8, Grok 4\.5, Gemini 3\.6 Flash, Composer 2\.5, MiMo V2\.5\) unloaded or tore down mandatory access control on every node to get past a cephadm host\-facts parsing bug; one Opus 4\.8 trial additionally bind\-mounted an empty file over the kernel’s AppArmor profiles interface, locally and then on every peer node\. None of these steps appears in the task’s reference solution; all were agent\-initiated shortcuts\.

*Destroying the state under repair\.*A MiMo V2\.5 trial on Cassandra Hung Recovery wiped the node’s system keyspace, commitlog, and caches nine times in one attempt while trying to force the node back into the ring, and stopped Cassandra on both surviving nodes\. On DB WAL Recovery, three DeepSeek trials deleted the write\-ahead log the task asks them to recover before attempting recovery\.

*Incident re\-injection on Pelican Key Mismatch\.*A Composer 2\.5 trial and a Grok 4\.5 trial each rewrote the live registry database back to the pre\-incident issuer key they had been asked to replace, reversing their own partial progress rather than completing it; two Gemini 3\.6 Flash trials edited the registry’s trust material directly by SQL rather than through the service\.

*Tool\-destructive diagnosis on DB WAL Recovery \(§[4\.5](https://arxiv.org/html/2608.11234#S4.SS5)\)\.*The recurring pattern across nearly all configurations is opening the target database directly before snapshotting the write\-ahead log, so the client’s own auto\-checkpoint behavior discards the uncommitted pages the task asks the agent to recover—the diagnosis step destroys the evidence needed for the fix\.