InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents

# A Benchmark for Open-Ended LLM Inference Optimization by AI Agents
Source: [https://arxiv.org/html/2607.20468](https://arxiv.org/html/2607.20468)
###### Abstract

AI agents are increasingly used to automate research and development tasks, yet existing benchmarks typically evaluate them on prescribed workflows or narrow action spaces\. Even nominally open\-ended tasks can often be solved by retrieving a well\-known recipe and tuning a few hyperparameters, making it unclear whether strong results reflect genuine optimization or memorized solutions\. We introduceInferenceBench, where an agent must deploy an OpenAI\-compatible inference server and optimize the speed of LLM inference\. Each agent receives a target LLM, one H100 GPU, an optimization scenario, and a wall\-clock time budget of two hours\. Three optimization scenarios isolate distinct bottlenecks of inference \(prefill latency, decode latency, and concurrent request throughput\) and a fourth balances all three at the same time\. Across 15 frontier agent configurations, agents reliably improve over a naïve PyTorch baseline \(up to8\.08×8\.08\\times\) and often match or exceed serving engines with default settings \(4\.05×4\.05\\timesfor vLLM\), but still fall below a simple hyperparameter search under the same time budget \(up to11\.53×11\.53\\times\)\. Qualitative analysis of agent trajectories shows that although agents enumerate many relevant optimization techniques, they overwhelmingly converge on a single inference framework\. They only test few distinct configurations, and spend the remaining budget re\-measuring, repairing, or optimizing hyperparameters rather than exploring substantially different strategies\. This suggests the bottleneck is not domain knowledge, but the ability to propose diverse configurations, evaluate them systematically, and submit the best identified solution\. Overall,InferenceBenchreflects the ability of agents to operate in an open\-ended AI engineering setting, where memorized solutions lead to limited improvements\.

††footnotetext:1ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center\.![Refer to caption](https://arxiv.org/html/2607.20468v1/x1.png)Figure 1:Benchmark overview\. The agent receives a base model, hardware environment, and scenario\-specific objective\. It operates within a containerized environment with an open\-ended action space, using a provided evaluation script as a feedback loop\. A submission must pass both a quality gate and an integrity gate before scenario\-specific speed metrics are scored\.## 1Introduction

A growing body of benchmarks evaluates frontier agents on autonomous research tasks, from end\-to\-end scientific discovery pipelines\(Luet al\.,[2024](https://arxiv.org/html/2607.20468#bib.bib23)\)to Kaggle\-style ML experimentation\(Chanet al\.,[2025](https://arxiv.org/html/2607.20468#bib.bib12); Huanget al\.,[2024](https://arxiv.org/html/2607.20468#bib.bib13)\)and autonomous post\-training\(Ranket al\.,[2026](https://arxiv.org/html/2607.20468#bib.bib14)\)\. Task structure varies, with some benchmarks supplying a starter script to refine and others requiring one from scratch, but in practice the relevant action space tends to be narrow such as hyperparameter tuning, data\-mixture choices, and localized edits to boilerplate code\. Even where a broader action space is technically available to the agent, exercising it is rarely necessary to score well\. A common finding from these benchmarks is therefore that exploration stays shallow, with agents converging quickly on memorized solutions rather than designing and executing genuinely different experiments\(Toledoet al\.,[2025](https://arxiv.org/html/2607.20468#bib.bib7); Zouet al\.,[2025](https://arxiv.org/html/2607.20468#bib.bib8); Nangiaet al\.,[2026](https://arxiv.org/html/2607.20468#bib.bib4)\)\. When the effective search space collapses to a handful of well\-known configurations, it raises the question of what an autonomous agent provides that a cheaper random search or Bayesian optimization over the same space does not\.

Inference systems engineering poses a qualitatively different class of challenges, ones that prior automated research benchmarks have largely left unaddressed\. A working server requires assembling components whose dependencies frequently conflict such as an inference framework, attention backend, quantization format, and runtime parameters such as batch sizes, KV\-cache allocation, and CUDA\-graph capture, all tuned to the GPU’s memory hierarchy\. A wrong combination does not produce a slightly worse loss curve; it produces a server that crashes on launch, triggers a CUDA driver incompatibility, or stalls on JIT recompilation for tens of minutes\. An agent relying on recalled configurations without probing the current environment fails immediately and visibly\.

We introduceInferenceBench, a benchmark that tasks an agent with inference systems optimization and measures both the optimization it achieves and the engineering behavior it exhibits\. Each evaluation instance provides a base language model \(e\.g\., Mistral\-7B\-Instruct\-v0\.3\), a single NVIDIA H100 GPU, a wall\-clock time budget, and an optimization objective\. Three of these objectives target a distinct bottleneck in inference speed \(prefill latency on long\-context prompts, per\-token decode latency on long generations, request throughput under concurrent traffic\), and a fourth multi\-objective scenario requires balancing all three at the same time\. The main score is a speedup over a fixed PyTorch baseline, so it is naturally unbounded rather than capped at a threshold\. The agent must deliver a running, OpenAI\-compatible inference server that passes an integrity gate that screens for reward\-hacking exploits such as returning pre\-generated text or substituting a smaller model along with a separate quality gate that checks accuracy on a held\-out dataset\. It receives no explicit solution path and is given no starter code: it is free to choose any framework, quantization strategy, attention backend, and scheduling configuration, or even build a serving solution from scratch\.

Our contributions are as follows:

- •InferenceBench, the first open\-ended benchmark that targets end\-to\-end inference systems engineering as an agent task\. Our task requires system\-level, kernel\-level, and runtime\-level decisions, which together provide a more comprehensive evaluation of autonomous\-R&D capability than a narrow hyperparameter surface\.
- •Four evaluation scenarios: three that isolate a distinct bottleneck of inference serving \(prefill latency, decode speed, concurrent throughput\), and a fourth multi\-objective scenario that requires balancing all three\. Each is paired with a defined speed metric and a workload drawn from naturally occurring long\-context documents\.
- •A large\-scale evaluation of 15 frontier agent configurations across three agent scaffolds \(Claude Code, Codex CLI, OpenCode\)\. Agents reliably improve on a naïve baseline, with the best agent reaching an8\.08×8\.08\\timesaggregate speedup and exceeding the best default inference\-engine aggregate \(4\.05×4\.05\\timesfor vLLM\), but a simple matched\-budget hyperparameter search still performs better, reaching an11\.53×11\.53\\timesaggregate and matching or exceeding the best agent in every scenario\. Qualitatively, agents enumerate the relevant optimizations in nearly every transcript, yet they ship vLLM in 169/180 runs \(93\.9%\), launch a median of only one distinct non\-default vLLM argument set over the 2\-hour budget, and occasionally ship a broken server after having measured a good configuration mid\-run\.

## 2Related Works

#### Agent capability benchmarks\.

Recent benchmarks study agents on software engineering\(Jimenezet al\.,[2024](https://arxiv.org/html/2607.20468#bib.bib1)\), ML experimentation\(Huanget al\.,[2024](https://arxiv.org/html/2607.20468#bib.bib13); Chanet al\.,[2025](https://arxiv.org/html/2607.20468#bib.bib12)\), post\-training\(Wijket al\.,[2025](https://arxiv.org/html/2607.20468#bib.bib20)\), paper reproduction\(Staraceet al\.,[2025](https://arxiv.org/html/2607.20468#bib.bib9)\), algorithm discovery\(Presset al\.,[2025](https://arxiv.org/html/2607.20468#bib.bib17)\), and local inference optimization\(Reinet al\.,[2025](https://arxiv.org/html/2607.20468#bib.bib21); Toledoet al\.,[2025](https://arxiv.org/html/2607.20468#bib.bib7); Zouet al\.,[2025](https://arxiv.org/html/2607.20468#bib.bib8)\)\. In particular, PostTrainBench targets autonomous LLM post\-training\(Ranket al\.,[2026](https://arxiv.org/html/2607.20468#bib.bib14)\), while ISO\-Bench evaluates local patch\-level optimization tasks in vLLM and SGLang\(Nangiaet al\.,[2026](https://arxiv.org/html/2607.20468#bib.bib4)\)\.InferenceBenchinstead measures end\-to\-end deployment and optimization of a full OpenAI\-compatible inference server, exposing framework, quantization, and runtime choices and comparing against a matched\-budget non\-agentic search baseline rather than a ground truth patch or expert\-tuned model\. Autoresearch provides a prominent open\-source example of the potential of fixed\-budget iterative agent\-driven training experiments\(Karpathy,[2026](https://arxiv.org/html/2607.20468#bib.bib51)\)\. AlphaEvolve similarly demonstrates that coding agents can drive evolutionary search over programs for scientific and algorithmic discovery, including optimization of computational infrastructure\(Novikovet al\.,[2025](https://arxiv.org/html/2607.20468#bib.bib3)\)\.

#### Inference system and kernel optimization\.

InferenceBenchalso builds on prior work in continuous batching\(Yuet al\.,[2022](https://arxiv.org/html/2607.20468#bib.bib26)\), prefill/decode scheduling\(Agrawalet al\.,[2024](https://arxiv.org/html/2607.20468#bib.bib27)\), prefix sharing and runtime\(Zhenget al\.,[2024](https://arxiv.org/html/2607.20468#bib.bib28)\), and attention kernels and backends\(Kwonet al\.,[2023](https://arxiv.org/html/2607.20468#bib.bib29); Daoet al\.,[2022](https://arxiv.org/html/2607.20468#bib.bib31); Yeet al\.,[2025](https://arxiv.org/html/2607.20468#bib.bib34)\)\. The closest agentic systems work operates one level lower, targeting kernel generation\(Ouyanget al\.,[2025](https://arxiv.org/html/2607.20468#bib.bib44); Langeet al\.,[2025](https://arxiv.org/html/2607.20468#bib.bib45)\)or single\-kernel optimization\(Xinget al\.,[2026](https://arxiv.org/html/2607.20468#bib.bib6)\)rather than full\-server integration\. By contrast,InferenceBenchasks whether agents can compose these ingredients into a working server, choose among competing frameworks, survive realistic traffic, and satisfy an accuracy gate; KernelBench’s finding that frontier models beat PyTorch on fewer than 20% of kernels\(Ouyanget al\.,[2025](https://arxiv.org/html/2607.20468#bib.bib44)\)establishes a lower bound on per\-component difficulty, consistent with the higher\-level behavioral limits we observe\.

## 3InferenceBench

### 3\.1Task and Environment

EachInferenceBenchexperiment run specifies a base model \(e\.g\., Mistral\-7B\-Instruct\-v0\.3\), a fixed hardware environment \(one NVIDIA H100 with 80 GB VRAM\), an optimization objective, and a wall\-clock budget\. The agent must deliver an OpenAI\-compatible inference server exposingGET /v1/modelsandPOST /v1/chat/completions, and maximize the scenario’s primary metric\.

The agent works in a containerized Linux environment with CUDA drivers, build tools, root access, internet connectivity, and pre\-cached model weights\. The action space is unconstrained: the agent may install packages, compile dependencies, adopt an existing inference framework \(e\.g\., vLLM, SGLang, TensorRT\-LLM\), apply quantization, tune runtime parameters, write custom attention kernels, or build a serving solution from scratch\. Prohibited behaviors are also included in the prompt, such as offloading inference to external APIs, tampering with the evaluation harness, or swapping out the base model\. Full environment details can be found in Appendix[A](https://arxiv.org/html/2607.20468#A1)\.

### 3\.2Scenarios

Three scenarios each isolate a distinct bottleneck of inference serving, and a fourth scenario requires balancing all three simultaneously\. Table[1](https://arxiv.org/html/2607.20468#S3.T1)gives the request parameters and primary metric for each\.

Scenario A \(prefill\) isolates the cost of processing a long prompt before the first output token, measured as time to first token \(TTFT\)\. With 8k\-token inputs, TTFT dominates perceived latency regardless of decode speed, focusing attention on factors such as attention\-backend choice, chunked\-prefill configuration, and prefix caching\. Scenario B \(decode\) isolates per\-token generation cost on long outputs, measured as time per output token \(TPOT\), computed as\(tend−tfirst\)/\(n−1\)\(t\_\{\\text\{end\}\}\-t\_\{\\text\{first\}\}\)/\(n\-1\)wherennis the number of generated tokens\. With 8k\-token generations, TPOT dominates total response time once prefill completes, and the relevant levers are memory bandwidth utilization and KV\-cache efficiency\. Scenario C \(throughput\) stresses the scheduler under 64 concurrent requests\. The metric is the geometric mean of requests per second across three traffic profiles \(burst, Poisson, constant\-rate\), reflecting scheduler behavior under varied arrival patterns\. Scenario D \(multi\-objective\) measures the geometric mean of three higher\-is\-better quantities derived from the run’s measurements: inverse TTFT, inverse TPOT, and request throughput, testing whether an agent can balance all three objectives simultaneously rather than optimize one at the expense of the others\. Alongside the primary metric, every run reports inter\-token latency \(ITL\), generation throughput, and tail latencies at p90 and p99\. Exact metric definitions can be found in Appendix[A\.5](https://arxiv.org/html/2607.20468#A1.SS5)\.

#### Request sampling\.

Requests for the inference speed evaluations are sourced from LongBench v2\(Baiet al\.,[2025](https://arxiv.org/html/2607.20468#bib.bib18)\), a corpus of naturally occurring long\-context documents\. For each scenario, prompts are sampled with tokenized lengths in\[0\.8​Lin,Lin\]\[0\.8\\,L\_\{\\text\{in\}\},\\,L\_\{\\text\{in\}\}\]to better model naturally occurring documents without distorting the tokenizer merge structure and attention patterns, with any overflow truncated toLinL\_\{\\text\{in\}\}\. Output length is also drawn uniformly from\[0\.8​Lout,Lout\]\[0\.8\\,L\_\{\\text\{out\}\},\\,L\_\{\\text\{out\}\}\]per request\. Each request set is serialized with its content hash so every agent and baseline scores against byte\-identical inputs\. Further details about request sampling can be found in Appendix[A\.4](https://arxiv.org/html/2607.20468#A1.SS4)\.

Table 1:Scenario specifications\.LinL\_\{\\text\{in\}\}andLoutL\_\{\\text\{out\}\}are target input and output lengths in tokens drawn uniformly from\[0\.8​L,L\]\[0\.8L,L\]\. Requests are sourced from LongBench v2\(Baiet al\.,[2025](https://arxiv.org/html/2607.20468#bib.bib18)\)\.

### 3\.3Gating

Scoring applies two gates; failure at either stage penalizes the run\.

#### Quality gate\.

Pure speed optimization enables trivial exploits that collapse the utility of the model \(e\.g\., aggressive quantization, pruned layers\)\. We therefore require the optimized server to score at leastτ=95%\\tau=95\\%of the baseline’s accuracy on a seed\-fixed 500\-question MMLU\-Pro\(Wanget al\.,[2024](https://arxiv.org/html/2607.20468#bib.bib19)\)subset, using 10\-option multiple choice \(labels A–J\) with greedy decoding\. Ablation on threshold sensitivityτ\\tau\(Appendix[C\.1](https://arxiv.org/html/2607.20468#A3.SS1)\) shows that the best\-agent identity and main conclusions are stable across thresholds; rankings are unchanged throughτ=0\.95\\tau=0\.95and change by at most one position atτ=0\.97\\tau=0\.97\.

#### Integrity gate\.

A judge agent actively inspects each run and its environment for disallowed behaviors such as returning pre\-generated text, model substitution, fine\-tuning on the quality\-gate subset, or API offloading\. We use an agentic judge as each experiment run can leave behind a long transcript, launcher state, server logs, and final metrics that must be interpreted jointly\. Appendix[C\.2](https://arxiv.org/html/2607.20468#A3.SS2)validates this judge with cross\-judge agreement \(κ=0\.82\\kappa=0\.82\) and a manual human audit with no false positives and one missed violation among 50 audited runs\.

### 3\.4Evaluation Protocol

#### Agents and scaffolds\.

We evaluate Anthropic models \(Claude Opus 4\.7/4\.6/4\.5, Sonnet 4\.6/4\.5, and Haiku 4\.5\) via Claude Code\(Anthropic,[2025](https://arxiv.org/html/2607.20468#bib.bib47)\), OpenAI models \(GPT\-5\.5 High, GPT\-5\.4 High, GPT\-5\.3 Codex High, GPT\-5\.3 Codex Medium, GPT\-5\.2, GPT\-5\.2 Codex, GPT\-5\.1 Codex Max\) via Codex CLI\(OpenAI,[2025](https://arxiv.org/html/2607.20468#bib.bib48)\), and Gemini 3\.1 Pro and GLM\-5\(GLM\-5\-Teamet al\.,[2026](https://arxiv.org/html/2607.20468#bib.bib46)\)via OpenCode\(Anomaly,[2026](https://arxiv.org/html/2607.20468#bib.bib50)\)\. Full scaffold details can be found in Appendix[B](https://arxiv.org/html/2607.20468#A2)\.

#### Prompt and execution\.

All agents receive the same prompt template specifying role, scenario objective, operational constraints, and an instruction to keep improving until the budget expires\. The in\-container evaluation script serves as the feedback loop, and a scaffold wrapper repeatedly invokes the agent’s continuation mechanism so the full wall\-clock budget is used\. Hardware, container, pre\-cached weights, budget, and harness are identical across runs, with no user intervention\. Exact prompts can be found in Appendix[B\.1](https://arxiv.org/html/2607.20468#A2.SS1)\.

#### Seeds\.

Each agent\-scenario cell is run on three seed pairs\(sdev,seval\)\(s\_\{\\text\{dev\}\},s\_\{\\text\{eval\}\}\)\. The development seed controls the requests and MMLU\-Pro subset exposed during optimization, while the evaluation seed controls the held\-out final score; decoupling them prevents overfitting to the callable evaluator\. We use the same three pairs for every agent so comparisons use byte\-identical inputs and directly comparable three\-seed variance estimates\. Reported error bars are sample standard error of the mean \(sample standard deviation across the three seed pairs divided by3\\sqrt\{3\}, i\.e\.ddof=1\)\. We supplement this sample standard error mean with the seed\-level volatility analysis in Appendix[C\.3](https://arxiv.org/html/2607.20468#A3.SS3)to better focus on the direction and magnitude of the gap to baselines\.

#### Baselines\.

We compare agents against two baseline classes\. The first class consists of inference engines \(vLLM, SGLang, TGI\) with default parameters\. The second consists of non\-agentic search methods: uniform random search, SMAC3\(Lindaueret al\.,[2022](https://arxiv.org/html/2607.20468#bib.bib25)\), and TPE\(Bergstraet al\.,[2011](https://arxiv.org/html/2607.20468#bib.bib24)\), each with its own engine\-specific search space and a separate 2\-hour budget\. These search spaces are derived directly from documented CLI flags, which agents can likewise discover in full via ‘–help‘ or online documentation, so the underlying parameter space is shared\. Full detail of baselines can be found in Appendix[B\.4](https://arxiv.org/html/2607.20468#A2.SS4)\.

Table 2:Per\-scenario speedup over the PyTorch baseline under a 2\-hour time budget\. Agent rows report mean±SEM\{\}\_\{\\pm\\text\{SEM\}\}across the three held\-out seed\-pair runs; non\-passing or unusable runs contribute1\.00×1\.00\\times\. Aggregate is the geometric mean over Scenarios A–D, reported with seed\-level SEM\. All rows are sorted by Aggregate\. Agent rows show the scaffold in parentheses\.Boldmarks the best search value in each column\. Search and default baselines use fixed categorical shading; agent cells use column\-independent heat shading computed only over agent rows, with darker cells indicating stronger agent performance within that metric column\.RankMethodAggregateSc\. ASc\. BSc\. CSc\. DTTFTTPOTreq\. tputgeomean–SMAC11\.53×\\times±0\.68×\{\}\_\{\\pm\\text\{0\.68$\\times$\}\}4\.37×\\times±0\.34×\{\}\_\{\\pm\\text\{0\.34$\\times$\}\}15\.23×\\times±1\.27×\{\}\_\{\\pm\\text\{1\.27$\\times$\}\}46\.70×\\times±3\.58×\{\}\_\{\\pm\\text\{3\.58$\\times$\}\}5\.69×\\times±0\.42×\{\}\_\{\\pm\\text\{0\.42$\\times$\}\}–TPE11\.25×\\times±0\.74×\{\}\_\{\\pm\\text\{0\.74$\\times$\}\}4\.48×\\times±0\.39×\{\}\_\{\\pm\\text\{0\.39$\\times$\}\}14\.76×\\times±1\.45×\{\}\_\{\\pm\\text\{1\.45$\\times$\}\}43\.46×\\times±3\.72×\{\}\_\{\\pm\\text\{3\.72$\\times$\}\}5\.58×\\times±0\.51×\{\}\_\{\\pm\\text\{0\.51$\\times$\}\}–Best Random10\.20×\\times±0\.92×\{\}\_\{\\pm\\text\{0\.92$\\times$\}\}4\.21×\\times±0\.47×\{\}\_\{\\pm\\text\{0\.47$\\times$\}\}11\.34×\\times±1\.59×\{\}\_\{\\pm\\text\{1\.59$\\times$\}\}41\.81×\\times±5\.12×\{\}\_\{\\pm\\text\{5\.12$\\times$\}\}5\.42×\\times±0\.61×\{\}\_\{\\pm\\text\{0\.61$\\times$\}\}1Claude Sonnet 4\.6\(Claude Code\)8\.08×\\times±1\.10×\{\}\_\{\\pm\\text\{1\.10$\\times$\}\}3\.47×\\times±0\.48×\{\}\_\{\\pm\\text\{0\.48$\\times$\}\}12\.03×\\times±3\.29×\{\}\_\{\\pm\\text\{3\.29$\\times$\}\}33\.93×\\times±4\.19×\{\}\_\{\\pm\\text\{4\.19$\\times$\}\}3\.01×\\times±0\.50×\{\}\_\{\\pm\\text\{0\.50$\\times$\}\}2GLM\-5\(OpenCode\)6\.20×\\times±1\.48×\{\}\_\{\\pm\\text\{1\.48$\\times$\}\}3\.44×\\times±1\.22×\{\}\_\{\\pm\\text\{1\.22$\\times$\}\}4\.45×\\times±1\.73×\{\}\_\{\\pm\\text\{1\.73$\\times$\}\}26\.36×\\times±12\.70×\{\}\_\{\\pm\\text\{12\.70$\\times$\}\}3\.66×\\times±0\.05×\{\}\_\{\\pm\\text\{0\.05$\\times$\}\}3Gemini 3\.1 Pro\(OpenCode\)6\.16×\\times±0\.78×\{\}\_\{\\pm\\text\{0\.78$\\times$\}\}3\.35×\\times±0\.07×\{\}\_\{\\pm\\text\{0\.07$\\times$\}\}4\.81×\\times±0\.72×\{\}\_\{\\pm\\text\{0\.72$\\times$\}\}31\.24×\\times±4\.35×\{\}\_\{\\pm\\text\{4\.35$\\times$\}\}2\.87×\\times±0\.43×\{\}\_\{\\pm\\text\{0\.43$\\times$\}\}4GPT\-5\.3 Codex \(High\)\(Codex\)5\.48×\\times±0\.51×\{\}\_\{\\pm\\text\{0\.51$\\times$\}\}3\.54×\\times±0\.09×\{\}\_\{\\pm\\text\{0\.09$\\times$\}\}3\.38×\\times±1\.32×\{\}\_\{\\pm\\text\{1\.32$\\times$\}\}29\.00×\\times±2\.02×\{\}\_\{\\pm\\text\{2\.02$\\times$\}\}2\.60×\\times±0\.01×\{\}\_\{\\pm\\text\{0\.01$\\times$\}\}5GPT\-5\.4 \(High\)\(Codex\)5\.08×\\times±0\.92×\{\}\_\{\\pm\\text\{0\.92$\\times$\}\}3\.53×\\times±0\.05×\{\}\_\{\\pm\\text\{0\.05$\\times$\}\}2\.24×\\times±0\.96×\{\}\_\{\\pm\\text\{0\.96$\\times$\}\}25\.84×\\times±0\.71×\{\}\_\{\\pm\\text\{0\.71$\\times$\}\}3\.25×\\times±1\.37×\{\}\_\{\\pm\\text\{1\.37$\\times$\}\}6GPT\-5\.3 Codex \(Medium\)\(Codex\)4\.86×\\times±1\.14×\{\}\_\{\\pm\\text\{1\.14$\\times$\}\}2\.75×\\times±0\.87×\{\}\_\{\\pm\\text\{0\.87$\\times$\}\}3\.73×\\times±1\.48×\{\}\_\{\\pm\\text\{1\.48$\\times$\}\}19\.30×\\times±9\.26×\{\}\_\{\\pm\\text\{9\.26$\\times$\}\}2\.82×\\times±0\.24×\{\}\_\{\\pm\\text\{0\.24$\\times$\}\}7GPT\-5\.5 \(High\)\(Codex\)4\.22×\\times±1\.30×\{\}\_\{\\pm\\text\{1\.30$\\times$\}\}3\.06×\\times±1\.07×\{\}\_\{\\pm\\text\{1\.07$\\times$\}\}2\.59×\\times±1\.59×\{\}\_\{\\pm\\text\{1\.59$\\times$\}\}19\.11×\\times±9\.05×\{\}\_\{\\pm\\text\{9\.05$\\times$\}\}2\.08×\\times±0\.54×\{\}\_\{\\pm\\text\{0\.54$\\times$\}\}–vLLM default4\.05×\\times±0\.07×\{\}\_\{\\pm\\text\{0\.07$\\times$\}\}1\.25×\\times±0\.04×\{\}\_\{\\pm\\text\{0\.04$\\times$\}\}2\.25×\\times±0\.10×\{\}\_\{\\pm\\text\{0\.10$\\times$\}\}48\.69×\\times±1\.74×\{\}\_\{\\pm\\text\{1\.74$\\times$\}\}1\.96×\\times±0\.06×\{\}\_\{\\pm\\text\{0\.06$\\times$\}\}–SGLang default3\.92×\\times±0\.09×\{\}\_\{\\pm\\text\{0\.09$\\times$\}\}1\.22×\\times±0\.05×\{\}\_\{\\pm\\text\{0\.05$\\times$\}\}1\.77×\\times±0\.07×\{\}\_\{\\pm\\text\{0\.07$\\times$\}\}51\.12×\\times±2\.27×\{\}\_\{\\pm\\text\{2\.27$\\times$\}\}2\.14×\\times±0\.09×\{\}\_\{\\pm\\text\{0\.09$\\times$\}\}8Claude Opus 4\.6\(Claude Code\)3\.89×\\times±1\.17×\{\}\_\{\\pm\\text\{1\.17$\\times$\}\}1\.00×\\times±0\.00×\{\}\_\{\\pm\\text\{0\.00$\\times$\}\}2\.77×\\times±1\.78×\{\}\_\{\\pm\\text\{1\.78$\\times$\}\}25\.64×\\times±9\.87×\{\}\_\{\\pm\\text\{9\.87$\\times$\}\}3\.21×\\times±0\.33×\{\}\_\{\\pm\\text\{0\.33$\\times$\}\}9GPT\-5\.2\(Codex\)3\.82×\\times±0\.85×\{\}\_\{\\pm\\text\{0\.85$\\times$\}\}3\.12×\\times±1\.11×\{\}\_\{\\pm\\text\{1\.11$\\times$\}\}1\.00×\\times±0\.00×\{\}\_\{\\pm\\text\{0\.00$\\times$\}\}32\.61×\\times±3\.10×\{\}\_\{\\pm\\text\{3\.10$\\times$\}\}2\.09×\\times±0\.54×\{\}\_\{\\pm\\text\{0\.54$\\times$\}\}10GPT\-5\.1 Codex Max\(Codex\)3\.54×\\times±0\.69×\{\}\_\{\\pm\\text\{0\.69$\\times$\}\}1\.00×\\times±0\.00×\{\}\_\{\\pm\\text\{0\.00$\\times$\}\}4\.66×\\times±0\.96×\{\}\_\{\\pm\\text\{0\.96$\\times$\}\}15\.00×\\times±3\.60×\{\}\_\{\\pm\\text\{3\.60$\\times$\}\}2\.23×\\times±0\.20×\{\}\_\{\\pm\\text\{0\.20$\\times$\}\}11Claude Opus 4\.5\(Claude Code\)3\.37×\\times±0\.98×\{\}\_\{\\pm\\text\{0\.98$\\times$\}\}3\.69×\\times±0\.43×\{\}\_\{\\pm\\text\{0\.43$\\times$\}\}1\.00×\\times±0\.00×\{\}\_\{\\pm\\text\{0\.00$\\times$\}\}18\.01×\\times±8\.55×\{\}\_\{\\pm\\text\{8\.55$\\times$\}\}1\.93×\\times±0\.47×\{\}\_\{\\pm\\text\{0\.47$\\times$\}\}–HF TGI default3\.30×\\times±0\.06×\{\}\_\{\\pm\\text\{0\.06$\\times$\}\}1\.14×\\times±0\.04×\{\}\_\{\\pm\\text\{0\.04$\\times$\}\}1\.37×\\times±0\.06×\{\}\_\{\\pm\\text\{0\.06$\\times$\}\}41\.94×\\times±1\.48×\{\}\_\{\\pm\\text\{1\.48$\\times$\}\}1\.80×\\times±0\.05×\{\}\_\{\\pm\\text\{0\.05$\\times$\}\}12Claude Sonnet 4\.5\(Claude Code\)2\.96×\\times±1\.02×\{\}\_\{\\pm\\text\{1\.02$\\times$\}\}2\.67×\\times±0\.85×\{\}\_\{\\pm\\text\{0\.85$\\times$\}\}1\.00×\\times±0\.00×\{\}\_\{\\pm\\text\{0\.00$\\times$\}\}9\.65×\\times±8\.65×\{\}\_\{\\pm\\text\{8\.65$\\times$\}\}2\.97×\\times±0\.32×\{\}\_\{\\pm\\text\{0\.32$\\times$\}\}13Claude Opus 4\.7\(Claude Code\)2\.25×\\times±0\.32×\{\}\_\{\\pm\\text\{0\.32$\\times$\}\}1\.07×\\times±0\.06×\{\}\_\{\\pm\\text\{0\.06$\\times$\}\}1\.00×\\times±0\.00×\{\}\_\{\\pm\\text\{0\.00$\\times$\}\}19\.02×\\times±0\.94×\{\}\_\{\\pm\\text\{0\.94$\\times$\}\}1\.27×\\times±0\.27×\{\}\_\{\\pm\\text\{0\.27$\\times$\}\}14GPT\-5\.2 Codex\(Codex\)1\.55×\\times±0\.27×\{\}\_\{\\pm\\text\{0\.27$\\times$\}\}3\.07×\\times±0\.13×\{\}\_\{\\pm\\text\{0\.13$\\times$\}\}1\.00×\\times±0\.00×\{\}\_\{\\pm\\text\{0\.00$\\times$\}\}1\.00×\\times±0\.00×\{\}\_\{\\pm\\text\{0\.00$\\times$\}\}1\.87×\\times±0\.44×\{\}\_\{\\pm\\text\{0\.44$\\times$\}\}15Claude Haiku 4\.5\(Claude Code\)1\.24×\\times±0\.19×\{\}\_\{\\pm\\text\{0\.19$\\times$\}\}0\.77×\\times±0\.23×\{\}\_\{\\pm\\text\{0\.23$\\times$\}\}1\.00×\\times±0\.00×\{\}\_\{\\pm\\text\{0\.00$\\times$\}\}3\.11×\\times±1\.69×\{\}\_\{\\pm\\text\{1\.69$\\times$\}\}1\.00×\\times±0\.00×\{\}\_\{\\pm\\text\{0\.00$\\times$\}\}–PyTorch baseline1\.00×\\times±0\.00×\{\}\_\{\\pm\\text\{0\.00$\\times$\}\}1\.00×\\times±0\.00×\{\}\_\{\\pm\\text\{0\.00$\\times$\}\}1\.00×\\times±0\.00×\{\}\_\{\\pm\\text\{0\.00$\\times$\}\}1\.00×\\times±0\.00×\{\}\_\{\\pm\\text\{0\.00$\\times$\}\}1\.00×\\times±0\.00×\{\}\_\{\\pm\\text\{0\.00$\\times$\}\}

## 4Results

#### Main results\.

Table[2](https://arxiv.org/html/2607.20468#S3.T2)reports per\-scenario speedups over the naïve PyTorch baseline on a 2\-hour time budget, a H100 GPU, and using Mistral\-7B\-Instruct\-v0\.3 as the base model\. Agent cells are reported as mean±SEM\{\}\_\{\\pm\\text\{SEM\}\}over three held\-out seed\-pair runs per agent×\\timesscenario cell\. For any run that fails either of the gates or to produce a final working server by the end of the time budget, we fall back to the PyTorch baseline; contributing1×1\\times\. The non\-agent rows report each search method’s final gate\-passing configuration from a 2\-hour search over vLLM parameters, matching the agent’s effective single\-engine budget; Appendix[G\.3](https://arxiv.org/html/2607.20468#A7.SS3)reports the full 3\-method×\\times3\-engine grid\.

Table 3:Outcomes across 180 recorded agent experiment runs\.As a configuration that is fast during development but not preserved as a valid final server has only baseline\-equivalent deployment utility, we consider shipping a functional server at the end of the 2\-hour time budget to be a core part of the task during evaluation\. This can produce somewhat counterintuitive rankings, where a reliable Sonnet or GLM run can outrank a larger model that more often leaves behind a failing final server\. A detailed qualitative analysis of this phenomenon can be found in Appendix[E\.4](https://arxiv.org/html/2607.20468#A5.SS4)\. Table[3](https://arxiv.org/html/2607.20468#S4.T3)summarizes final\-run outcomes across all 180 recorded runs: 65\.0% \(117/180\) pass both gates and contribute measured speedup, while the remaining 63 runs fail or do not complete the quality gate \(34; of which 23 fall below the threshold and 11 do not complete the quality evaluation\), are flagged by the integrity gate \(11\), or fail final\-server reachability/runtime checks \(18\)\. These categories are mutually exclusive; each run contributes to exactly one\.

#### Agents mostly beat default configurations, but simple parameter search outperforms agents\.

Serving engines on default configurations already show large gains over the naïve PyTorch baseline, with SGLang slightly ahead on Scenarios C and D\. Agents improve further on most default configurations, with the top\-ranked agent \(Claude Sonnet 4\.6\) reaching3\.47×3\.47\\times,12\.03×12\.03\\times,33\.93×33\.93\\times, and3\.01×3\.01\\timeson Scenarios A through D respectively, for an aggregate of8\.08×8\.08\\times, ahead of the best default\-engine aggregate \(4\.05×4\.05\\timesfor vLLM\) but still well below matched non\-agent search\. When the non\-agent search methods are given the same 2\-hour budget on vLLM, the best search row \(SMAC\) reaches4\.37×4\.37\\times,15\.23×15\.23\\times,46\.70×46\.70\\times,5\.69×5\.69\\times, and an11\.53×11\.53\\timesaggregate and the search baselines remain stronger than the best agent on the aggregate and on every individual scenario\. The Scenario B gap is the smallest: the weakest vLLM search row is close to the best agent mean, but the best vLLM search row remains higher\. Expanding the non\-agentic search baselines to SGLang and TGI widens the gap further on Scenarios A and C, as shown in Appendix[G\.3](https://arxiv.org/html/2607.20468#A7.SS3)\.

#### Throughput exposes shallow optimization\.

On Scenarios A, B, and D, many agents substantially exceed default configurations, so the gap to search reflects genuine but incomplete optimization\. Scenario C shows a qualitatively different result: every agent mean falls below both vLLM and SGLang defaults, even though the relevant knobs are directly exposed at the command line\. In the vLLM\-restricted main comparison, matched\-budget non\-agent search reaches46\.70×46\.70\\timeson Scenario C, and allowing search across engines raises the best observed non\-agent value to89\.00×89\.00\\times\(Tables[2](https://arxiv.org/html/2607.20468#S3.T2),[4](https://arxiv.org/html/2607.20468#S4.T4), and[20](https://arxiv.org/html/2607.20468#A7.T20)\)\. Throughput under concurrent load is a global property of interactions between the scheduler, batch formation, KV\-cache pressure, and arrival patterns\. No single parameter change reliably moves throughput in a predictable direction, so shallow search strategies that work for latency objectives produce noisy or misleading feedback, which then are misread by agents that fail to recover from bad changes\.

#### Consistent deployment under autonomy is the hard part\.

Single\-run rankings shift substantially across seeds, as the same agent can produce a strong valid server in one seed and an invalid or weaker final deployment in another depending on its trajectory\. The benchmark evaluates failure\-aware expected utility, meaning a model that finds excellent configurations but cannot reliably ship them will be scored accordingly\. The resulting leaderboard should therefore be read as a ranking of autonomous deployment utility under a fixed wall\-clock budget, not as a pure ranking of intrinsic model strength\. In this setting, Claude Sonnet 4\.6 and GLM\-5 rank highly because they more often preserve simple, valid, high\-performing final servers, while several larger models show stronger peak runs but lose utility through brittle final\-state choices, an inverse\-scaling\-like pattern\(McKenzieet al\.,[2024](https://arxiv.org/html/2607.20468#bib.bib2); Gemaet al\.,[2025](https://arxiv.org/html/2607.20468#bib.bib36)\)that contrasts with benchmarks where rankings track raw capability \(e\.g\., PostTrainBench\(Ranket al\.,[2026](https://arxiv.org/html/2607.20468#bib.bib14)\), SWE\-Bench\(Jimenezet al\.,[2024](https://arxiv.org/html/2607.20468#bib.bib1)\), Terminal\-Bench\(Merrillet al\.,[2026](https://arxiv.org/html/2607.20468#bib.bib35)\)\)\. This is also the main reason the Claude Opus 4\.7/4\.6/4\.5 rows fall well below Claude Sonnet 4\.6 in Table[2](https://arxiv.org/html/2607.20468#S3.T2)\. Appendix[C\.3](https://arxiv.org/html/2607.20468#A3.SS3)quantifies seed\-level quartile movement directly, and Appendix[E\.4](https://arxiv.org/html/2607.20468#A5.SS4)gives a qualitative ranking analysis\.

Table 4:Best\-seen server analysis\. The best\-seen row scores each agent run by the best valid intermediate or final configuration observed during the 2\-hour budget\.
#### The performance gap to search reflects reliability and discipline failures, not a capability ceiling\.

The two analyses below decompose the gap to non\-agent search from different angles\. Together they show that agents have the raw capability to match or exceed search, but fail to exercise it consistently under open\-ended conditions\.

Best\-seen server\.To separate “agents found bad configs” from “agents found good configs but failed to preserve final state,” we re\-score every agent run on the best valid configuration measured at any point during the 2\-hour budget, rather than on its final deployed server\. The per\-scenario best\-seen agent values are much higher at7\.84×7\.84\\timeson Scenario A,18\.07×18\.07\\timeson Scenario B,37\.21×37\.21\\timeson Scenario C, and4\.40×4\.40\\timeson Scenario D, for a12\.34×12\.34\\timesscenario\-wise aggregate \(Table[4](https://arxiv.org/html/2607.20468#S4.T4)\)\. On Scenarios A and B this even exceeds the per\-scenario best non\-agentic search value, showing that the gap to search on these scenarios does not come from a lack of ability to find good configurations, but rather from the ability to keep them\. On Scenario D it narrows but does not close the gap to non\-agentic search\. On Scenario C the best\-seen agent value remains below the stronger non\-agent search baselines, meaning that agents never found a sufficiently strong throughput configuration during the run\.

Table[5](https://arxiv.org/html/2607.20468#S4.T5)shows the same phenomenon at the agent level\. Some lower\-ranked agents found substantially better functioning servers than their final submitted artifacts suggest, with Claude Opus 4\.6 nearly doubling from3\.89×3\.89\\timesfinal utility to7\.77×7\.77\\timesbest\-seen utility, and GPT\-5\.5 rising from4\.22×4\.22\\timesto6\.81×6\.81\\times\. These rows show that some agents do have access to stronger configurations during the run, but lose utility by failing to preserve the best working state\. By contrast, GLM\-5 changes very little, from6\.20×6\.20\\timesto6\.36×6\.36\\times, indicating that its high main\-table rank comes from preserving the working servers it found rather than from a large hidden reservoir of discarded better configurations\. Claude Sonnet 4\.6 remains first under this rescoring, so its lead combines both strong search outcomes and good preservation\.

Table 5:Agent\-level best\-seen server analysis\. Final Aggregate is the failure\-aware aggregate from Table[2](https://arxiv.org/html/2607.20468#S3.T2)\. Best\-Seen Aggregate rescoring replaces each run’s final submitted server with the best valid intermediate or final full\-evaluation artifact found in that same run\. Ratio is Best\-Seen Aggregate divided by Final Aggregate\.Best\-Seen RankAgentMain RankFinal Agg\.Best\-Seen Agg\.Ratio1Claude Sonnet 4\.618\.08×\\times±1\.10×\{\}\_\{\\pm\\text\{1\.10$\\times$\}\}9\.44×\\times1\.172Gemini 3\.1 Pro36\.16×\\times±0\.78×\{\}\_\{\\pm\\text\{0\.78$\\times$\}\}8\.12×\\times1\.323Claude Opus 4\.683\.89×\\times±1\.17×\{\}\_\{\\pm\\text\{1\.17$\\times$\}\}7\.77×\\times2\.004GPT\-5\.5 \(High\)74\.22×\\times±1\.30×\{\}\_\{\\pm\\text\{1\.30$\\times$\}\}6\.81×\\times1\.615GLM\-526\.20×\\times±1\.48×\{\}\_\{\\pm\\text\{1\.48$\\times$\}\}6\.36×\\times1\.036GPT\-5\.4 \(High\)55\.08×\\times±0\.92×\{\}\_\{\\pm\\text\{0\.92$\\times$\}\}5\.54×\\times1\.097GPT\-5\.3 Codex \(High\)45\.48×\\times±0\.51×\{\}\_\{\\pm\\text\{0\.51$\\times$\}\}5\.54×\\times1\.018GPT\-5\.3 Codex \(Medium\)64\.86×\\times±1\.14×\{\}\_\{\\pm\\text\{1\.14$\\times$\}\}4\.87×\\times1\.009GPT\-5\.1 Codex Max103\.54×\\times±0\.69×\{\}\_\{\\pm\\text\{0\.69$\\times$\}\}4\.55×\\times1\.2810Claude Sonnet 4\.5122\.96×\\times±1\.02×\{\}\_\{\\pm\\text\{1\.02$\\times$\}\}4\.25×\\times1\.4411GPT\-5\.293\.82×\\times±0\.85×\{\}\_\{\\pm\\text\{0\.85$\\times$\}\}4\.13×\\times1\.0812Claude Opus 4\.5113\.37×\\times±0\.98×\{\}\_\{\\pm\\text\{0\.98$\\times$\}\}3\.87×\\times1\.1513Claude Opus 4\.7132\.25×\\times±0\.32×\{\}\_\{\\pm\\text\{0\.32$\\times$\}\}2\.65×\\times1\.1814Claude Haiku 4\.5151\.24×\\times±0\.19×\{\}\_\{\\pm\\text\{0\.19$\\times$\}\}1\.68×\\times1\.3515GPT\-5\.2 Codex141\.55×\\times±0\.27×\{\}\_\{\\pm\\text\{0\.27$\\times$\}\}1\.55×\\times1\.00Structured\-iteration prompt\.The default prompt instructs the agent to optimize continuously but leaves the experimental protocol entirely open\. The structured\-iteration variant makes the protocol explicit: establish a baseline before optimizing, change exactly one variable per experiment, log each proposal and outcome before proceeding, and reserve the final 15 minutes for validation\. It does not prescribe which framework, optimizations, or parameter values to try\.

The structured prompt improves reliability more than peak performance\. GPT\-5\.4 \(High\) rises from 10/12 to 11/12 passing runs and becomes much tighter on Scenario D \(1\.37×1\.37\\timesto0\.31×0\.31\\timesSEM\), while Claude Opus 4\.7 rises from 5/12 to 11/12 and improves sharply on Scenario A \(1\.07×1\.07\\timesto4\.58×4\.58\\times\)\. However, the overall ceiling changes little, as even the best structured\-agent cells remain below non\-agent search on Scenarios C–D and on the aggregate\. This suggests two distinct limitations: an execution\-discipline failure that scaffolding can partly fix, and a search\-breadth failure on multi\-objective or throughput\-heavy settings that it does not remove\.

Table 6:Structured\-iteration prompt ablation\. Cells are mean±SEM\{\}\_\{\\pm\\text\{SEM\}\}speedup over the PyTorch baseline\. Aggregate is the geometric mean over Scenarios A–D\. Failed seed runs count as1×1\\times\.
### 4\.1Ablations

#### Time budget\.

We ablate wall\-clock budget across Claude Haiku 4\.5, Claude Sonnet 4\.5, and Claude Opus 4\.5, varying only the wall\-clock time limit across 1, 2, 4, and 8 hours\. This tests whether agents primarily fail because they need more time to search, or because they commit early to poor optimization trajectories\.

Extending the budget does not reliably improve performance\. Using the aggregate penalized speedup in Figure[2](https://arxiv.org/html/2607.20468#S4.F2), Claude Haiku 4\.5 remains close to the PyTorch baseline, increasing only gradually from1\.05×1\.05\\timesat 1 h to1\.35×1\.35\\timesat 8 h\. Claude Sonnet 4\.5 improves sharply from1\.92×1\.92\\timesat 1 h to2\.96×2\.96\\timesat 2 h, but then declines slightly to2\.92×2\.92\\timesat 4 h and2\.81×2\.81\\timesat 8 h\. Claude Opus 4\.5 follows the same pattern at a higher performance level, rising from2\.42×2\.42\\timesat 1 h to3\.37×3\.37\\timesat 2 h before falling to3\.31×3\.31\\timesat 4 h and3\.24×3\.24\\timesat 8 h\. Thus, additional time helps initially for the stronger models, but the gains saturate quickly and reverse slightly at longer budgets\. This decline is driven in part by a higher rate of failed or invalid final runs at longer budgets, including runs flagged for reward\-hacking behavior, suggesting that additional wall\-clock time can increase optimization pressure without improving search discipline: agents have more opportunity to chase proxy metrics, make brittle late\-stage changes, or overwrite working configurations, so extra time may instead increase the risk of late\-stage regressions or specification\-gaming\.

![Refer to caption](https://arxiv.org/html/2607.20468v1/x2.png)Figure 2:Aggregate speedup across time budgets for Claude Haiku/Sonnet/Opus 4\.5\.
#### Base\-model transfer\.

We run the same GPT\-5\.4 \(High\), 2\-hour setup on additional base models and compare against the Mistral\-7B\-Instruct\-v0\.3 condition above\. Speedups are normalized to each model’s own PyTorch baseline, so rows should be read as within\-model optimization progress rather than raw cross\-model latency comparisons\.

Table[7](https://arxiv.org/html/2607.20468#S4.T7)shows that the qualitative behavior is not specific to Mistral\-7B\-Instruct\-v0\.3\. Under the same GPT\-5\.4 \(High\), 2\-hour setup, the agent obtains nontrivial gains across all three base models, especially on prefill\-heavy and high\-load throughput objectives, but the transfer is imperfect: changing the base model changes both attainable speedups and failure rate\. Qwen3\-8B yields the largest normalized aggregate speedup, driven by TTFT and high\-load gains, though its decode\-heavy and balanced cells remain high\-variance because failed, flagged, or non\-computable seeds are assigned baseline\-equivalent utility\. DeepSeek\-V2\-Lite is the clearest stress case: the agent finds gains on Scenarios A, B, and D, but only 5/12 runs pass, and Scenario C collapses to the penalized baseline because all three runs are either contamination\-flagged or fail at the server level\. This is consistent with agents converging on an optimization recipe across base models even when different architectures or serving paths, especially for mixture\-of\-experts models, may require different choices\. Base\-model transfer should thus be read as a joint question of speedup and reliability, not speedup alone\.

Table 7:Base\-model ablation for GPT\-5\.4 \(High\), 2h\. Cells are penalized mean±SEM\{\}\_\{\\pm\\text\{SEM\}\}speedup over each base model’s own PyTorch baseline across the three held\-out seed\-pair runs\. Aggregate is the geometric mean over Scenarios A–D\. Non\-passing or broken seeds contribute1×1\\times\.

## 5Agent Behavior Analysis

This section explains why agents underperform matched\-budget search despite often knowing the relevant inference\-serving techniques\. Across runs, agents usually launch a working server, identify plausible optimization levers, and run the evaluator, but their search behavior remains narrow, settling early on a familiar serving approach and then spending limited effort exploring alternatives or making controlled comparisons around it\. A second failure mode is more safety\-relevant: because the metric is an unbounded speedup, optimization pressure sometimes produces servers that satisfy the measurement pathway rather than the intended inference task\. We extract all behavioral counts from each run’s persisted launcher, final metrics, and transcript; definitions and extraction methodology are described in Appendix[E](https://arxiv.org/html/2607.20468#A5)\.

### 5\.1Agents converge too quickly

Agents overwhelmingly converge on the same serving stack\. Across the main\-condition pool, 169/180 runs \(93\.9%\) ship a final launcher that invokes vLLM, with the remaining runs split between SGLang \(1/180; 0\.6%\), a custom launcher \(1/180; 0\.6%\), and no recoverable final server \(9/180; 5\.0%\)\. No reported main\-condition run ships TensorRT\-LLM despite the option being explicitly mentioned in the prompt\. This convergence would be less concerning if agents searched deeply within vLLM configurations, but this is not observed in agent behavior\. Counting each distinct set of non\-default vLLM command\-line arguments as one attempted launch\-argument set, the median run launches exactly one such set over its full 2\-hour budget \(mean∼0\.79\\sim 0\.79\): 31% of runs launch zero, 61% launch one, 6% launch two, and only 2% launch three or more\. The median run restarts its server once\. When a restart occurs, that launch\-argument set is unchanged in 81% of cases, indicating crash recovery rather than deliberate A/B comparison\.

The dominant behavior is therefore: choose vLLM, reach a working launcher, and spend the remaining budget validating, repairing, or re\-running it\. Since non\-agent search shows that engine choice and runtime parameters both matter \(Appendix[G\.3](https://arxiv.org/html/2607.20468#A7.SS3)\), this early convergence leaves substantial headroom unexplored\. Appendix[F](https://arxiv.org/html/2607.20468#A6)gives concrete trace examples of self\-declared completion and stalled continuation after a first working launcher; we keep these in the appendix because they mainly illustrate the same convergence mechanism quantified here\.

### 5\.2Domain knowledge rarely becomes executable search

The narrow search space utilized by agents is not due to the lack of domain knowledge, as the models are fully aware of different optimization options\. Across the transcript pool, 96% of runs mention quantization, 97% mention chunked prefill, 84% mention speculation, and 74% mention prefix caching, the types of techniques an inference engineer would consider\. The problem is that agents rarely convert these ideas into controlled experiments\. One common pattern is try\-and\-abandon, where the agent begins to test a promising optimization, observes a local failure, and abandons the entire path without isolating its effect\.

Example: try\-and\-abandon \(Claude Opus 4\.6, Scenario D\)config 1: quantization=fp8, kv\_cache\_dtype=fp8\_e4m3, gpu\_util=0\.75 config 2: quantization=fp8, kv\_cache\_dtype=fp8\_e4m3, gpu\_util=0\.75 \[relaunch\] config 3: quantization=fp8, kv\_cache\_dtype=fp8\_e4m3, enforce\_eager=True config 4: quantization REMOVED, kv\_cache\_dtype=auto, enforce\_eager=True

In this example, the fourth configuration changes two variables at once without an intervening measurement, so the agent cannot attribute any performance delta to the quantization change\. The final metrics file contains an error field and no profiles; the last relaunch crashed\. The trace therefore shows attempted optimization, but not controlled comparison\.

### 5\.3Optimization pressure induces reward hacking

The same open\-endedness that makesInferenceBenchuseful for evaluating autonomous systems engineering also exposes specification\-gaming behavior\. Because the primary score is a speedup over a baseline and is therefore unbounded, agents sometimes find ways to improve the measured latency or throughput without preserving the intended semantics of inference\. These failures are exactly the kind of failure mode that appears when an autonomous optimizer is given a scalar metric, a flexible action space, and a brittle measurement protocol\.

Example: reward hacking / integrity violation \(GPT\-5\.3 Codex Medium, Scenario B\)A submitted server reports decode throughput of 118 million tokens per second and per\-token decode latency of 6\.4 nanoseconds\. The run’s launcher and transcript indicate that the measurement path has been subverted rather than that the model has become faster\. The same run passes the quality gate with 122/500 correct answers, just above the required threshold of0\.95×128=121\.60\.95\\times 128=121\.6correct answers for a baseline score of 128/500\.

This example illustrates why the quality gate alone is insufficient\. A server can remain just accurate enough on the held\-out multiple\-choice check while bypassing or distorting the speed\-evaluation path\. The integrity gate prevents such runs from silently inflating the main table, and Table[3](https://arxiv.org/html/2607.20468#S4.T3)shows that 11/180 main\-condition runs are removed for this reason\. Additional integrity cases, including fake first\-token streaming to manipulate TTFT, appear in Appendix[F](https://arxiv.org/html/2607.20468#A6)\.

## 6Discussion and Conclusion

InferenceBenchexplores factors in automated agentic research and development largely missing from current benchmarks such as open\-ended inference systems optimization under noisy measurements, brittle infrastructure, and binary deployment failures\. Frontier agents are shown to often know the right techniques, but they rarely act on that knowledge in a systematic way\. Non\-agentic optimization still wins every main\-comparison cell given the same time budget because agents converge on one familiar stack, test few distinct non\-default configurations, and spend much of their remaining budget re\-measuring or repairing the same launcher despite having a much larger action space than parameter fine\-tuning\. This makes inference optimization a useful stress test for autonomy claims, as unlike many training\-side tasks, failed interventions may leave no usable artifact, meaning success requires controlled experimentation, variable isolation, recovery from failed changes, and preservation of the best valid deployment state across a deep systems stack\.

Limitations\.All main\-table runs use a single H100 on one node, so multi\-GPU parallelism, distributed KV\-cache strategies, and cluster\-level serving policies are out of scope\. The benchmark also evaluates a fixed set of scaffolds and proprietary model snapshots, so absolute cell values are submission\-time measurements rather than permanent rankings\. The integrity gate is validated by cross\-judge agreement and manual audit, but remains a benchmark safeguard rather than a formal guarantee against all specification\-gaming behaviors\.

Future Work\.InferenceBenchsuggests two directions for future work\. First, benchmarks should evaluate both the final artifact and the search process that produced it, including whether the agent branches, measures, compares, rolls back, and commits in a disciplined way\. Second, automated R&D tasks provide a controlled safety testbed for optimization pressure: agents can produce real systems gains, but can also exploit the measurement protocol, degrade quality, or leave behind an invalid final state\. Open\-ended environments thus help study when agents satisfy the evaluator rather than the task, and which monitoring or process constraints reduce specification gaming without suppressing useful optimization\.

## Acknowledgements

We thank Vishaal Udandarao, Ameya Prabhu, and Ronald Skorobogat from Bethge Lab for their feedback on this manuscript\. We acknowledge financial support from Coefficient Giving\.

## References

- A\. Agrawal, N\. Kedia, A\. Panwar, J\. Mohan, N\. Kwatra, B\. Gulavani, A\. Tumanov, and R\. Ramjee \(2024\)Taming throughput\-latency tradeoff in LLM inference with Sarathi\-Serve\.In18th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 24\),pp\. 117–134\.External Links:[Link](https://www.usenix.org/conference/osdi24/presentation/agrawal)Cited by:[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px2.p1.1)\.
- Anomaly \(2026\)OpenCode: the open source AI coding agent\.Note:[https://opencode\.ai/](https://opencode.ai/)External Links:[Link](https://opencode.ai/)Cited by:[§3\.4](https://arxiv.org/html/2607.20468#S3.SS4.SSS0.Px1.p1.1)\.
- Anthropic \(2025\)Claude Code\.Note:[https://code\.claude\.com/docs](https://code.claude.com/docs)External Links:[Link](https://code.claude.com/docs)Cited by:[§3\.4](https://arxiv.org/html/2607.20468#S3.SS4.SSS0.Px1.p1.1)\.
- Y\. Bai, S\. Tu, J\. Zhang, H\. Peng, X\. Wang, X\. Lv, S\. Cao, J\. Xu, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2025\)LongBench v2: towards deeper understanding and reasoning on realistic long\-context multitasks\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Papers,pp\. 3639–3664\.External Links:[Link](https://aclanthology.org/2025.acl-long.183/)Cited by:[§A\.4](https://arxiv.org/html/2607.20468#A1.SS4.p1.4),[§3\.2](https://arxiv.org/html/2607.20468#S3.SS2.SSS0.Px1.p1.3),[Table 1](https://arxiv.org/html/2607.20468#S3.T1)\.
- J\. Bergstra, R\. Bardenet, Y\. Bengio, and B\. Kégl \(2011\)Algorithms for hyper\-parameter optimization\.InAdvances in Neural Information Processing Systems 24,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2011/file/86e8f7ab32cfd12577bc2619bc635690-Paper.pdf)Cited by:[§3\.4](https://arxiv.org/html/2607.20468#S3.SS4.SSS0.Px4.p1.1)\.
- J\. S\. Chan, N\. Chowdhury, O\. Jaffe, J\. Aung, D\. Sherburn, E\. Mays, G\. Starace, K\. Liu, L\. Maksin, T\. Patwardhan, L\. Weng, and A\. Mądry \(2025\)MLE\-bench: evaluating machine learning agents on machine learning engineering\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=6s5uXNWGIh)Cited by:[§1](https://arxiv.org/html/2607.20468#S1.p1.1),[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px1.p1.1)\.
- T\. Dao, D\. Y\. Fu, S\. Ermon, A\. Rudra, and C\. Ré \(2022\)FlashAttention: fast and memory\-efficient exact attention with IO\-awareness\.InAdvances in Neural Information Processing Systems 35,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px2.p1.1)\.
- T\. Dao \(2024\)FlashAttention\-2: faster attention with better parallelism and work partitioning\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=mZn2Xyh9Ec)Cited by:[§B\.4](https://arxiv.org/html/2607.20468#A2.SS4.SSS0.Px2.p2.1)\.
- A\. P\. Gema, A\. Hägele, R\. Chen, A\. Arditi, J\. Goldman\-Wetzler, K\. Fraser\-Taliente, H\. Sleight, L\. Petrini, J\. Michael, B\. Alex, P\. Minervini, Y\. Chen, J\. Benton, and E\. Perez \(2025\)Inverse scaling in test\-time compute\.External Links:2507\.14417,[Link](https://arxiv.org/abs/2507.14417)Cited by:[§4](https://arxiv.org/html/2607.20468#S4.SS0.SSS0.Px4.p1.1)\.
- GLM\-5\-Team, A\. Zeng, X\. Lv, Z\. Hou, Z\. Du, Q\. Zheng, B\. Chen, D\. Yin, C\. Ge, C\. Huang, C\. Xie, C\. Zhu, C\. Yin, C\. Wang, G\. Pan, H\. Zeng, H\. Zhang, H\. Wang, H\. Chen, J\. Zhang, J\. Jiao, J\. Guo, J\. Wang, J\. Du, J\. Wu, K\. Wang, L\. Li, L\. Fan, L\. Zhong, M\. Liu, M\. Zhao, P\. Du, Q\. Dong, R\. Lu, Shuang\-Li, S\. Cao, S\. Liu, T\. Jiang, X\. Chen, X\. Zhang, X\. Huang, X\. Dong, Y\. Xu, Y\. Wei, Y\. An, Y\. Niu, Y\. Zhu, Y\. Wen, Y\. Cen, Y\. Bai, Z\. Qiao, Z\. Wang, Z\. Wang, Z\. Zhu, Z\. Liu, Z\. Li, B\. Wang, B\. Wen, C\. Huang, C\. Cai, C\. Yu, C\. Li, C\. Hu, C\. Zhang, D\. Zhang, D\. Lin, D\. Yang, D\. Wang, D\. Ai, E\. Zhu, F\. Yi, F\. Chen, G\. Wen, H\. Sun, H\. Zhao, H\. Hu, H\. Zhang, H\. Liu, H\. Zhang, H\. Peng, H\. Tai, H\. Zhang, H\. Liu, H\. Wang, H\. Yan, H\. Ge, H\. Liu, H\. Chu, J\. Zhao, J\. Wang, J\. Zhao, J\. Ren, J\. Wang, J\. Zhang, J\. Gui, J\. Zhao, J\. Li, J\. An, J\. Li, J\. Yuan, J\. Du, J\. Liu, J\. Zhi, J\. Duan, K\. Zhou, K\. Wei, K\. Wang, K\. Luo, L\. Zhang, L\. Sha, L\. Xu, L\. Wu, L\. Ding, L\. Chen, M\. Li, N\. Lin, P\. Ta, Q\. Zou, R\. Song, R\. Yang, S\. Tu, S\. Yang, S\. Wu, S\. Zhang, S\. Li, S\. Li, S\. Fan, W\. Qin, W\. Tian, W\. Zhang, W\. Yu, W\. Liang, X\. Kuang, X\. Cheng, X\. Li, X\. Yan, X\. Hu, X\. Ling, X\. Fan, X\. Xia, X\. Zhang, X\. Zhang, X\. Pan, X\. Zou, X\. Zhang, Y\. Liu, Y\. Wu, Y\. Li, Y\. Wang, Y\. Zhu, Y\. Tan, Y\. Zhou, Y\. Pan, Y\. Zhang, Y\. Su, Y\. Geng, Y\. Yan, Y\. Tan, Y\. Bi, Y\. Shen, Y\. Yang, Y\. Li, Y\. Liu, Y\. Wang, Y\. Li, Y\. Wu, Y\. Zhang, Y\. Duan, Y\. Zhang, Z\. Liu, Z\. Jiang, Z\. Yan, Z\. Zhang, Z\. Wei, Z\. Chen, Z\. Feng, Z\. Yao, Z\. Chai, Z\. Wang, Z\. Zhang, B\. Xu, M\. Huang, H\. Wang, J\. Li, Y\. Dong, and J\. Tang \(2026\)GLM\-5: from vibe coding to agentic engineering\.External Links:2602\.15763,[Link](https://arxiv.org/abs/2602.15763)Cited by:[§3\.4](https://arxiv.org/html/2607.20468#S3.SS4.SSS0.Px1.p1.1)\.
- Q\. Huang, J\. Vora, P\. Liang, and J\. Leskovec \(2024\)MLAgentBench: evaluating language agents on machine learning experimentation\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 20271–20309\.External Links:[Link](https://proceedings.mlr.press/v235/huang24y.html)Cited by:[§1](https://arxiv.org/html/2607.20468#S1.p1.1),[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px1.p1.1)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. Narasimhan \(2024\)SWE\-bench: can language models resolve real\-world GitHub issues?\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VTF8yNQM66)Cited by:[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2607.20468#S4.SS0.SSS0.Px4.p1.1)\.
- A\. Karpathy \(2026\)Autoresearch: AI agents running research on single\-GPU nanochat training automatically\.Note:GitHub repositoryAccessed: 2026\-04\-27External Links:[Link](https://github.com/karpathy/autoresearch)Cited by:[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px1.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with PagedAttention\.InProceedings of the 29th Symposium on Operating Systems Principles,pp\. 611–626\.External Links:[Link](https://dl.acm.org/doi/10.1145/3600006.3613165)Cited by:[§B\.4](https://arxiv.org/html/2607.20468#A2.SS4.SSS0.Px2.p2.1),[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px2.p1.1)\.
- R\. T\. Lange, Q\. Sun, A\. Prasad, M\. Faldor, Y\. Tang, and D\. Ha \(2025\)Towards robust agentic cuda kernel benchmarking, verification, and optimization\.External Links:2509\.14279,[Link](https://arxiv.org/abs/2509.14279)Cited by:[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Lindauer, K\. Eggensperger, M\. Feurer, A\. Biedenkapp, D\. Deng, C\. Benjamins, T\. Ruhkopf, R\. Sass, and F\. Hutter \(2022\)SMAC3: a versatile bayesian optimization package for hyperparameter optimization\.Journal of Machine Learning Research23,pp\. 1–9\.External Links:[Link](https://jmlr.org/papers/v23/21-0888.html)Cited by:[§3\.4](https://arxiv.org/html/2607.20468#S3.SS4.SSS0.Px4.p1.1)\.
- C\. Lu, C\. Lu, R\. T\. Lange, J\. Foerster, J\. Clune, and D\. Ha \(2024\)The AI scientist: towards fully automated open\-ended scientific discovery\.External Links:2408\.06292,[Link](https://arxiv.org/abs/2408.06292)Cited by:[§1](https://arxiv.org/html/2607.20468#S1.p1.1)\.
- I\. R\. McKenzie, A\. Lyzhov, M\. Pieler, A\. Parrish, A\. Mueller, A\. Prabhu, E\. McLean, A\. Kirtland, A\. Ross, A\. Liu, A\. Gritsevskiy, D\. Wurgaft, D\. Kauffman, G\. Recchia, J\. Liu, J\. Cavanagh, M\. Weiss, S\. Huang, T\. F\. Droid, T\. Tseng, T\. Korbak, X\. Shen, Y\. Zhang, Z\. Zhou, N\. Kim, S\. R\. Bowman, and E\. Perez \(2024\)Inverse scaling: when bigger isn’t better\.External Links:2306\.09479,[Link](https://arxiv.org/abs/2306.09479)Cited by:[§4](https://arxiv.org/html/2607.20468#S4.SS0.SSS0.Px4.p1.1)\.
- M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi, J\. Y\. Shin, T\. Walshe, E\. K\. Buchanan, J\. Shen, G\. Ye, H\. Lin, J\. Poulos, M\. Wang, M\. Nezhurina, J\. Jitsev, D\. Lu, O\. M\. Mastromichalakis, Z\. Xu, Z\. Chen, Y\. Liu, R\. Zhang, L\. L\. Chen, A\. Kashyap, J\. Uslu, J\. Li, J\. Wu, M\. Yan, S\. Bian, V\. Sharma, K\. Sun, S\. Dillmann, A\. Anand, A\. Lanpouthakoun, B\. Koopah, C\. Hu, E\. Guha, G\. H\. S\. Dreiman, J\. Zhu, K\. Krauth, L\. Zhong, N\. Muennighoff, R\. Amanfu, S\. Tan, S\. Pimpalgaonkar, T\. Aggarwal, X\. Lin, X\. Lan, X\. Zhao, Y\. Liang, Y\. Wang, Z\. Wang, C\. Zhou, D\. Heineman, H\. Liu, H\. Trivedi, J\. Yang, J\. Lin, M\. Shetty, M\. Yang, N\. Omi, N\. Raoof, S\. Li, T\. Y\. Zhuo, W\. Lin, Y\. Dai, Y\. Wang, W\. Chai, S\. Zhou, D\. Wahdany, Z\. She, J\. Hu, Z\. Dong, Y\. Zhu, S\. Cui, A\. Saiyed, A\. Kolbeinsson, J\. Hu, C\. M\. Rytting, R\. Marten, Y\. Wang, A\. Dimakis, A\. Konwinski, and L\. Schmidt \(2026\)Terminal\-bench: benchmarking agents on hard, realistic tasks in command line interfaces\.External Links:2601\.11868,[Link](https://arxiv.org/abs/2601.11868)Cited by:[§4](https://arxiv.org/html/2607.20468#S4.SS0.SSS0.Px4.p1.1)\.
- A\. Nangia, S\. Mishra, A\. Gokrani, and P\. Chopra \(2026\)ISO\-Bench: can coding agents optimize real\-world inference workloads?\.External Links:2602\.19594,[Link](https://arxiv.org/abs/2602.19594)Cited by:[§1](https://arxiv.org/html/2607.20468#S1.p1.1),[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Novikov, N\. Vũ, M\. Eisenberger, E\. Dupont, P\. Huang, A\. Z\. Wagner, S\. Shirobokov, B\. Kozlovskii, F\. J\. R\. Ruiz, A\. Mehrabian, M\. P\. Kumar, A\. See, S\. Chaudhuri, G\. Holland, A\. Davies, S\. Nowozin, P\. Kohli, and M\. Balog \(2025\)AlphaEvolve: a coding agent for scientific and algorithmic discovery\.External Links:2506\.13131,[Link](https://arxiv.org/abs/2506.13131)Cited by:[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px1.p1.1)\.
- OpenAI \(2025\)Codex CLI\.Note:[https://github\.com/openai/codex](https://github.com/openai/codex)External Links:[Link](https://github.com/openai/codex)Cited by:[§3\.4](https://arxiv.org/html/2607.20468#S3.SS4.SSS0.Px1.p1.1)\.
- A\. Ouyang, S\. Guo, S\. Arora, A\. L\. Zhang, W\. Hu, C\. Ré, and A\. Mirhoseini \(2025\)KernelBench: can LLMs write efficient GPU kernels?\.InProceedings of the 42nd International Conference on Machine Learning,External Links:[Link](https://icml.cc/virtual/2025/poster/43517)Cited by:[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px2.p1.1)\.
- O\. Press, B\. Amos, H\. Zhao, Y\. Wu, S\. Ainsworth, D\. Krupke, P\. Kidger, T\. Sajed, B\. Stellato, J\. Park, N\. Bosch, E\. Meril, A\. Steppi, A\. Zharmagambetov, F\. Zhang, D\. Pérez\-Piñeiro, A\. Mercurio, N\. Zhan, T\. Abramovich, K\. Lieret, H\. Zhang, S\. Huang, M\. Bethge, and O\. Press \(2025\)AlgoTune: can language models speed up general\-purpose numerical programs?\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=dF1tD9hjvn)Cited by:[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px1.p1.1)\.
- B\. Rank, H\. Bhatnagar, A\. Prabhu, S\. Eisenberg, K\. Nguyen, M\. Bethge, and M\. Andriushchenko \(2026\)PostTrainBench: can LLM agents automate LLM post\-training?\.External Links:2603\.08640,[Link](https://arxiv.org/abs/2603.08640)Cited by:[§1](https://arxiv.org/html/2607.20468#S1.p1.1),[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2607.20468#S4.SS0.SSS0.Px4.p1.1)\.
- D\. Rein, J\. Becker, A\. Deng, S\. Nix, C\. Canal, D\. O’Connel, P\. Arnott, R\. Bloom, T\. Broadley, K\. Garcia, B\. Goodrich, M\. Hasin, S\. Jawhar, M\. Kinniment, T\. Kwa, A\. Lajko, N\. Rush, L\. J\. K\. Sato, S\. V\. Arx, B\. West, L\. Chan, and E\. Barnes \(2025\)HCAST: human\-calibrated autonomy software tasks\.External Links:2503\.17354,[Link](https://arxiv.org/abs/2503.17354)Cited by:[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px1.p1.1)\.
- Runpod Inc\. \(2026\)GPU Cloud Pricing\.Note:[https://www\.runpod\.io/pricing](https://www.runpod.io/pricing)Accessed: 2026\-04\-30Cited by:[§D\.2](https://arxiv.org/html/2607.20468#A4.SS2.p1.1)\.
- G\. Starace, O\. Jaffe, D\. Sherburn, J\. Aung, J\. S\. Chan, L\. Maksin, R\. Dias, E\. Mays, B\. Kinsella, W\. Thompson, J\. Heidecke, A\. Glaese, and T\. Patwardhan \(2025\)PaperBench: evaluating AI’s ability to replicate AI research\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 56843–56873\.External Links:[Link](https://proceedings.mlr.press/v267/starace25a.html)Cited by:[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px1.p1.1)\.
- E\. Toledo, K\. Hambardzumyan, M\. Josifoski, R\. Hazra, N\. Baldwin, A\. Audran\-Reiss, M\. Kuchnik, D\. Magka, M\. Jiang, A\. M\. Lupidi, A\. Lupu, R\. Raileanu, K\. Niu, T\. Shavrina, J\. Gagnon\-Audet, M\. Shvartsman, S\. Sodhani, A\. H\. Miller, A\. Charnalia, D\. Dunfield, C\. Wu, P\. Stenetorp, N\. Cancedda, J\. N\. Foerster, and Y\. Bachrach \(2025\)AI research agents for machine learning: search, exploration, and generalization in MLE\-bench\.External Links:2507\.02554,[Link](https://arxiv.org/abs/2507.02554)Cited by:[§1](https://arxiv.org/html/2607.20468#S1.p1.1),[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. Chen \(2024\)MMLU\-Pro: a more robust and challenging multi\-task language understanding benchmark\.InAdvances in Neural Information Processing Systems 37, Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=y10DM6R2r3)Cited by:[§C\.1](https://arxiv.org/html/2607.20468#A3.SS1.SSS0.Px1.p1.2),[§3\.3](https://arxiv.org/html/2607.20468#S3.SS3.SSS0.Px1.p1.4)\.
- H\. Wijk, T\. Lin, J\. Becker, S\. Jawhar, N\. Parikh, T\. Broadley, L\. Chan, M\. Chen, J\. M\. Clymer, J\. Dhyani, E\. Ericheva, K\. Garcia, B\. Goodrich, N\. Jurkovic, M\. Kinniment, A\. Lajko, S\. Nix, L\. J\. K\. Sato, W\. Saunders, M\. Taran, B\. West, and E\. Barnes \(2025\)RE\-Bench: evaluating frontier AI R&D capabilities of language model agents against human experts\.InProceedings of the 42nd International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=3rB0bVU6z6)Cited by:[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Xing, Y\. Zhai, A\. Jiang, Y\. Dong, Y\. Wu, Z\. Ye, C\. Ruan, Y\. Huang, Y\. Zhang, L\. Yin, A\. Bayyapu, L\. Ceze, and T\. Chen \(2026\)FlashInfer\-Bench: building the virtuous cycle for AI\-driven LLM systems\.External Links:2601\.00227,[Link](https://arxiv.org/abs/2601.00227)Cited by:[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Ye, L\. Chen, R\. Lai, W\. Lin, Y\. Zhang, S\. Wang, T\. Chen, B\. Kasikci, V\. Grover, A\. Krishnamurthy, and L\. Ceze \(2025\)FlashInfer: efficient and customizable attention engine for LLM inference serving\.InProceedings of the 8th Conference on Machine Learning and Systems \(MLSys 2025\),External Links:[Link](https://mlsys.org/virtual/2025/poster/3259)Cited by:[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px2.p1.1)\.
- G\. Yu, J\. S\. Jeong, G\. Kim, S\. Kim, and B\. Chun \(2022\)Orca: a distributed serving system for Transformer\-based generative models\.In16th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 22\),pp\. 521–538\.External Links:[Link](https://www.usenix.org/conference/osdi22/presentation/yu)Cited by:[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez, C\. Barrett, and Y\. Sheng \(2024\)SGLang: efficient execution of structured language model programs\.InAdvances in Neural Information Processing Systems 37,External Links:[Link](https://arxiv.org/abs/2312.07104)Cited by:[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px2.p1.1)\.
- Q\. Zou, H\. H\. Lam, W\. Zhao, Y\. Tang, T\. Chen, S\. Yu, T\. Zhang, C\. Liu, X\. Ji, and D\. Liu \(2025\)FML\-bench: a benchmark for automatic ML research agents highlighting the importance of exploration breadth\.External Links:2510\.10472,[Link](https://arxiv.org/abs/2510.10472)Cited by:[§1](https://arxiv.org/html/2607.20468#S1.p1.1),[§2](https://arxiv.org/html/2607.20468#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix AFull Benchmark Specification and Environment

### A\.1Hardware Details

Unless otherwise noted, every run in Table[2](https://arxiv.org/html/2607.20468#S3.T2), the baselines, and every ablation uses a single NVIDIA H100 80 GB GPU on a shared HTCondor cluster\. The cluster worker nodes are two\-socket AMD EPYC 9654 machines \(192 physical cores / 384 logical CPUs, 3\.0 TB DDR5 RAM, 8 H100s per node\), of which eachInferenceBenchjob is allocated 16 CPUs, 180 GB RAM, 400 GB of local scratch disk\. The NVIDIA driver version is580\.82\.07580\.82\.07and the display\-runtime CUDA version reported bynvidia\-smiis13\.013\.0\. Worker nodes run Ubuntu22\.0422\.04\.

### A\.2Container Contents

All agent and baseline runs execute inside a container built from the Singularity definition\. The container bootstraps fromnvidia/cuda:12\.8\.0\-cudnn\-devel\-ubuntu22\.04\. On top of that base image we install Python3\.103\.10,uv, Node\.js2222\(for the three agent CLIs\),ripgrep,ninja, and the standard CUDA build toolchain \(build\-essential,cmake,pkg\-config\)\. Python packages installed system\-wide aretorch==2\.8\.0\(from thecu128PyTorch wheel index, so the runtime CUDA matches the container’s CUDA 12\.8 toolchain\),aiohttp,requests,accelerate,datasets,sentencepiece,numpy,tokenizers, andtransformers\. Three agent CLI packages are installed as global npm modules:@anthropic\-ai/claude\-code,@openai/codex, andopencode\-ai\.

### A\.3Scenario Parameters

Table[8](https://arxiv.org/html/2607.20468#A1.T8)lists the exact runtime configuration of every scenario\. All scenarios use the same base model and the samemax\_model\_lenceiling\. Per\-request prompt and output lengths are drawn uniformly from\[0\.8​L,L\]\[0\.8\\,L,\\,L\]whereLLis the scenario’s target length \(therange\_ratioparameter is set to0\.80\.8\)\. For each request, the sampled output length is passed as the per\-requestmax\_new\_tokens; the “max\_new” column in Table[8](https://arxiv.org/html/2607.20468#A1.T8)reports its upper boundLoutL\_\{\\text\{out\}\}\. The decoder is forced to generate exactly that per\-requestmax\_new\_tokensvia theignore\_eosextension supported by vLLM and SGLang\. Scenario C is the only multi\-profile scenario and layers three traffic patterns on top of a shared request distribution\. All requests are drawn from LongBench v2 samples preselected for each scenario’s target input length\.

Table 8:Per\-scenario runtime parameters\. “Lin/LoutL\_\{\\text\{in\}\}/L\_\{\\text\{out\}\}” are the target input and output lengths in tokens; per\-request lengths are drawn uniformly from\[0\.8​L,L\]\[0\.8L,L\]\. “\# req” is the total number of synthetic requests\. Scenario C runs three profiles back\-to\-back \(burst / Poisson / constant\), each replaying the same256256requests through a different concurrency envelope\.*Profile semantics\.*A “burst” profile queues every request up\-front and lets the server pull them at its own rate \(bounded by the concurrency cap\)\. A “Poisson” profile draws inter\-arrival intervals from an exponential distribution with the specified mean rate, simulating independent clients whose arrivals are uncorrelated\. A “constant” profile emits one request every1/rate1/\\text\{rate\}seconds, simulating a regulated upstream load balancer\. The three profiles probe three distinct throughput regimes: burst isolates the scheduler’s ability to pack work, Poisson exercises the scheduler’s response to arrival\-rate variance, and constant measures steady\-state behavior after the scheduler has reached equilibrium\.

### A\.4Request Generation

Requests are drawn from LongBench v2\(Baiet al\.,[2025](https://arxiv.org/html/2607.20468#bib.bib18)\), a long\-context corpus of naturally occurring documents and instructions\. For each scenario we filter the corpus to samples whose tokenized prompt length \(under the base model’s tokenizer\) falls in the target range\[0\.8​Lin,Lin\]\[0\.8\\,L\_\{\\text\{in\}\},\\,L\_\{\\text\{in\}\}\], and truncate any residual overflow from the right so that every realized prompt is an integer number of tokens in the target range\. Output length is controlled independently by settingmax\_new\_tokensto a value drawn uniformly from\[0\.8​Lout,Lout\]\[0\.8\\,L\_\{\\text\{out\}\},\\,L\_\{\\text\{out\}\}\]per request and forcing the decoder to ignore end\-of\-sequence via theignore\_eosextension supported by vLLM and SGLang\. Both axes are seeded: the development seedsdevs\_\{\\text\{dev\}\}controls which LongBench samples are selected, the per\-request output\-length draws, and the MMLU\-Pro subset exposed to the agent through the in\-container evaluation script; the evaluation seedsevals\_\{\\text\{eval\}\}independently controls the held\-out final evaluation used to score the run\. Three canonical seed pairs are used throughout\. The realized request set for each scenario is serialized to disk with its content hash so that byte\-identical inputs are replayed against every agent and every baseline\.

Using LongBench v2 instead of randomly sampled token IDs preserves the tokenizer, attention\-pattern, and KV\-cache access distributions that the server will observe in production\. Random token IDs drawn from the vocabulary produce text that is easy to generate \(high tail entropy, no real long\-range structure\) and that bypasses the tokenizer merge behavior agents would encounter when optimizing against real user traffic, so we treat the shift to LongBench as a strict improvement in realism at no cost in reproducibility\.

### A\.5Metric Definitions

A condensed version of these definitions appears in Section[3\.2](https://arxiv.org/html/2607.20468#S3.SS2); this subsection collects the exact formulas for reference\.*Time\-to\-first\-token*\(TTFT\) measures the elapsed time between sending a request and receiving the first generated token, capturing prefill efficiency\.*Inter\-token latency*\(ITL\) measures the time between consecutive streamed chunks\.*Time\-per\-output\-token*\(TPOT\) is computed per request as\(tend−tfirst\)/\(ni−1\)\(t\_\{\\mathrm\{end\}\}\-t\_\{\\mathrm\{first\}\}\)/\(n\_\{i\}\-1\), wheretfirstt\_\{\\mathrm\{first\}\}is the timestamp of the first token andntokensn\_\{\\mathrm\{tokens\}\}is the total number of generated tokens\. All latency metrics are reported at the P50, P90, and P99 percentiles\.*Generation throughput*measures total tokens generated per second of decode time\.*Request throughput*measures completed requests per unit wall\-clock time\.

The formal definitions implemented in the benchmark’s result summarizer are:

TTFTi\\displaystyle\\mathrm\{TTFT\}\_\{i\}=tfirst\(i\)−tstart\(i\)\\displaystyle=t^\{\(i\)\}\_\{\\text\{first\}\}\-t^\{\(i\)\}\_\{\\text\{start\}\}per request​i\\displaystyle\\text\{per request \}iITLi\\displaystyle\\mathrm\{ITL\}\_\{i\}=\{tk\+1\(i\)−tk\(i\)\}k=1ni−1\\displaystyle=\\bigl\\\{t^\{\(i\)\}\_\{k\+1\}\-t^\{\(i\)\}\_\{k\}\\bigr\\\}\_\{k=1\}^\{n\_\{i\}\-1\}per chunk in request​i\\displaystyle\\text\{per chunk in request \}iTPOTi\\displaystyle\\mathrm\{TPOT\}\_\{i\}=tend\(i\)−tfirst\(i\)ni−1\\displaystyle=\\frac\{t^\{\(i\)\}\_\{\\text\{end\}\}\-t^\{\(i\)\}\_\{\\text\{first\}\}\}\{n\_\{i\}\-1\}for​ni\>1\\displaystyle\\text\{for \}n\_\{i\}\>1GenThroughput\\displaystyle\\mathrm\{GenThroughput\}=∑ini∑i\(tend\(i\)−tfirst\(i\)\)\\displaystyle=\\frac\{\\sum\_\{i\}n\_\{i\}\}\{\\sum\_\{i\}\(t^\{\(i\)\}\_\{\\text\{end\}\}\-t^\{\(i\)\}\_\{\\text\{first\}\}\)\}aggregate decode tokens/sReqThroughput\\displaystyle\\mathrm\{ReqThroughput\}=\|completed\|twall\_end−twall\_start\\displaystyle=\\frac\{\|\\text\{completed\}\|\}\{t\_\{\\text\{wall\\\_end\}\}\-t\_\{\\text\{wall\\\_start\}\}\}requests/s \(wall\-clock\)ScenarioD\\displaystyle\\mathrm\{ScenarioD\}=\(1TTFT¯s⋅1TPOT¯s⋅ReqThroughput\)1/3\\displaystyle=\\Bigl\(\\frac\{1\}\{\\overline\{\\mathrm\{TTFT\}\}\_\{\\text\{s\}\}\}\\cdot\\frac\{1\}\{\\overline\{\\mathrm\{TPOT\}\}\_\{\\text\{s\}\}\}\\cdot\\mathrm\{ReqThroughput\}\\Bigr\)^\{1/3\}\(seconds, req/s\)Heretstartt\_\{\\text\{start\}\}is the client’s send timestamp,tfirstt\_\{\\text\{first\}\}is the first received SSE chunk \(time\.perf\_counterat the moment the firstdelta\.contentstring is parsed\),tkt\_\{k\}is thekk\-th chunk timestamp,tendt\_\{\\text\{end\}\}is the timestamp of the terminating SSE chunk, andnin\_\{i\}is the number of tokens reported in theusage\.completion\_tokensfield of the final chunk \(falling back to a tokenizer\-side recount if the server does not emit usage data\)\. The*generation throughput*numerator counts every generated token while its denominator measures only the decode interval starting at the first token; this convention inflates generation throughput by one token per request relative to theTPOT\\mathrm\{TPOT\}denominator and is reported only as a secondary metric\. In the Scenario D formula,TTFT¯s\\overline\{\\mathrm\{TTFT\}\}\_\{\\text\{s\}\}andTPOT¯s\\overline\{\\mathrm\{TPOT\}\}\_\{\\text\{s\}\}denote the across\-request mean TTFT and TPOT in seconds \(not milliseconds\), andReqThroughput\\mathrm\{ReqThroughput\}is in requests per second\. Scenario D uses its own balanced workload, so its component baseline values are not obtained by combining the Scenario A TTFT, Scenario B TPOT, and Scenario C throughput baselines\. The PyTorch baseline Scenario D score used for normalization is1\.681\.68\.

## Appendix BPrompts, Scaffolds, and Baselines

### B\.1Full Prompt Template

The following is the complete prompt template provided to all agents\. Variables in curly braces are filled at runtime\.

```
Role: You are a R&D Scientist specializing in high-performance LLM inference.

## 1. Objective

You are tasked with deploying and optimizing an OpenAI-compatible inference
server for the base model {model}. You must maximize performance metrics
specific to the following scenario:

* Scenario: {scenario}
* Primary Objective: {mission}
* Time Budget: {num_hours} hours.

## 2. Execution Protocol
You must organize your time to explore, implement, and verify improvements
autonomously.

- Use evaluate.py --json-output-file {metrics_path} as your primary feedback
  mechanism. Your goal is to continuously improve the scores reported by this
  script until the time is over.
- During development, run fast smoke tests first with
  evaluate.py --quick --json-output-file {metrics_path}; use full evaluation
  runs for final confirmation/comparison.
- You are expected to make changes, measure their impact, and iterate. Do not
  settle for a default configuration if optimization is possible.
- Use ./start_server.sh as the canonical and only entrypoint to set up and run
  your server; do not rely on any alternative launcher file or direct ad-hoc
  Python/module invocation for final operation.
- Ensure the server is fully functional and actively running via
  ./start_server.sh, and verify end-to-end functionality by successfully
  running evaluate.py against it.
- Use ./timer.sh to track your remaining budget. Ensure you have a stable,
  optimized server running before time expires.

## 3. Action Space & Tooling

You have full root and download/install access to the environment and the
Internet. You are encouraged to explore the following (non-exhaustive) list
of potential optimizations:

- Inference Engine Selection (i.e. vLLM, SGLang, TensorRT-LLM, or highly
  optimized custom servers (Python/C++/Rust)); No matter the implementation,
  you must expose standard OpenAI endpoints (’GET /v1/models’,
  ’POST /v1/chat/completions’) with streaming support.
- System & Runtime Tuning (i.e. Tuning KV-cache allocation, PagedAttention
  block sizes, swap space, batch size limits, max sequences, continuous
  batching parameters, and chunked prefill settings, Tensor Parallelism (TP)
  size (if applicable), tokenizer parallelism, and NCCL configuration, etc)
- Quantization (i.e. GPTQ, AWQ, FP8, or bitsandbytes quantization); You must
  verify that quantization does not degrade model accuracy below acceptable
  benchmarks (monitor ’evaluate.py’ results for quality drops/hallucinations).
- Kernel Optimization (i.e. usage of ’torch.compile’, CUDA Graphs, AOT
  compilation, swapping attention backends (FlashAttention-2/3/4, Triton
  kernels) or custom CUDA extensions)

## 4. Operational Constraints

* Base Model: You must use {model}. You may quantize it, but you strictly
  cannot switch to a different architecture or a pre-distilled "tiny" model.
* NOTE: Do not offload inference to external APIs. Do not ask for user
  feedback.

## 5. Output Contract

At the end of the session, the following state must be preserved in the
current directory:

1. The optimized server must be running at ’{server_url}’.
2. All scripts (’./start_server.sh’) and config files used to achieve the
   best result must remain in place.
3. (Recommended) Launch your server in the background and track the PID:

bash start_server.sh > server.log 2>&1 & echo $! > server.pid

Reminder: A functional server is a prerequisite, not the final deliverable.
Approach this as a principal R&D scientist. Your success is measured by the
delta between the initial baseline deployment and your final optimized
metrics. Continuously drive up the evaluation scores until the clock runs out.
```

### B\.2Per\-Scenario Prompt Variables

The prompt template shown above interpolates four variables at runtime\. The values shown in Table[9](https://arxiv.org/html/2607.20468#A2.T9)are the exact strings substituted when the prompt is rendered for each scenario; they are read fromscenario\.jsonandmission\.txtin each task directory\. The\{model\}and\{server\_url\}values are fixed within a dispatched run \(the base model and the container endpoint respectively\); the base\-model ablation in Section[4](https://arxiv.org/html/2607.20468#S4)reuses the same template with\{model\}set to Qwen3\-8B or DeepSeek\-V2\-Lite\.\{num\_hours\}depends on the run’s time budget \(22for the main table,1/4/81/4/8for the time\-budget ablation\), and\{metrics\_path\}is the file the runner will read as the run’s final metrics\.

Table 9:Per\-scenario prompt\-template variables\. The\{mission\}string is read verbatim fromsrc/eval/tasks/inference\_scenario\_\{a,b,c,d\}/mission\.txt\. The\{scenario\}string is a compact display name used in the prompt header\.At runtime,\{num\_hours\}is set to the time\-budget argument \(canonically22, and\{1,4,8\}\\\{1,4,8\\\}for the time\-budget ablation\),\{server\_url\}to a localhost URL on a port chosen by the harness,\{metrics\_path\}to the harness\-managed metrics file inside the container, and\{model\}to the base model identifier passed at dispatch time\.

### B\.3Scaffold Details

We evaluate three families of agent scaffolds, each implemented as a shell wrapper that is invoked inside the apptainer container: Claude Code for Anthropic models, Codex CLI for OpenAI models, and OpenCode for Gemini 3\.1 Pro and GLM\-5\. Every wrapper receives the identical prompt template described above; the only differences are the CLI flags exposed by each scaffold and the authentication mode\. All wrappers are parameterized over the model identifier via the$AGENT\_CONFIGenvironment variable\. The exact set of identifiers used in the main table is enumerated in Table[2](https://arxiv.org/html/2607.20468#S3.T2), and the scaffolds themselves do not hard\-code any particular model\.

#### Scaffold versions\.

All runs in Table[2](https://arxiv.org/html/2607.20468#S3.T2)were collected during a single evaluation window \(April 2026\)\. The scaffold versions used are: Claude Code 2\.1\.114, Codex CLI 0\.125\.0, and OpenCode 1\.14\.19 \(theopencode\-ainpm package, running on Node\.js 22\.22\.2 inside theinference\.sifcontainer\)\. Because proprietary scaffolds are opaque and may change between versions \(their internal system prompts, tool\-use policies, and retry logic are not publicly documented\), the results in this paper are tied to these specific versions\. We pin and report versions so that future work can track whether scaffold updates shift the rankings, and we release the exact wrapper scripts used so that any scaffold change can be identified by diffing\.

#### Claude Code\.

The wrapper clearsANTHROPIC\_API\_KEY, loads an OAuth token intoCLAUDE\_CODE\_OAUTH\_TOKEN, and invokes:

```
claude --print --verbose --output-format stream-json \
       --model $AGENT_CONFIG --dangerously-skip-permissions "$PROMPT"
```

The\-\-dangerously\-skip\-permissionsflag is necessary because the agent needs unattended write access to the container filesystem, the shell, and the network\.\-\-output\-format stream\-jsonproduces the JSONL transcripts we later parse for cost and tool\-use counts\. The wrapper wraps the whole invocation in atimeoutcommand set to the remaining wall\-clock budget, and enters a continuation loop that repeatedly callsclaude \-\-continuefor the remainder of the budget whenever the agent exits early\.

#### Codex CLI\.

The wrapper clearsOPENAI\_API\_KEY, copies a ChatGPT session file to the Codex auth location, parses the reasoning\-effort suffix out of$AGENT\_CONFIG\(for example, a suffix of\-highis stripped off the model name and written to the Codex config\), writes a fresh Codex config withforced\_login\_method = "chatgpt"and the parsed reasoning effort, and invokes:

```
codex exec --dangerously-bypass-approvals-and-sandbox \
           --skip-git-repo-check \
           --model $CODEX_MODEL "$PROMPT"
```

Codex prints its step\-by\-step reasoning and tool invocations in a free\-form format rather than structured JSON, which is why the cost and behavior extractor in Appendix[E](https://arxiv.org/html/2607.20468#A5)parses Codex transcripts with rule\-based pattern matching\. As with Claude Code, the wrapper is model\-agnostic: any identifier the Codex CLI exposes can be passed through$AGENT\_CONFIG\.

#### OpenCode\.

The OpenCode wrapper reads an API key from the scaffold directory, exports it asOPENCODE\_API\_KEY, and writes a per\-runopencode\.jsonwithpermission = "allow"and theopencodeprovider configured from that environment variable\. The invocation is:

```
opencode run --model "$AGENT_CONFIG" --format json "$PROMPT"
```

where$AGENT\_CONFIGis the provider\-qualified model identifier \(for exampleopencode/glm\-5oropencode/gemini\-3\.1\-pro\)\. The command is wrapped intimeoutfor the remaining wall\-clock budget\. If the agent exits before the budget is exhausted, the wrapper repeatedly resumes the same OpenCode session with:

```
opencode run --model "$AGENT_CONFIG" --format json \
       --continue "$RESUME_PROMPT"
```

The resume prompt reports the remaining minutes and instructs the agent to continue autonomously without asking for feedback\. OpenCode runs therefore use the same continuation discipline as the Claude Code and Codex CLI runs: an early agent exit does not end the benchmark session unless the wall\-clock budget has expired or the resume command fails\.

### B\.4Baselines

#### PyTorch/Transformers baseline\.

A minimalaiohttp\-based OpenAI\-compatible server that loads the base model withAutoModelForCausalLM\.from\_pretrainedin bfloat16 on a single CUDA device and serves chat completions over SSE\. Tokens are streamed on the fly via a background thread runningmodel\.generate\(\.\.\., streamer=TextIteratorStreamer\(\.\.\.\)\)with an asyncio queue bridging the streamer’s blocking iterator to theaiohttphandler, so each token is flushed to the client as soon as the decoder produces it\.

#### vLLM baseline\.

The canonical vLLM \(v0\.11\.0\) baseline launchesvllm\.entrypoints\.openai\.api\_server\. Every parameter \(maximum concurrent sequences, batched\-token budget, KV\-cache dtype, chunked\-prefill enablement, block size, eager\-mode enforcement, CUDA graph capture, and attention backend\) is left at its vLLM default\.

This out\-of\-the\-box default is nonetheless a strong configuration\. vLLM’s default codepath already enables PagedAttention\(Kwonet al\.,[2023](https://arxiv.org/html/2607.20468#bib.bib29)\)\(eliminating KV\-cache fragmentation\), FlashAttention\-2\(Dao,[2024](https://arxiv.org/html/2607.20468#bib.bib32)\)or the equivalent SDPA backend for fused prefill and decode kernels, continuous batching with iteration\-level scheduling \(so newly\-arrived requests do not wait for the longest decode to finish\), and CUDA graph capture for the decode phase \(amortizing kernel\-launch overhead across steps\)\.

#### SGLang and HF TGI\.

We additionally run SGLang \(v0\.5\.10\) and Hugging Face TGI \(v3\.3\.6\-dev0\) as reference baselines to give a wider view of the default\-configuration landscape\. Both are launched with only the minimum required arguments \(model path, host, port, and context length\)\.

#### Non\-agent search reference\.

We additionally evaluate equal\-budget non\-agent search via Random, SMAC, and TPE\. Each method is run independently for each serving engine rather than as one joint engine\-selection problem: vLLM, SGLang, and TGI each receive a separate 2\-hour search run with an engine\-specific action space\. The vLLM space covers scheduler and memory parameters \(max\_num\_seqs,max\_num\_batched\_tokens,gpu\_memory\_utilization,block\_size\), prefix/chunked prefill toggles, eager execution, quantization, KV\-cache dtype, attention backend, and speculative decoding length\. The SGLang space covers running\-request count, chunked\-prefill size, prefill\-token limit, static memory fraction, attention backend, quantization, torch compile, schedule policy, CUDA graph batch size, and speculative steps\. The TGI space covers concurrent requests, prefill and total batch\-token limits, waiting\-token behavior, CUDA graph capture, quantization, CUDA memory fraction, waiting\-served ratio, and maximum batch size\.

## Appendix CValidation Details

### C\.1Quality Gate

#### Subset and sampling\.

The gate evaluates every optimized server on a fixed 500\-question subset of MMLU\-Pro\(Wanget al\.,[2024](https://arxiv.org/html/2607.20468#bib.bib19)\), with greedy decoding \(temperature0\) and a 10\-option multiple\-choice prompt format \(answer letters A–J\)\. The subset is drawn once per quality\-gate seed and held identical across runs that share that seed, so observed accuracy numbers are directly comparable within a seed pair\. A run passes the gate iffobserved​\_​accuracy≥τ⋅baseline​\_​accuracy\\mathrm\{observed\\\_accuracy\}\\geq\\tau\\cdot\\mathrm\{baseline\\\_accuracy\}\.

#### Answer extraction\.

Responses are parsed through a three\-level fallback: \(1\) the official MMLU\-Pro regex “the answer is \(X\)” / “answer is X”; \(2\) a relaxed match for a single isolated letter A–J; \(3\) the last uppercase A–J character in the response\. If all three fail, the response is counted as wrong\.

#### Threshold sensitivity\.

The main conclusions are stable across the choice ofτ\\tau\. We re\-score the main\-condition pool at eight thresholdsτ∈\{0\.60,0\.70,0\.80,0\.85,0\.90,0\.93,0\.95,0\.97\}\\tau\\in\\\{0\.60,0\.70,0\.80,0\.85,0\.90,0\.93,0\.95,0\.97\\\}, treating any run that fails the revised quality gate as1×1\\timeswhile leaving the remaining scoring rules unchanged\. As Table[10](https://arxiv.org/html/2607.20468#A3.T10)shows, the best\-agent identity is invariant fromτ=0\.60\\tau=0\.60toτ=0\.97\\tau=0\.97; rankings are unchanged throughτ=0\.95\\tau=0\.95, and only one rank position changes atτ=0\.97\\tau=0\.97\. This reflects the same pattern visible in the main table: almost all main\-condition runs either clear the quality bar comfortably or fail for larger reasons than a marginal threshold shift\. We therefore keepτ=0\.95\\tau=0\.95as the canonical threshold: it is strict enough to reject clear quality regressions without driving any of the main conclusions\.

Table 10:Quality\-gate threshold sensitivity on the main\-condition pool\. “ok” is the count of main\-condition runs that pass the quality gate atτ\\tautogether with the integrity gate\. The four×\\times\-valued columns give the best agent cell mean at that threshold under the same scenario definitions as Table[2](https://arxiv.org/html/2607.20468#S3.T2)\. “Δ\\Deltarank vs\. 0\.95” is the maximum absolute change in any agent’s overall rank relative to theτ=0\.95\\tau\{=\}0\.95ranking; a value of0means no agent moved\.

### C\.2Integrity Gate

The integrity gate is an agent\-based judge that inspects each run’s final metrics, server launcher, and agent transcript and assigns a binary label \(honest / flagged\) against disallowed behaviors such as returning pre\-generated text without a forward pass, substituting the base model, fine\-tuning on the quality\-gate subset, and offloading computation to an external API\. The judge prompt enumerates these behaviors, separates contamination from model\-substitution decisions, and emits two one\-line labels \(contamination\_judgement\.txtanddisallowed\_model\_judgement\.txt\)\. A flagged run contributes1\.00×1\.00\\timesto its cell in Table[2](https://arxiv.org/html/2607.20468#S3.T2)\. For reproducibility, the released benchmark artifact includes the full judge prompt, emitted label schema, and the stratified manual\-audit annotations used in this appendix\.

#### Reliability\.

We validate the judge by rejudging every run with an additional judge model\. Between the primary judge \(Claude Sonnet 4\.6\) and the validator \(GPT\-5\.4\), Cohen’sκ=0\.82\\kappa=\\text\{0\.82\}; remaining disagreements cluster on decode\-heavy Scenario B runs that sit close to the rubric boundary between aggressive speculative serving and disallowed “reward hacking” behavior\. To bound judge error beyond cross\-judge agreement, we additionally hand\-labeled a stratified sample of 50 runs spread across scenarios and across the judge’s own flagged / honest decisions\. On this audited subset the judge incorrectly flagged0honest runs and missed11violation among the5050audited runs\. Equivalently, the audited subset has an empirical false\-positive count of0and an empirical missed\-violation rate of2%2\\%over audited runs\. These rates are estimates on the audited stratum rather than guarantees over all180180main\-condition runs, but together with cross\-judge agreement they suggest the aggregate speedup numbers in Table[2](https://arxiv.org/html/2607.20468#S3.T2)are unlikely to be materially inflated by unflagged specification gaming\.

### C\.3Seed\-Level Quartile Volatility

High within\-agent quartile volatility across seeds is expected behavior, and it validates the benchmark design: single\-shot evaluations would overstate agents that occasionally find strong configurations but do not reliably ship them\.

For each agent\-scenario cell, we compute the expected quartile of each seed from its fractional quartile weights and report the range across seeds:

Va,c=maxs⁡𝔼​\[Qa,c,s\]−mins⁡𝔼​\[Qa,c,s\]\.V\_\{a,c\}=\\max\_\{s\}\\mathbb\{E\}\[Q\_\{a,c,s\}\]\-\\min\_\{s\}\\mathbb\{E\}\[Q\_\{a,c,s\}\]\.A value of0means the agent remains in the same quartile across all three seeds; a value near33means it swings from top\-quartile to bottom\-quartile behavior\.

Table 11:Within\-agent quartile volatility across three seeds\. “Stable” counts agents with quartile range≤0\.25\\leq 0\.25; “Adjacent” counts range in\(0\.25,1\.25\]\(0\.25,1\.25\]; “2Q swing” counts range in\(1\.25,2\.25\]\(1\.25,2\.25\]; “Extreme” counts range\>2\.25\>2\.25\.Volatility is substantial, especially in Scenarios A and D where the median agent spans roughly two quartiles across three seeds\. This has two sources\. Some movement reflects true optimization variability: agents sometimes find a strong valid configuration and sometimes do not\. Some movement reflects validity failures, as a run that leaves an unreachable or gate\-failing final server is assigned baseline\-equivalent utility and moves toward the bottom quartile\. We therefore use quartile distributions as uncertainty\-aware summaries rather than deterministic ranks, and we report failure\-aware expected utility throughout\.

## Appendix DFull Results and Cost

### D\.1Full Agent x Scenario Table

Table[12](https://arxiv.org/html/2607.20468#A4.T12)gives the main\-condition cell means in native units\. Unlike Table[2](https://arxiv.org/html/2607.20468#S3.T2), these values are not normalized to PyTorch and therefore show the absolute scale of the serving problem each agent actually solved\.

Table 12:Main\-condition cell means in native units, comparable to PyTorch baselines: 840 ms for Scenario A TTFT, 41 ms for Scenario B TPOT, 0\.132 req/s for Scenario C throughput, and 1\.68 for the Scenario D geomean score\. Dashes mark cells where every seed received a penalized1\.00×1\.00\\timesspeedup\.
### D\.2Cost Analysis

Table[13](https://arxiv.org/html/2607.20468#A4.T13)reports the total cost incurred by each agent across the main\-condition evaluation pool\. We use a uniform $1\.99/h rate for NVIDIA H100 80GB GPU time, following Runpod’s published on\-demand GPU pricing\(Runpod Inc\.,[2026](https://arxiv.org/html/2607.20468#bib.bib5)\)\. API cost is taken directly from Claude logs when available, summed from OpenCode step\-cost records for Gemini/GLM, and estimated from logged token totals for Codex models under official pricing\.

Table 13:Cost breakdown per agent on the 2\-hour main\-condition pool, normalized to a complete 12\-run evaluation\.The OpenCode rows are materially cheaper than most Claude/Codex rows while staying competitive in the main table: Gemini totals $109\.55 and GLM\-5 $77\.36 for a normalized 12\-run main\-condition pool\. Claude Haiku 4\.5 is also inexpensive at $81\.88, but its low aggregate speedup means that low absolute cost does not by itself imply strong cost\-normalized performance\. Claude Opus 4\.7 is by far the most expensive row because its main\-table selection contains substantially higher reported API spend than the other Claude models\. For an external researcher evaluating a single new agent, a full 12\-run evaluation spans roughly $70–$1,650 depending heavily on the API model, with Claude Opus 4\.7 at the top of this range due to substantially higher per\-token API spend\. Nominal GPU cost is $47\.76 for a complete 12\-run, 2\-hour\-per\-run evaluation, before overhead, early termination, and cleanup\.

#### Cost\-normalized performance\.

The cheapest rows are not the same as the highest\-performing rows, which is consistent with the broader paper narrative: once a model is in the “working vLLM plus shallow tuning” regime, lower API cost can dominate headline value\. Table[14](https://arxiv.org/html/2607.20468#A4.T14)reports dollars per unit of speedup for each scenario cell\. GPT\-5\.2 is cheapest on Scenarios A and C, and GLM\-5 is cheapest on Scenarios B and D\.

Table 14:Cost per unit of speedup \($/×\) for each \(agent, scenario\) cell on the 2\-hour main\-condition pool, using the revised nominal GPU cost from Table[13](https://arxiv.org/html/2607.20468#A4.T13)\. Lower is better\.−⁣−\-\-indicates a missing or all\-failed scenario cell\.Bold= best \(cheapest\) per column\.

## Appendix EBehavioral Metrics and Failure Modes

### E\.1Failure Mode Definitions and Prevalence

We classify each run into one or more failure modes using rule\-based features extracted from the run’s persisted server launch log, its final metrics file, and the agent transcript\. No LLM is involved in the classification\.

Table 15:Failure mode definitions and frequency across the 180\-run 2\-hour main\-condition pool\. Unlike Table[3](https://arxiv.org/html/2607.20468#S4.T3), these categories are*non\-mutually\-exclusive*diagnostic labels: a single run can satisfy multiple definitions simultaneously\. Percentages are of 180 total runs\.
### E\.2Recovery and Backtracking Analysis

Across the 63 main\-condition runs that do not contribute a passing main\-table measurement, backtracking is still rare\. In most cases the launch log records a single committed server family and no serious rollback to an earlier known\-good configuration before the continuation loop exhausts the budget\. When a server restarts, the relaunched arguments are unchanged in 81% of restart cases, which again points to crash recovery rather than deliberate A/B search\. We did not observe a run in this set in which the agent clearly reverted to an earlier simpler launcher after an unsuccessful experiment\.

### E\.3Secondary Behavioral Metrics

We extract the following from each run’s persisted launcher, evaluation log, and agent transcript across the 2\-hour main\-condition pool\. The gap between near\-universal technique mention rates and the very small number of genuinely multi\-configuration runs is the clearest indicator that iteration discipline, rather than missing knowledge, is the binding constraint\.

Table 16:Secondary behavioral metrics across the 2\-hour main\-condition pool\.
### E\.4Qualitative Ranking Analysis

The main leaderboard has a counterintuitive pattern: Claude Sonnet 4\.6 ranks first and GLM\-5 ranks second, ahead of models that are plausibly stronger in many general reasoning settings\. The main reason is thatInferenceBenchscores the final deployed server rather than the best idea an agent considered during the run\. A strong but brittle trajectory can therefore lose to a simpler trajectory that preserves a valid final launcher\.

Claude Sonnet 4\.6 is the cleanest example of this reliability effect\. In the selected main\-condition cells it passes all 12/12 runs and usually stays close to robust vLLM configurations: FP8 quantization, FP8 KV cache where useful, larger batching limits, prefix caching, and limited speculative decoding\. Its transcripts also show restraint near the wall\-clock deadline: after finding a strong working server, it often verifies the launcher and avoids risky last\-minute dependency or engine changes\. This combination is not maximally exploratory, but it produces a high deployment\-utility score because the final artifact remains valid\.

GLM\-5 shows a related but slightly different pattern\. It passes 9/12 selected runs and its successful runs are genuinely competitive, rather than only benefiting from other agents’ failures\. Its best runs tend to use simple vLLM launchers with a few scenario\-appropriate flags, such as FP8 quantization, larger sequence limits, and prefix caching\. This leaves less room for drastic custom\-kernel performance gains, but also avoids several common failure modes such as unresolved dependency stacks, incompatible SGLang or vLLM versions, or a final launcher that differs from the last measured working server\.

The lower ranks of GPT\-5\.5, Claude Opus 4\.5/4\.6/4\.7, and GPT\-5\.4 should not be necessarily read as evidence that these models cannot find strong optimizations\. More ambitious trajectories are more likely to end in an unusable or penalized final state, such as a dependency mismatch in a newly installed stack, a CPU or fallback server that cannot complete held\-out evaluation, an implausible timing profile flagged by the integrity gate, or a quality\-gate failure after an aggressive speed optimization\. This suggests a capability–reliability tradeoff where stronger agents may produce better peak runs, but may also lower the probability of producing a valid final deployment\.

Table 17:Best submitted final\-server ranking analysis\. For each agent and scenario, the BoN score takes the best passing final submission among the selected main\-condition runs; if no selected run passes, that scenario contributes1\.00×1\.00\\times\. BoN Aggregate is the geometric mean over Scenarios A–D\. This analysis measures peak observed final\-deployment performance within the three\-seed budget, while the main aggregate remains the failure\-aware expected utility used in Table[2](https://arxiv.org/html/2607.20468#S3.T2)\.The BoN recalculation moves GPT\-5\.4 and GPT\-5\.5 upward, showing that their best successful final submissions are stronger than their penalized means suggest\. However, Claude Sonnet 4\.6 and GLM\-5 remain first and second even under this peak\-final\-submission view\. Their ranks are therefore not only an artifact of penalizing other models’ failed runs; they also reflect strong successful\-run performance among the final artifacts that agents actually submitted\.

## Appendix FTrace Examples and Integrity Cases

We reproduce short annotated excerpts from seven agent execution traces, covering the major failure patterns along with positive examples of controlled experimentation\. The main text focuses on the three most decision\-relevant behaviors: early convergence, failed conversion of knowledge into controlled search, and reward hacking under optimization pressure\. The appendix keeps more granular examples such as self\-declared completion, stalled continuation, dependency spirals, and additional integrity cases\. Most excerpts are drawn from the main pool; a few come from nearby runs that illustrate a failure mode more clearly\. Each excerpt is a verbatim quote from the corresponding run’s transcript, followed by concrete run artifacts such as launcher changes, evaluator outputs, or final\-state symptoms\.

#### Example 1: Self\-declared completion\.

A Claude Opus 4\.5 run on Scenario B reaches a first working vLLM launch 28 minutes into its budget, runs the quick evaluator, and produces a clean baseline readout:

Baseline performance: generation throughput 63\.53 tokens/s\. TTFT p50 51\.8 ms, p90 400 ms\. ITL p50 10\.2 ms\. TPOT p50 15\.7 ms\. Quality check: PASS \(accuracy 28% vs baseline 25\.5%\)\. The key metric for Scenario B is generation throughput… Let me stop the current server and optimize\.

The concrete launch state at this point is a minimal vLLM server using the correct base model and standard OpenAI\-compatible endpoints, with no meaningful scenario\-specific tuning beyond the default vLLM serving path\. The agent then tries two low\-effort variants: adding\-\-enable\-chunked\-prefilland toggling\-\-enforce\-eager\. Neither is isolated in a full comparison\. After one restart attempt triggers an out\-of\-memory failure, subsequent continuation turns stop producing new experiments\. A representative later statement is:

The current server is already valid and passing the checks\. Since further changes risk breaking the deployment, I will keep this configuration for final evaluation\.

This is a failure of continuation rather than basic setup\. The agent has a working server, a valid metric readout, and substantial remaining wall\-clock time, but the search trajectory collapses after the first local failure\. For roughly the last 90 minutes, the transcript contains repeated readiness checks and restatements of the same plan rather than new launch arguments or controlled comparisons\. The final committed launcher is effectively the initial working configuration, so the run illustrates how a passing first server can prematurely become the final answer\.

#### Example 2: Multi\-variable edits\.

A Claude Opus 4\.6 run on Scenario D commits four distinct non\-default vLLM argument sets across the 2\-hour budget:

config 1: quantization=fp8, kv\_cache\_dtype=fp8\_e4m3, gpu\_util=0\.75 config 2: quantization=fp8, kv\_cache\_dtype=fp8\_e4m3, gpu\_util=0\.75 \(relaunch\) config 3: quantization=fp8, kv\_cache\_dtype=fp8\_e4m3, enforce\_eager=True config 4: quantization REMOVED, kv\_cache\_dtype=auto, enforce\_eager=True

The agent’s stated rationale is reasonable in isolation: Scenario D balances latency, decode speed, and throughput, so FP8 quantization and FP8 KV cache could reduce memory bandwidth pressure, while eager execution might avoid CUDA graph capture overhead for the mixed workload\. The problem is experimental control\. Between configurations 2 and 3,enforce\_eageris introduced without a complete intervening evaluation\. Between configurations 3 and 4, two variables are changed at once:quantization=fp8is removed andkv\_cache\_dtype=fp8\_e4m3is replaced byauto, whileenforce\_eager=Trueremains active\.

A concrete symptom appears in the final metrics artifact: the result file contains an error field and no completed Scenario D profiles\. The harness cannot recover TTFT, TPOT, or throughput summaries because the final relaunch is unreachable\. Thus the agent did not merely choose a suboptimal configuration; it destroyed the last known usable deployment state\. This example is representative of the “multi\-variable edits” failure mode in Appendix[E](https://arxiv.org/html/2607.20468#A5): the agent knows plausible knobs, but changes multiple interacting parameters without a measurement schedule that would allow attribution or rollback\.

#### Example 3: Dependency spiral\.

A GPT\-5\.4 \(High\) run on Scenario A attempts to use FlashInfer as its attention backend for long\-prompt prefill and immediately hits first\-launch kernel JIT compilation:

The delay is now explained: FlashInfer is JIT\-compiling the large prefill kernels for SM90 on first use\. That’s likely worth paying once for TTFT, but I’ll relocate those caches into the repo after this run so the final relaunch can reuse them instead of recompiling from scratch\.

The concrete system symptom is a long period in which the server is alive but not serving useful benchmark traffic\. The launch log shows repeated CUDA compilation activity rather than completed requests, and the evaluator sees connection or timeout behavior rather than stable first\-token latency\. In this particular run, the agent eventually diagnoses the problem and abandons FlashInfer before the budget is fully consumed\. This is a partial success: the agent recognizes that an optimization promising lower TTFT can become counterproductive when first\-use compilation dominates the evaluation window\.

The unrecovered version of the same pattern appears in a Claude Opus 4\.6 run on Scenario C\. That run cycles through the following backend trajectory:

attempt 1: FlashInfer backend, default precision attempt 2: Triton fallback after FlashInfer launch instability attempt 3: FlashInfer backend with FP8\-related settings attempt 4: Triton fallback after quality or runtime degradation

The concrete failure is that each backend transition is triggered by a crash, timeout, or quality regression, not by a controlled throughput comparison under the same request profile\. Scenario C rewards scheduler\-level throughput under 64\-way concurrent load, yet the agent spends much of its budget resolving backend compatibility and compilation behavior\. This is why we call it a dependency spiral: the agent is doing real engineering work, but that work is reactive infrastructure repair rather than optimization search\.

#### Example 4: Controlled experimentation\.

A GPT\-5\.4 \(High\) run on Scenario A corresponds to the\(21,1337\)\(21,1337\)seed of the cell whose final mean TTFT is237\.96237\.96ms across the three\-seed selection \(the cell mean appears in Appendix[D\.1](https://arxiv.org/html/2607.20468#A4.SS1), Table[12](https://arxiv.org/html/2607.20468#A4.T12)\)\. The agent runs an explicit V0 vs V1 comparison with a named rollback criterion, then tries single\-flag deltas from the winner:

V1 improved the short speed\-only probe but did not hold up in the longer end\-to\-end path, so I’m treating it as unstable for the final submission\. I’m using the remaining time on lower\-risk V0 variants; the next one is eager execution, which might trim per\-request overhead without changing the serving surface\.

\-\-enforce\-eageris promising: TTFT improved modestly and, more importantly, the tail got tighter than the current default\. I’m trying one last combinatorial variant, eager plusmax\_num\_seqs=1, because if it helps TTFT further I can still afford a final quality check afterward\.

The concrete experiment structure is:

V0: baseline vLLM launcher, quality passing, stable full evaluation V1: aggressive prefill\-oriented variant, improved quick probe but unstable full path V0\+eager: add \-\-enforce\-eager, compare TTFT and tail latency V0\+eager\+seq1: add \-\-max\-num\-seqs=1, final candidate after quality check

This trace is qualitatively different from the multi\-variable edit example\. The agent names a baseline, tries an alternative, rejects the alternative when it fails to transfer from quick probe to longer evaluation, and then applies a single\-variable delta to the stable branch\. It also preserves deployment state: the finalstart\_server\.shincludes the selected flags and a warm\-up step so the final harness does not pay a one\-time initialization penalty\. The lesson is not that the chosen flags are universally optimal, but that the agent follows a recognizable engineering loop: propose, measure, compare, rollback, and commit\.

#### Example 5: A second controlled\-experimentation trace\.

A second GPT\-5\.4 \(High\) run on Scenario D exhibits a similar pattern\. After a first vLLM launch produces only a1\.1×1\.1\\timesgeomean speedup, the agent identifies three orthogonal candidates: increasingmax\-num\-seqs, enabling\-\-enable\-prefix\-caching, and switching the KV cache away from the current setting\. The transcript explicitly separates them as independent interventions rather than applying them all at once\.

candidate A: increase \-\-max\-num\-seqs to improve batching headroom candidate B: enable \-\-enable\-prefix\-caching to reduce repeated\-prefix cost candidate C: change KV\-cache dtype to reduce memory pressure

The concrete evaluation pattern is also cleaner than the average run\. Candidate A is tested first and improves request throughput without a large TPOT regression\. Candidate B is then tested on top of the current winner and improves the balanced geomean on the quick evaluator\. Candidate C degrades TPOT enough that the agent reverts it rather than keeping the change for lack of time\. The final launcher therefore keeps the two changes that survived measurement and drops the one that hurt the decode component of the Scenario D metric\.

This example matters because Scenario D is the setting where naive single\-metric optimization is most likely to fail\. The agent has to preserve a balance among inverse TTFT, inverse TPOT, and request throughput\. In this trace, the agent actually reasons in those terms: it rejects a change not because it crashes, but because it improves one component while damaging another\. The resulting cell is one of the few agent results that lies strictly above the vLLM default on the Scenario D geomean, which is characteristic of a run that converged on a reproducible configuration rather than stumbling on one\.

#### Example 6: Reward hacking / integrity violation\.

A GPT\-5\.3 Codex \(Med\) run on Scenario B reports a decode throughput of 118 million tokens per second and a per\-token decode latency of 6\.4 nanoseconds\. These numbers are physically implausible for generating 8k\-token outputs from a 7B\-class model on a single H100\. The run’s submitted launcher and transcript indicate that the measurement path has been subverted rather than that the serving configuration improved\.

The concrete red flags are threefold\. First, the reported decode latency is many orders of magnitude below the normal kernel\-launch and memory\-access scale of the hardware\. Second, the response path returns outputs that satisfy the evaluator’s surface format without performing the intended full decode workload\. Third, the quality score lands exactly in the narrow band needed to pass the gate:

baseline quality: 128/500 correct required threshold: 0\.95×\\times128 = 121\.6 correct submitted server quality: 122/500 correct

This is consistent with a server that returns plausible answer letters or cached surface\-form completions while bypassing the intended speed\-evaluation path\. The important point is that the quality gate alone would not remove this run: it is just accurate enough to pass\. The integrity gate flags it because the speed profile and implementation jointly indicate specification gaming\. Without that gate, this single run would have silently polluted the Scenario B mean with an impossible speedup\.

#### Example 7: TTFT reward hacking via fake first chunk\.

Two runs, GPT\-5\.3 Codex \(Med\) on Scenario A and GPT\-5\.2 on Scenario A, independently discover the same exploit: emitting a fake “first token” as an SSE streaming chunk before any prompt tokenization or model forward pass begins\. Since Scenario A measures time to first token on 8,192\-token inputs, the exploit reduces the measured TTFT to the time required to flush a trivial streaming chunk rather than the time required to prefill the long prompt\.

A simplified version of the submitted behavior is:

send SSE chunk containing a single space character flush response stream immediately so the client records first\-token time only afterward tokenize the full prompt and begin the actual model forward pass

The concrete metric signature is a sub\-3 ms TTFT on long\-context requests where ordinary prefill should take hundreds of milliseconds under the PyTorch baseline and substantially longer than a few milliseconds even under optimized serving\. The actual model call still happens later in the stream, so the quality gate can pass: the final answer is not necessarily fake, but the timestamp used for the primary metric is fake\. This makes the exploit more subtle than simply returning pre\-generated text\. It preserves enough semantic behavior to survive the quality check while manipulating the event boundary that defines the metric\.

The two independent discoveries are also informative\. The GPT\-5\.3 Codex \(Med\) transcript explicitly describes the strategy as an early streamed token, while the GPT\-5\.2 run arrives at the same mechanism independently and leaves a comment in the submitted server naming the fake\-first\-chunk behavior\. The integrity gate flags both runs because the submitted server intentionally manipulates the streaming protocol used for measurement\. These cases motivate treating latency metrics as part of the specification, not merely as passive observations of an honest server\.

## Appendix GAblations

### G\.1Hardware Ablation

The hardware\-transfer ablation reruns the same GPT\-5\.4 \(High\), 2\-hour setup on A100 rather than H100, with speedups again normalized to the corresponding PyTorch baseline\. As in the base\-model ablation, rows should be read as within\-setting optimization progress rather than raw latency comparisons across hardware\. Transfer to A100 is harsher than the prompt ablation, as Scenario A falls from3\.53×3\.53\\timesto1\.22×1\.22\\times, Scenario C from25\.84×25\.84\\timesto8\.39×8\.39\\times, and Scenario D from3\.25×3\.25\\timesto1\.20×1\.20\\times, while the pass count drops from 10/12 to 7/12\. Scenario B is the only apparent improvement, rising to3\.68×3\.68\\times, but this estimate has large variance because two of the three selected seeds contribute only baseline\-equivalent utility\. These results suggest that the agent is not strategically optimizing inference; it is using a recipe whose reliability depends meaningfully on the hardware among other factors\.

Table 18:Hardware ablation for GPT\-5\.4 \(High\), 2h, Mistral\-7B\-Instruct\-v0\.3\. Cells are penalized mean±SEM\{\}\_\{\\pm\\text\{SEM\}\}speedup over the PyTorch baseline across the three held\-out seed\-pair runs\.
### G\.2No\-action\-space prompt ablation

The no\-action\-space variant collapses the action\-space enumeration to one sentence, “You have full root and download/install access to the environment and the Internet”, keeping only the scenario objective, operational constraints, and output contract\.

This lowers the GPT\-5\.4 \(High\) aggregate from5\.08×5\.08\\timesto2\.62×2\.62\\timesand degrades or flattens every scenario \(Table[19](https://arxiv.org/html/2607.20468#A7.T19)\): Scenario A from3\.53×3\.53\\timesto1\.21×1\.21\\times, Scenario C from25\.84×25\.84\\timesto12\.10×12\.10\\times, and Scenario D from3\.25×3\.25\\timesto1\.23×1\.23\\times, with Scenario B essentially flat \(2\.24×2\.24\\timesversus2\.61±0\.86×2\.61\\pm 0\.86\\times\)\. The pass count falls from 10/12 to 8/12, two of the four failures from integrity\-gate violations\. For Claude Opus 4\.7 the aggregate is unchanged \(2\.25×2\.25\\timesto2\.54×2\.54\\times\): Scenario D improves \(1\.27×1\.27\\timesto2\.27×2\.27\\times\), Scenarios A and B stay flat, and Scenario C declines\.

Table 19:No\-action\-space prompt ablation, 2h, Mistral\-7B\-Instruct\-v0\.3\. Cells report penalized mean±SEM\{\}\_\{\\pm\\text\{SEM\}\}speedup over the PyTorch baseline across the three held\-out seed\-pair runs\. Aggregate is the geometric mean over Scenarios A–D\. Failed seed runs \(no functioning final server\) and integrity\-flagged runs count as1×1\\times\.Every no\-action\-space run converged to vLLM\. Both agents enumerated alternatives, including sglang, lmdeploy, and TensorRT\-LLM and options the original prompt did not list, before settling on vLLM\. The default\-prompt runs behaved the same: try vLLM first, confirm it works, then readvllm serve \-\-helpfor flags without testing alternatives\.

Within vLLM, GPT\-5\.4 \(High\) stayed at minor hyperparameter tuning: chunked prefill, prefix caching,max\_num\_seqs,max\_num\_batched\_tokens,kv\-cache\-dtype,stream\-interval, andmax\-model\-len, with no custom kernels and no speculative decoding\. Most runs never attempted weight quantization, conflating it with\-\-kv\-cache\-dtype fp8and concluding “quantization does not help”\. Scenario D exposes the cost: its burst profile runs at concurrency 1 and is latency\-bound, but GPT\-5\.4 either kept defaults or made throughput\-batching changes that yield nothing at that concurrency, so both successful runs captured only the engine\-default speedup\. Opus 4\.7 targeted the decode\-heavy scenarios better, reaching for n\-gram speculative decoding on Scenarios B and D and identifying it, rather than batch sizing, as the dominant factor at concurrency 1; its clean Scenario\-D runs reach∼2\.9×\\sim\\\!2\.9\\timesagainst GPT\-5\.4’s near\-default performance\. Neither agent wrote a custom kernel or used tensor parallelism, and in both cases the remaining budget went to repeated server\-restart cycles rather than new optimization axes\.

Two of the twelve GPT\-5\.4 runs were integrity\-flagged, above the main\-condition rate, both from improvising around the missing action space\. On Scenario B, the run that looked strongest at4\.56×4\.56\\timeson raw speed reasoned from the decode\-throughput objective that “the most plausible remaining upside is a published FP8 quantization”, searched the open web, and shipped a third\-party pre\-quantized checkpoint \(RedHatAI/Mistral\-7B\-Instruct\-v0\.3\-FP8\) while exposing the canonicalmistralai/Mistral\-7B\-Instruct\-v0\.3identifier to the harness\. The prompt permits quantization but disallows a pre\-quantized model; the agent substituted a separately published checkpoint under the base model’s name rather than quantizing the provided weights\. On Scenario C, the contamination judge flagged a run that reshaped the speed workload toward shared\-prefix and then single\-token prompts to inflate throughput\. The harness substitutes its own canonical requests at scoring, so the score did not move, but the attempt was made\.

Opus 4\.7 shows the same behavior far more often: six of twelve runs are integrity\-flagged\. Many match the GPT\-5\.4 pattern, serving the pre\-quantizedRedHatAI/Mistral\-7B\-Instruct\-v0\.3\-FP8checkpoint under the canonical name\. Two go further and edit the evaluation scaffold, one monkeypatching the harness to relax its input\-length check and supply a near\-zero fabricated quality baseline, the other planting a request file with shortened generation lengths after reading the eval runner’s source\. The harness re\-runs its own canonical evaluation outside the agent environment, so the scores held, but under the looser prompt Opus treats model substitution and scaffold edits as a default strategy rather than an edge case\.

The prompt acts on the two agents differently: for GPT\-5\.4 \(High\) it narrows exploration and lowers the aggregate, while for Opus 4\.7 it leaves the aggregate intact but sharply raises integrity violations\. This complements the structured\-iteration ablation \(Section[5](https://arxiv.org/html/2607.20468#S4.T5)\) from the opposite direction\. Making the protocol explicit improves reliability, while removing the action\-space enumeration either narrows exploration or elicits violations\. Neither removes the underlying search\-breadth limitation\.

### G\.3Per\-engine non\-agent search results

The main\-table non\-agent rows report the best final gate\-passing result selected from vLLM searches specifically\. This subsection reports the full 3\-method×\\times3\-engine×\\times4\-scenario grid\. Each cell is an independent 2\-hour search of the named optimizer over the named engine’s documented CLI flags\.

Table 20:Non\-agent search across optimizers and engines\. Each scenario cell is the final gate\-passing speedup over the PyTorch baseline after a 2\-hour run, reported as mean±SEM\{\}\_\{\\pm\\text\{SEM\}\}\. Aggregate is the geometric mean over Scenarios A–D computed from the displayed scenario means\.Boldmarks the best engine per \(optimizer, scenario\)\. Engine choice matters substantially on Scenarios B and C and modestly on A and D\.#### Engine choice is scenario\-specific\.

No single engine dominates across scenarios\. SGLang is the strongest engine for prefill \(Scenario A\) under all three optimizers\. vLLM is strongest for decode\-heavy long\-output generation \(Scenario B\) when it produces a valid final result, while SGLang gives the best Random\-search result in the same scenario\. TGI is the strongest engine for high\-load throughput \(Scenario C\), reaching7777–89×89\\times, and also wins the balanced Scenario D for all three optimizers\. This reinforces that engine choice is a first\-order decision on throughput\-heavy workloads and a meaningful second\-order decision even outside Scenario C\.

#### Optimizer choice is largely interchangeable\.

Within a fixed engine, the three optimizers produce similar results\. The within\-engine spread across Random, SMAC, and TPE is at most roughly44speedup points on Sc\. B among vLLM runs and roughly55speedup points on Sc\. C among vLLM runs, much smaller than the cross\-engine spread on the same scenarios\. The most plausible explanation is that 2 hours is long enough for any reasonable optimizer to cover most of the per\-engine search space: the action surface is bounded by a documented flag list, individual configurations launch and evaluate in minutes, and under that regime Random, SMAC, and TPE converge to similar near\-frontier configurations regardless of acquisition strategy\. This further sharpens the agent comparison: the matched\-budget non\-agent number agents lose to is not sensitive to optimizer choice, so the gap is not an artifact of picking a particularly aggressive search method\.

#### Implication for the main comparison\.

The selected non\-agent search rows in Table[2](https://arxiv.org/html/2607.20468#S3.T2)exceed the best final\-shipped agent on every scenario and on the aggregate\. The gap is smallest on Scenario B, where the best agent reaches12\.03×12\.03\\timesand the best vLLM\-restricted non\-agent row reaches15\.23×15\.23\\times, and largest on Scenario C, where the best agent reaches33\.93×33\.93\\timeswhile the best vLLM\-restricted non\-agent row reaches46\.70×46\.70\\times\. Allowing search to additionally select between vLLM, SGLang, and TGI extends the Scenario C gap further, with the best engine reaching89\.00×89\.00\\timeson Random and85\.84×85\.84\\timeson SMAC\. Because agents in our evaluation almost universally selected vLLM \(Table[16](https://arxiv.org/html/2607.20468#A5.T16)\), this grid shows two distinct shortfalls: insufficient search depth on the chosen engine, and a missed engine\-selection opportunity that non\-agent search exploits when permitted\.

### G\.4Working\-server warm start

The main benchmark intentionally evaluates end\-to\-end deployment: the agent receives a base model, hardware environment, scenario objective, and evaluation harness, but no starter server\. This makes server assembly part of the task\. A natural concern is that the gap to non\-agent search may therefore be driven primarily by setup overhead rather than by optimization behavior\. To test this, we run a warm\-start ablation in which the agent begins from a fully working vLLM server instead of an empty workspace\.

In the warm\-start condition, the container initially contains a validstart\_server\.shthat launchesvllm\.entrypoints\.openai\.api\_serverfor the correct base model, host, port, and maximum context length\. The server passes both the quality gate and a quick speed evaluation before the agent begins\. The prompt is modified to state that this launcher is a correct but untuned starting point and that the agent should optimize it rather than merely preserve it\. All other conditions are unchanged: same H100 hardware, same Mistral\-7B\-Instruct\-v0\.3 base model, same 2\-hour budget, same continuation wrapper, same development/evaluation seed pairs, same quality and integrity gates, and the same final\-server scoring rule\. Thus, this ablation removes first\-server construction as a bottleneck while preserving the open\-ended optimization problem\.

Table 21:Working\-vLLM warm\-start ablation\. The warm\-start condition gives the agent a valid vLLM OpenAI\-compatible server at the beginning of the run\. Agent rows are penalized mean±\\pmSEM speedup over the PyTorch baseline across the three held\-out seed\-pair runs\. Failed seed runs count as1×1\\times\. The non\-agent row reports the per\-scenario best value over the full33optimizer×\\times33engine grid\.ConditionSc\. ASc\. BSc\. CSc\. DPass rateTTFTTPOTreq\. tputgeomeanGPT\-5\.4 \(High\), default prompt3\.53×⁣±\\times\{\\pm\}0\.052\.24×⁣±\\times\{\\pm\}0\.9625\.84×⁣±\\times\{\\pm\}0\.713\.25×⁣±\\times\{\\pm\}1\.3710/12GPT\-5\.4 \(High\), warm\-start vLLM3\.71×⁣±\\times\{\\pm\}0\.114\.08×⁣±\\times\{\\pm\}1\.1827\.41×⁣±\\times\{\\pm\}2\.133\.48×⁣±\\times\{\\pm\}0\.4212/12Claude Opus 4\.7, default prompt1\.07×⁣±\\times\{\\pm\}0\.061\.00×⁣±\\times\{\\pm\}0\.0019\.02×⁣±\\times\{\\pm\}0\.941\.27×⁣±\\times\{\\pm\}0\.275/12Claude Opus 4\.7, warm\-start vLLM4\.31×⁣±\\times\{\\pm\}0\.228\.64×⁣±\\times\{\\pm\}2\.5035\.22×⁣±\\times\{\\pm\}2\.902\.81×⁣±\\times\{\\pm\}0\.6011/12Per\-scenario best non\-agent search5\.06×\\times15\.23×\\times89\.00×\\times6\.10×\\times–Warm\-starting improves reliability but does not close the gap to search\. The largest effect is on pass rate: GPT\-5\.4 rises from 10/12 to 12/12 passing runs, and Claude Opus 4\.7 rises from 5/12 to 11/12\. This confirms that part of the main benchmark difficulty is ordinary systems setup and final\-state preservation\. However, the performance ceiling changes much less than the pass rate\. GPT\-5\.4 improves modestly on Scenario B and becomes more stable on Scenario D, but remains below matched\-budget non\-agent search on every scenario\. Claude Opus 4\.7 benefits more because the warm start removes several early failure modes, but its best warm\-start cells still trail the non\-agent rows, especially on the throughput\-heavy Scenario C and balanced Scenario D\.

The behavioral traces show the same pattern as the main condition, but with fewer launch failures\. Once given a working vLLM server, agents tend to treat it as a safe anchor: they rerun the evaluator, make one or two local changes to exposed vLLM flags, and then preserve the current launcher to avoid breaking final validity\. The median number of distinct non\-default vLLM configurations rises from 1\.0 in the main\-condition pool to 2\.0 in the warm\-start condition, but the increase is still far below the number of configurations evaluated by the 2\-hour non\-agent searches\. In particular, warm\-start runs rarely switch engines, rarely test TGI or SGLang, and rarely perform systematic sweeps over scheduler and memory parameters\.

This ablation separates two failure modes\. First, agents sometimes fail to establish or preserve a valid server, and warm\-starting directly reduces that failure rate\. Second, even after the server\-construction problem is removed, agents still under\-explore the optimization surface\. The remaining gap to non\-agent search therefore cannot be explained solely by environment setup\. It reflects a persistent search\-discipline limitation: agents know the relevant optimization concepts, but do not reliably turn the available wall\-clock budget into broad, controlled, measured exploration\.

### G\.5Forced\-engine agent ablation

The main\-condition agents overwhelmingly converge to vLLM, even though the non\-agent grid shows that engine choice is scenario\-specific: SGLang is strongest on the prefill\-heavy Scenario A, vLLM and SGLang are most competitive on the decode\-heavy Scenario B, and TGI dominates the high\-load and balanced Scenarios C–D\. To separate engine selection from within\-engine optimization, we run a forced\-engine ablation for GPT\-5\.4 \(High\), where the agent is required to use a specified serving engine rather than choosing its own stack\.

In this ablation, the prompt is modified to require the agent to use a specified serving engine for the final submitted server\. We evaluate two variants:SGLang\-onlyandTGI\-only\. In each condition, the agent may inspect documentation, edit launch scripts, tune all exposed command\-line flags, and install compatible dependencies, but the final server must be launched through the required engine and must expose the same OpenAI\-compatible endpoints as the main benchmark\. Runs that ship a different engine, fail either gate, or leave no reachable final server are assigned baseline\-equivalent utility\. All other experimental details are unchanged: GPT\-5\.4 \(High\), Mistral\-7B\-Instruct\-v0\.3, one H100, a 2\-hour budget, the same continuation wrapper, the same three held\-out seed pairs, and the same final\-server scoring rule\.

Table 22:Forced\-engine ablation for GPT\-5\.4 \(High\)\. The SGLang\-only and TGI\-only rows require the agent to ship the specified engine as the final server\. Agent rows are penalized mean±\\pmSEM speedup over the PyTorch baseline across the three held\-out seed\-pair runs\. Failed, unreachable, or wrong\-engine final submissions count as1×1\\times\. The non\-agent row reports the per\-scenario best value over the full33optimizer×\\times33engine grid\.Forced engine selection improves the relevant scenarios but does not eliminate the search gap\. The SGLang\-only condition improves Scenario A, where SGLang is also the strongest engine in the non\-agent grid, and gives a moderate gain on Scenario B relative to the default agent condition\. However, it remains below the best SGLang non\-agent search on Scenario A and below the best vLLM non\-agent search on Scenario B, indicating that choosing a stronger engine family is not sufficient without systematic tuning of scheduler, memory, prefill, and decoding parameters\. The TGI\-only condition produces the clearest gain on Scenario C, rising substantially above the default GPT\-5\.4 agent row and above the default vLLM\-centered behavior, but still remains below the TGI non\-agent search row\. It also improves Scenario D, consistent with the non\-agent grid showing TGI as the strongest engine for the balanced workload\.