CraftBench-UE：Unreal Engine编码代理的确定性评估

# CraftBench-UE: Deterministic Evaluation for Coding Agents in Unreal Engine
Source: [https://arxiv.org/html/2609.23142](https://arxiv.org/html/2609.23142)
###### Abstract

Building gameplay features in a game engine requires more than code, as code that compiles and runs does not necessarily implement the requested gameplay\. We introduce CraftBench\-UE, an evaluation harness that runs agents in an isolated Unreal Engine environment, reconstructs their saved submissions in fresh projects, and applies deterministic build, asset, and runtime checks without an LLM judge\. Based on the harness, we build a benchmark consisting of 70 tasks spanning C\+\+ source, Blueprint assets, and editor scripting\. We evaluate seven models under two editor\-tool configurations, with a file\-and\-shell baseline on C\+\+ tasks\. We further pair tasks that specify the same gameplay and use the same runtime tests, but require C\+\+ and Blueprint as the deliverables\. Across the 10 paired tasks, C\+\+ completion rates exceed Blueprint by 30\.0 and 42\.9 percentage points in the two tool configurations\. Among on\-time Blueprint submissions in this paired set that pass asset checks, 42\.2% and 50\.0% fail explicit runtime assertions\. These submissions satisfy asset requirements but fail the required gameplay tests\. We will release the harness, task benchmark, and our trajectory findings with the report\.

CraftBench\-UE: Deterministic Evaluation for Coding Agents in Unreal Engine

Shutong Wu\*

University of Rochester

Kevin Calderone, Andy Tsen†

RamenVR

swu85@ur\.rochester\.edu[https://github\.com/ramenvr/craftbench\-ue](https://github.com/ramenvr/craftbench-ue)

††\*Work done during internship at RamenVR\.†††Corresponding author\.![Refer to caption](https://arxiv.org/html/2609.23142v1/task_overview.png)Figure 1:Sample benchmark tasks across the three deliverable groups\. The screenshots illustrate the glide task \(gp\-glide\-stamina\-cpp\) and mud\-wading task \(t1\-mud\-wade\-bp\)\. The editor\-scripting panel shows thekp\-spawn\-level\-actorstask, which requires creating and saving a level containing six named objects\.## 1Introduction

Implementing a gameplay feature involves more than producing code that compiles\. In Unreal Engine, a feature can span C\+\+ classes, Blueprint graphs, and other features such as the Niagara VFX system\. Blueprint is Unreal’s visual programming system, which uses node graphs saved in binary\.uassetfiles\. The evaluation question is therefore whether the saved project implements the requested behavior, not simply whether it compiles\.

Existing benchmarks evaluate generated functions, repository changes, and tasks in interactive environments\([Chen et al\., 2021](https://arxiv.org/html/2609.23142#bib.bib22);[Jimenez et al\., 2024](https://arxiv.org/html/2609.23142#bib.bib2);[Zhou et al\., 2024](https://arxiv.org/html/2609.23142#bib.bib3);[Merrill et al\., 2026](https://arxiv.org/html/2609.23142#bib.bib13)\)\. Game\-development benchmarks extend executable evaluation to game engines such as Godot, Unreal, and Roblox\([Chi et al\., 2026](https://arxiv.org/html/2609.23142#bib.bib8);[La et al\., 2026](https://arxiv.org/html/2609.23142#bib.bib9);[Zhang et al\., 2025](https://arxiv.org/html/2609.23142#bib.bib6)\)\. Evaluating projects that combine source code and native assets requires checking both the submitted files and the behavior they produce\. The benchmark harness should also differentiate between structural requirements, like a valid asset reference or successful compilation, and behavioral requirements, such as a character decelerating while gliding\.

We introduce CraftBench\-UE, an evaluation harness for coding agents working with Unreal projects\. Each task defines the starting state and the goal deliverables\. The harness collects the generated work, then reconstructs it in a fresh project and grades it based on the build, asset, and runtime checks\. Runtime checks observe behavior in Play\-in\-Editor \(PIE\), Unreal’s mode for running a game inside the editor\.

Based on the harness, we construct a benchmark of 70 bounded tasks: 33 C\+\+ tasks, 25 Blueprint/native\-asset tasks, and 12 editor\-scripting tasks\. The task suite contains 10 gameplay task pairs, each with both C\+\+ and Blueprint versions\. This design enables analysis of how the same agent approaches the same tasks across different deliverable formats\.

We evaluate seven models under two editor\-tool configurations, MCP1 and MCP2 \(§[4](https://arxiv.org/html/2609.23142#S4)\), with a file\-and\-shell C\+\+ baseline\. In the paired analysis, we found that C\+\+ completion exceeds Blueprint completion by 30\.0 and 42\.9 percentage points under MCP1 and MCP2 within the time budget\. Among on\-time Blueprint submissions in this set that satisfy asset checks, 42\.2% \(19/45\) under MCP1 and 50\.0% \(20/40\) under MCP2 fail explicit runtime assertions\. The comparison and result analyses address complementary questions: whether agents complete the same gameplay under different deliverable requirements, and what passing an asset check tells us about the final completed feature\.

Our contributions are:

1. 1\.An Unreal evaluation harness\.CraftBench\-UE runs an agent on a task in an isolated Unreal project, reconstructs its submission and grades it through build, asset, and runtime verifiers\.
2. 2\.A benchmark of 70 tasks\.The tasks cover source code, native assets, and editor scripting, including paired gameplay specifications with the same runtime tests\.
3. 3\.An empirical study of how coding agents work in game development tasks\.We share our results on how different models work through the benchmark, and how they perform and verify their work during the benchmarking process\. Paired comparisons quantify differences under two editor configurations; submission and trajectory analyses show where asset construction and authoring feedback fall short of completed gameplay\.

## 2Related Work

#### Execution\-based agent evaluation\.

Tests can reject a valid solution if they impose requirements that the task description does not state\. SWE\-bench Verified addresses this problem through developer review, screening out unclear tasks and tests that reject valid solutions\([OpenAI, 2024](https://arxiv.org/html/2609.23142#bib.bib18)\)\. The checks should match what the agent was asked to do, whether the task involves writing a function, modifying a repository, or working in a terminal or desktop environment\([Chen et al\., 2021](https://arxiv.org/html/2609.23142#bib.bib22);[Jimenez et al\., 2024](https://arxiv.org/html/2609.23142#bib.bib2);[Merrill et al\., 2026](https://arxiv.org/html/2609.23142#bib.bib13);[Xie et al\., 2024](https://arxiv.org/html/2609.23142#bib.bib1)\)\. We document our validation process in §[3](https://arxiv.org/html/2609.23142#S3)\.

#### Game\-development benchmarks\.

GameDevBench uses deterministic tests for Godot scripts, scenes, and resources and examines how image and video feedback affect agent performance\([Chi et al\., 2026](https://arxiv.org/html/2609.23142#bib.bib8)\)\. In GameEngineBench, agents make C\+\+ changes in open source Unreal projects, and the changes are examined by both build and runtime tests; an LLM judge then considers the test results and implementation work\([La et al\., 2026](https://arxiv.org/html/2609.23142#bib.bib9)\)\. OpenGameEval evaluates Roblox Studio tasks with executable tests\([Zhang et al\., 2025](https://arxiv.org/html/2609.23142#bib.bib6)\)\. GameXpert\-Bench spans generation, repair, and multi\-turn optimization of browser\-native JavaScript games\([Chen et al\., 2026](https://arxiv.org/html/2609.23142#bib.bib10)\)\. These benchmarks already evaluate the behavior of changes made by agents\. GBQA instead scores autonomous defect discovery in running web games by the share of human\-verified bugs found\([Jiang et al\., 2026](https://arxiv.org/html/2609.23142#bib.bib11)\)\.

Other work in Unreal covers cinematic assets and complete games\. CutsceneBench combines tool and asset checks for Level Sequence assets with evaluation of rendered videos\([He et al\., 2026](https://arxiv.org/html/2609.23142#bib.bib16)\)\. AutoUE focuses on end\-to\-end Unreal game generation using an LLM judge as the grader\([Yin et al\., 2026](https://arxiv.org/html/2609.23142#bib.bib7)\)\. Our tasks focus on individual gameplay features with outcomes that could be deterministically checked\. The harness reconstructs each submission in a fresh Unreal project and applies build, asset, and runtime checks\. Appendix[G](https://arxiv.org/html/2609.23142#A7)provides a detailed comparison of these benchmarks\.

#### Interfaces and tool use\.

MCP exposes application operations as tools\([Anthropic, 2024](https://arxiv.org/html/2609.23142#bib.bib23)\), and MCP benchmarks study tool selection and composition\([Luo et al\., 2025](https://arxiv.org/html/2609.23142#bib.bib4);[Wu et al\., 2025](https://arxiv.org/html/2609.23142#bib.bib5)\)\. Orak likewise standardizes agent access to games through MCP, but its deliverable is a gameplay trajectory scored by game outcomes rather than a saved engine artifact\([Park et al\., 2026](https://arxiv.org/html/2609.23142#bib.bib12)\)\. MCP\-Unity connects MCP clients through a Python server to a C\# Unity Editor plugin\([Wu and Barnett, 2025](https://arxiv.org/html/2609.23142#bib.bib17)\); Epic’s in\-editor Unreal MCP server exposes tools over local HTTP and dispatches calls on the game thread\([Epic Games, n\.d\.](https://arxiv.org/html/2609.23142#bib.bib21)\)\. These interfaces provide authoring tool sets; our harness evaluates saved submissions in fresh Unreal projects\.

Agent\-interface studies show that the surrounding configuration can affect coding outcomes\([Yang et al\., 2024](https://arxiv.org/html/2609.23142#bib.bib19);[Zhang et al\., 2026](https://arxiv.org/html/2609.23142#bib.bib20)\)\. SkillsBench compares matched conditions with and without curated procedural Skills using deterministic verifiers\([Li et al\., 2026](https://arxiv.org/html/2609.23142#bib.bib15)\), whereas we vary editor configurations and required deliverables\. We therefore describe editor services, editing permissions, and execution settings alongside the model panel\. The observed differences compare these configurations; they do not identify the effect of an individual tool operation\.

## 3CraftBench\-UE

### 3\.1Evaluation setting and task scope

CraftBench\-UE measures whether an agent can implement a bounded gameplay or editor behavior in an existing Unreal project\. The unit of evaluation is an*atomic mechanism task*: an instruction specifies a bounded gameplay or editor behavior, and states which project files or assets the agent may change\. Examples include attaching an object to a skeletal socket, enforcing a cooldown, persisting a Blueprint graph, and authoring a mini level \(Figure[1](https://arxiv.org/html/2609.23142#S0.F1)\)\. Each task focuses on a small set of engine mechanisms and uses named checks to identify which tested conditions fail\. Tasks may still couple subsystems, as a movement behavior can involve input, state, timing, and physics, but they do not combine unrelated features merely to increase difficulty\. Each score records whether the submitted project implements that mechanism under the stated task conditions\.

### 3\.2CraftBench\-UE evaluation harness

Container\-oriented frameworks such as Harbor package instructions, environments, and tests\([Harbor Framework Team, 2026](https://arxiv.org/html/2609.23142#bib.bib14)\)\. Unreal evaluation is comparatively more complex since the framework needs to control a versioned Editor and PIE environment, while preserving binary assets and saved changes\. A task package specifies the starting project, prompt, writable area, engine build, verifier, and reference implementation\. The harness reconstructs the project and delivers the prompt through the selected agent configuration\. Reference implementations and verifier sources are withheld during authoring; detected exposure excludes a run from scoring \(Appendix[B\.1](https://arxiv.org/html/2609.23142#A2.SS1)\)\.

The harness records interactions, timing, and project changes\. When authoring ends, it collects the saved changes allowed by the task and applies them to a fresh copy of the starting project\. This clean replay separates the submitted artifact from unrelated workspace modifications\. It then builds the project, runs behavioral checks, and inspects native assets as declared by the task\. Reports include the overall verdict, gate outcomes, and the failing checks identified\. Figure[2](https://arxiv.org/html/2609.23142#S3.F2)summarizes the workflow\.

![Refer to caption](https://arxiv.org/html/2609.23142v1/task_lifecycle_revision.png)Figure 2:Task construction, agent authoring, and evaluation of the saved submission in a fresh project\. Each task declares its required checks\. Reference\-pass and unchanged\-project\-fail are admission criteria; §[3\.5](https://arxiv.org/html/2609.23142#S3.SS5)describes task authoring and validation\.
### 3\.3Execution\-based deterministic grading

Each task specifies which of the following checks its submission must pass:

1. 1\.Build \(L1\)\.Compile the relevant editor and game targets\. Submission\-caused compiler or linker errors are failures\.
2. 2\.Behavior \(L2\)\.A fixed\-timestep Play\-in\-Editor \(PIE\) session will start and evaluate named assertions at certain timesteps using Unreal Functional Tests\.111Unreal’s Functional Testing framework runs scripted checks inside a level; tests can be written in C\+\+ or Blueprint\. See[Epic’s Functional Testing documentation](https://dev.epicgames.com/documentation/en-us/unreal-engine/functional-testing-in-unreal-engine)\.Assertions focus on state transitions and variable changes during the session\.
3. 3\.Artifact \(L2I\)\.Load the saved assets in the editor without its graphical interface and check the required graph topology, class hierarchy, typed properties, references, or persistence\.

For each task, we specify the checks a submission must pass and evaluate them directly in Unreal Engine\. The same assertions and checkpoint schedule are applied to every submission for a task\. A fixed simulation timestep \(1/60 second by default\) makes time\-dependent checks comparable, while per\-task numerical tolerances accommodate the quantities being tested; Appendix[B](https://arxiv.org/html/2609.23142#A2)lists the engine switches and the tolerances in force\. No LLM reads the submission, trace, screenshot, or test report to decide whether it passes or fails\. Screenshots may be retained for diagnosis, but never to change the final grade\. Structural checks establish required asset properties, while runtime checks establish the tested behavior\. This excludes creative tasks whose success depends on visual judgment, a limitation we discuss in Sec\.[8](https://arxiv.org/html/2609.23142#S8)\.

### 3\.4The benchmark and required deliverables

The task suite contains 33 C\+\+ tasks, 25 Blueprint tasks, and 12 editor\-scripting tasks\. C\+\+ tasks permit source code changes in a designated folder; BP tasks require saved native assets; scripting tasks are evaluated on their resulting saved project state\. The tasks cover domains including gameplay logic, character movement, animation, UI, and editor operations\. Of the 70 tasks, 50 start from the Unreal Engine 5\.8 Third Person template and 20 from a customized template\. Table[1](https://arxiv.org/html/2609.23142#S3.T1)gives their required checks; Appendix[F](https://arxiv.org/html/2609.23142#A6)lists the individual tasks\.

Figure[1](https://arxiv.org/html/2609.23142#S0.F1)shows the three task types in our benchmark\. Tasks usually contain multiple checks: glide, for example, must slow a falling character, consume Power, and restore to normal falling state when Power is drained\. Editor\-scripting tasks instead check the saved output, such as whether a level contains the requested objects\. The harness uses each task’s declared checks since not all check types are needed for each task\.

Table 1:Required checks in the measured suite\. Runtime and asset columns overlap; each paired BP gameplay task requires both\.Ten gameplay specifications have C\+\+ and BP versions that share a starting project template and runtime verifiers\. They contain the same interface and gameplay instruction; the only difference is the requested deliverable type \(BP and C\+\+\)\. BP additionally requires saved assets\. We inspect the submitted files when interpreting the comparison\. §[4](https://arxiv.org/html/2609.23142#S4)defines the paired analysis\.

### 3\.5Task authoring and validation

The tasks are designed around common Unreal workflows and informed by the team’s engine expertise and public engine documentation\. Team members, including artists, designers, and engineers, propose behaviors that represent the work they use agents to perform\. The lead author filters proposals using three admission criteria: the requested outcome must be \(i\) specific enough to verify with explicit checks, \(ii\) observable in a clean project without inspecting a solution, and \(iii\) implementable through a declared deliverable\. When a gameplay behavior is suitable for both C\+\+ and Blueprint, it can become a matched pair with different artifact requirements\.

The team refines each accepted proposal into a task package that includes instructions, starting project, grading criteria, and other necessary information for agent to start the implementation\. We verify that there are at least one reference solution that passes all required checks and that the unmodified starting project fails\. Appendix[A](https://arxiv.org/html/2609.23142#A1)follows one task from its instruction to the final grading pass\.

## 4Experimental Design

### 4\.1Models and tool configurations

We evaluate Claude Sonnet 5, DeepSeek V4 Pro\-0813, Gemini 3\.7 Flash, GPT\-5\.6 Luna, GPT\-5\.6 Sol, Grok 4\.6, and GLM\-5\.3 Flash\. The two editor\-tool configurations use Unreal MCP \(MCP1\) and Aura MCP \(MCP2\)\. Each model attempts every task under both configurations; C\+\+ also has a file/shell baseline without editor MCP\. All configurations use Claude Code with Unreal Engine 5\.8 on Windows\. Appendix[B](https://arxiv.org/html/2609.23142#A2)records model details\.

MCP1 exposes catalog discovery and a gateway for editor operations\. MCP2 exposes operations directly, including delegated Blueprint and Python tools\. The two MCPs provide different editor tools and file\-editing permissions, as summarized in Table[2](https://arxiv.org/html/2609.23142#S4.T2)\. On BP and Python, their completion differences are therefore not attributable to editor tools alone \(Appendix[B](https://arxiv.org/html/2609.23142#A2)\)\.

Table 2:Agent tool permissions\. Both editor configurations can perform several kinds of project edits; tool availability is part of the evaluated configuration\.
### 4\.2Task run and completion

Each task–model–MCP combination is evaluated with one scored attempt\. The agent starts in a fresh project with the task prompt and is prompted to conduct the task\. The prompt instructs agents to perform at most one final PIE playtest, which the harness does not enforce\. We added this instruction after observing repeated playtest loops in preliminary runs\. Agents receive no human feedback during authoring\.

We report pass@1 with a 40\-minute authoring budget\. An attempt is successful if the agent finishes within this budget and its final submission passes all checks\. Authoring time is measured from an agent launch to its exit, and the project setup and grading time are not included\. Sessions normally have a 60\-minute execution ceiling; final submissions returned after the 40\-minute budget but under the ceiling are retained for analysis\.

### 4\.3Paired comparisons and observed workflows

For each model, we compare its completion count under the two MCP settings and, on C\+\+, also against the file/shell baseline\. We also identify tasks that succeed in one setting and fail in the other, since equal totals can contain successes in different tasks\. The ten paired gameplay specifications compare C\+\+ and BP implementations of the same requested behavior\. We analyze the interaction logs alongside the final test reports to examine how agents conduct their work\. We also measure authoring time and the efficiency of tool calls\.

We compare completion using only the gameplay tests to see whether Blueprint’s extra asset checks account for its lower score\. Appendix[C](https://arxiv.org/html/2609.23142#A3)provides the analysis details\.

## 5Results

We report task completion for each model and configuration, then examine the saved submissions and tool requests to understand incomplete gameplay and the work performed during authoring\. Completion uses the 40\-minute budget; trajectory and runtime diagnostics examine the full recorded sessions and final submissions\. Appendix[E](https://arxiv.org/html/2609.23142#A5)reports authoring time and tool calls\.

Appendix Figure[5](https://arxiv.org/html/2609.23142#A3.F5)shows the full task result\. C\+\+ completion is 180/231 \(77\.9%\) with file/shell tools, 199/231 \(86\.1%\) with MCP1, and 203/231 \(87\.9%\) with MCP2\. Both MCP configurations complete 110/175 BP runs \(62\.9%\)\. Python completion is 62/84 \(73\.8%\) and 66/84 \(78\.6%\)\.

### 5\.1C\+\+ and Blueprint implementations of the same gameplay

Table 3:Completion within 40 minutes on the same ten gameplay tasks\. Seven models run each task, giving 70 runs per evaluated configuration and variant\.The ten paired gameplay tasks compare the same requested behavior across C\+\+ and BP \(Table[3](https://arxiv.org/html/2609.23142#S5.T3)\)\. MCP1 completes 46/70 C\+\+ runs \(65\.7%\) and 25/70 BP runs \(35\.7%\); MCP2 completes 50/70 \(71\.4%\) and 20/70 \(28\.6%\), respectively\. Every model completes more C\+\+ versions under both configurations \(Figure[3](https://arxiv.org/html/2609.23142#S5.F3); Table[7](https://arxiv.org/html/2609.23142#A3.T7)\)\. The C\+\+ file/shell baseline already completes 45/70 runs \(64\.3%\), so the higher C\+\+ completion also occurs without editor MCP on the C\+\+ side; most of each MCP configuration’s advantage over that baseline is on C\+\+ tasks without BP counterparts \(Table[8](https://arxiv.org/html/2609.23142#A3.T8)\)\.

The C\+\+ completion advantage is 30\.0 percentage points under MCP1 and 42\.9 under MCP2\. Counting submissions that pass the runtime checks without requiring the additional BP asset checks leaves all four completion counts unchanged \(Table[9](https://arxiv.org/html/2609.23142#A3.T9)\)\. Within the deadline, no BP submissions pass the runtime checks but fail the asset checks\. The BP deficit persists when completion is assessed by gameplay testing\. Producing saved native assets remains part of the BP authoring task\.

The submissions show how the successful implementations were authored\. Although some C\+\+ prompts also permit Blueprint assets, every successful matched C\+\+ submission under either MCP configuration changes C\+\+ source files without changing Blueprint assets\. Successful BP submissions change assets without changing the source\.

Figure 3:C\+\+ and BP completions within 40 minutes on the ten paired gameplay tasks\. Each line connects the same model’s totals under one MCP configuration, with one attempt per task variant\.The BP breakdown separates local edits from complete gameplay features \(Appendix Figure[6](https://arxiv.org/html/2609.23142#A3.F6)\)\. The three basic graph tasks ask for logging, arithmetic, or delayed movement; two local repair tasks target specified defects in existing behaviors\. Both configurations complete 34/35 runs \(97\.1%\) on these five tasks\. On ten asset\-configuration tasks, completion is 51/70 \(72\.9%\) and 56/70 \(80\.0%\); on the ten gameplay tasks, it falls to 25/70 \(35\.7%\) and 20/70 \(28\.6%\)\. Agents often complete the included local operations, while fewer submissions implement all parts of a gameplay feature correctly\.

### 5\.2Where Blueprint implementations fall short

On the ten paired gameplay tasks, seven additional MCP1 BP runs and ten additional MCP2 runs yield passing submissions after the 40\-minute deadline\. Across all final submissions, those that pass asset checks but fail an explicit runtime assertion account for 24/70 \(34\.3%\) under MCP1 and 28/70 \(40\.0%\) under MCP2\. Restricted to submissions whose authoring finished within 40 minutes, they are 19/45 and 20/40 of the on\-time submissions that pass asset checks\. Appendix[D\.3](https://arxiv.org/html/2609.23142#A4.SS3)lists the failed checks and the available reports\.

The runtime reports provide further observations about these incomplete features\. Among submissions that pass asset checks but fail runtime checks, 13 submissions across four BP tasks successfully activate an ability but fail to produce its required effect or meet the required resource check\. Seven of them concern the gliding task: the character must descend more slowly while consuming Power\. All seven abilities activate, and six consume Power, but none produce the required glide\. For example, in three implementations, Power always activates at the start but does not exhaust after gliding for a set time\. The other cases comprise four missing periodic burns, one second jump without its Power cost, and one damage operation that leaves Health unchanged\. Nine of these runs finish authoring within the scoring budget\. Appendix[D\.3](https://arxiv.org/html/2609.23142#A4.SS3)identifies the 13 submissions and the runtime observations supporting this breakdown\.

### 5\.3How Agents Inspect and Verify Their Work

We examine how agents respond to editing feedback and check their work under the instruction to perform at most one PIE playtest\.*Asset readback*means retrieving an asset’s current properties or graph from the editor\.*PIE state readback*means retrieving values from a running game, such as character position or remaining Power\. Both are actions during authoring; the harness applies the task’s runtime checks to the submitted project afterward\.

Agents revise both rejected and accepted edits\.Of the 175 MCP1 BP runs, 95 directly write Blueprint graphs; 75 read task assets before their first graph write\. Among the 58 runs with graph\-writing errors, 37 responded in their next tool call by changing the request for the same graph\. Agents also revised graphs after requests that returned without an error: this occurred in 30 of the 94 runs with such responses\. Before making these later changes, agents often inspected the graph or requested compilation\. Appendix[D\.1](https://arxiv.org/html/2609.23142#A4.SS1.SSS0.Px2)gives the detailed counts and sequence definitions\.

Asset readbacks are more frequent than PIE state readbacks\.Most BP runs contain asset readback or compilation requests, while PIE state readback appears in 24/175 MCP1 and 30/175 MCP2 runs \(Table[4](https://arxiv.org/html/2609.23142#S5.T4)\)\. Agents obtain these values through property queries, scripts that combine actions and measurements, and game\-state recordings\. C\+\+ agents also use shell commands to request game or engine\-test execution, in 57/231 MCP1 and 39/231 MCP2 runs, including unsuccessful attempts\. In ten Python MCP2 runs across six tasks and five models, agents find the relevant API or tool, run an editor script that reports success, and read back task\-asset state\. The analysis follows these different routes through the request arguments and responses \(Appendix[D\.1](https://arxiv.org/html/2609.23142#A4.SS1.SSS0.Px1)\)\.

Table 4:Checking actions in BP trajectories, counted over full authoring sessions\. Rows overlap\. Request counts include unsuccessful attempts; readback counts require returned values\. PIE starts include tool calls and starts within scripts\. Image requests can capture either the editor or gameplay\. Appendix[D\.1](https://arxiv.org/html/2609.23142#A4.SS1.SSS0.Px1)defines the categories and their coverage\.Success revisions do not guarantee correct gameplay\.In 39 MCP1 Blueprint runs that finished within 40 minutes, every graph\-writing error was followed by a changed request for the same graph that returned without an error\. Nevertheless, 16 submissions failed the final evaluation, including 15 that failed runtime assertions across six models and eight tasks\. A separate inspection of final submissions found five MCP1 Blueprint submissions with resource references that the engine could not resolve; these submissions also failed runtime checks \(Appendix[D\.2](https://arxiv.org/html/2609.23142#A4.SS2)\)\.

On the ten paired BP tasks, five MCP1 and ten MCP2 runs contain PIE state readback and pass the asset checks, yet fail the final runtime checks\. Reading a value during PIE can leave other required behaviors untested\.

## 6Discussion

#### Design tasks with interacting gameplay requirements\.

A glide feature must slow descent, consume Power, and restore normal falling when Power is exhausted\. These requirements need to hold together: checking resource consumption alone would not detect an implementation that leaves descent unchanged\. The runtime reports and graph\-revision results show why individual editing operations are not sufficient evidence of completion\. For bounded gameplay tasks, evaluation should cover each required behavior and the transitions between them within one gameplay mechanism\.

#### Implications for editor\-native assets\.

Our findings motivate evaluating native assets in the engine systems that use them\. For animation assets, materials, and scene configuration, the question is not only whether their saved properties are correct, but whether they produce the requested motion, appearance, or interaction\. CraftBench\-UE examines this connection in Unreal gameplay\. Whether the same failure patterns occur in other engines and asset workflows remains a question for future work\.

#### Verification inside the game engine\.

The observed workflows suggest two complementary forms of feedback\. Inspecting asset properties and compiling graphs helps agents check what they have built; running the game and measuring its state helps them check what it does\. Asset readbacks are more frequent than PIE state readbacks in the retained trajectories, measured under the prompt’s instruction to perform at most one PIE playtest \(§[4](https://arxiv.org/html/2609.23142#S4)\)\. The runtime reports show several errors that can be identified using this method, such as unresolved references and missing or incorrect resource usage\. Because game development offers many verification methods, such as screenshots, console logs, and PIE tests, how agents choose or combine these methods is worth studying with a larger sample size, especially when many methods are hints or stimuli rather than direct test feedback\. For example, console logs in major game engines can be helpful but may also include unrelated information that bloats the agent context\. For agent design, this suggests connecting each requested expected behavior to a set of verification methods and expected results, with a verification flow that connects these individual parts and automatically play\-tests the implementations\.

## 7Conclusion

CraftBench\-UE evaluates coding agents through their saved source and assets and the behavior those deliverables produce in Unreal Engine\. Its harness reconstructs submissions in fresh projects and applies task\-specific build, asset, and runtime checks\. The 70\-task benchmark supports comparisons across three deliverable groups\. On 10 matched gameplay specifications, C\+\+ completion exceeds BP completion under both editor configurations\. Submissions that pass asset checks can still fail runtime assertions, demonstrating the value of evaluating structural requirements and gameplay behavior together\.

## 8Limitations

Task and check scope\.The 70 tasks are selected for explicit, checkable outcomes rather than sampled representatively from game development\. They do not measure visual appeal or the integration of many features into a complete game\. Future work could extend the task set to long\-horizon projects and creative requirements with separately validated evaluation criteria\. Meanwhile, our current checks include three necessary types of tests in Unreal Engine, while future work could explore custom tests to verify generated features, potentially providing more coverage than this harness\.

Experimental scope\.Each task–model–configuration run has one attempt, with at most one PIE playtest during authoring\. Results do not estimate repeated\-attempt variability, long\-horizon task performance, or performance with unrestricted testing\. Future work could explore more about how agents verify themselves to produce the expected outcome\.

Environment coverage\.All experiments use Unreal Engine 5\.8 on Windows through Claude Code\. We have not tested whether the observed completion patterns persist across other engine versions, operating systems, or agent runtimes\.

## Acknowledgments

We thank the entire Aura team for their support and contributions to the benchmark tasks\.

## Ethics Statement

The authors’ organization develops MCP2, whose implementation remains proprietary\. Appendix[B](https://arxiv.org/html/2609.23142#A2)describes its exposed tool surface\. The benchmark team authors the tasks and verifiers\. Both editor configurations are evaluated on the same task suite using the same verifiers, and their results are reported separately\.

## Reproducibility Statement

The harness, benchmark materials, and analysis code will be released upon publication at[https://github\.com/ramenvr/craftbench\-ue](https://github.com/ramenvr/craftbench-ue)\. Task instructions, starting\-project configurations, verifier, and reference implementations are included as well\. Appendix[D](https://arxiv.org/html/2609.23142#A4)defines the trajectory analyses\. Appendix[B\.2](https://arxiv.org/html/2609.23142#A2.SS2)lists the release contents\. Users will need to install Unreal Engine and provide it in the environment; engine binaries and proprietary MCP internals are not redistributed\.

## AI Use Statement

Generative\-AI assistants supported task ideation, methodological critique, data analysis, and figure brainstorming\. The rule\-based pipeline in §[3](https://arxiv.org/html/2609.23142#S3)produced benchmark verdicts\. The authors remain responsible for the analysis, claims, and final manuscript\.

## References

- AnthropicIntroducing the Model Context Protocol\.Note:industry blogExternal Links:[Link](https://www.anthropic.com/news/model-context-protocol)Cited by:[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px3.p1.1)\.
- Chenet al\.\(2026\)K\. Chen, H\. Hong, P\. Gao, J\. Lin, T\. Luo, Y\. Xie, C\. Liu, J\. He, Z\. Liu, and Z\. ZengGameXpert\-Bench: How Far Are Coding Agents from Expert Game Development?\.External Links:2608\.21833,[Link](https://arxiv.org/abs/2608.21833)Cited by:[Table 18](https://arxiv.org/html/2609.23142#A7.T18.2.8.1.1.1),[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda,et al\.Evaluating Large Language Models Trained on Code\.External Links:2107\.03374,[Link](https://arxiv.org/abs/2107.03374)Cited by:[§1](https://arxiv.org/html/2609.23142#S1.p2.1),[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px1.p1.1)\.
- Chiet al\.\(2026\)W\. Chi, Y\. Fang, A\. Yayavaram, S\. Yayavaram, S\. Karten, Q\. A\. Wei, R\. Chen, A\. Wang, V\. Chen, A\. Talwalkar, and C\. DonahueGameDevBench: Evaluating Agentic Capabilities Through Game Development\.InICML 2026,External Links:2602\.11103,[Link](https://arxiv.org/abs/2602.11103)Cited by:[Table 18](https://arxiv.org/html/2609.23142#A7.T18.2.2.1.1.1),[§1](https://arxiv.org/html/2609.23142#S1.p2.1),[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px2.p1.1)\.
- Epic Games \(n\.d\.\)Epic GamesUnreal MCP in Unreal Editor\.Note:Unreal Engine 5\.8 official documentationAccessed September 14, 2026External Links:[Link](https://dev.epicgames.com/documentation/unreal-engine/unreal-mcp-in-unreal-editor)Cited by:[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px3.p1.1)\.
- Harbor Framework Team \(2026\)Harbor Framework TeamHarbor: A Framework for Evaluating and Optimizing Agents and Models in Container Environments\.Note:SoftwareExternal Links:[Link](https://github.com/harbor-framework/harbor)Cited by:[§3\.2](https://arxiv.org/html/2609.23142#S3.SS2.p1.1)\.
- Heet al\.\(2026\)L\. He, H\. Pang, Q\. Gan, X\. Shen, Z\. Zhang, Y\. Liu, G\. Fang, B\. Liu, K\. Sheng, S\. Zeng, C\. Li, Z\. Hui, K\. Zhou, L\. Zhou, and S\. DaiCutscene Agent: An LLM Agent Framework for Automated 3D Cutscene Generation\.External Links:2604\.25318,[Link](https://arxiv.org/abs/2604.25318)Cited by:[Table 18](https://arxiv.org/html/2609.23142#A7.T18.2.6.1.1.1),[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px2.p2.1)\.
- Jianget al\.\(2026\)S\. Jiang, C\. Chen, and Z\. ChenGBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineers\.Note:ICLR 2026 workshop paperExternal Links:2604\.02648,[Link](https://arxiv.org/abs/2604.02648)Cited by:[Table 18](https://arxiv.org/html/2609.23142#A7.T18.2.5.1.1.1),[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px2.p1.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. NarasimhanSWE\-bench: Can Language Models Resolve Real\-World GitHub Issues?\.InICLR 2024 \(oral\),External Links:2310\.06770,[Link](https://arxiv.org/abs/2310.06770)Cited by:[§1](https://arxiv.org/html/2609.23142#S1.p2.1),[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px1.p1.1)\.
- Laet al\.\(2026\)B\. La, S\. Chang, B\. Kim, J\. Bae, A\. A\. Beg, S\. Chang, G\. Gonzalez\-Pumariega, and K\. GoyalGameEngineBench: Evaluating Coding Agents on Real C\+\+ Runtime Environments\.External Links:2607\.03525,[Link](https://arxiv.org/abs/2607.03525)Cited by:[Table 18](https://arxiv.org/html/2609.23142#A7.T18.2.3.1.1.1),[§1](https://arxiv.org/html/2609.23142#S1.p2.1),[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2026\)X\. Li, Y\. Liu, W\. Chen, B\. You, Z\. Di, Y\. He, S\. Zheng, K\. W\. Choe,et al\.SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks\.External Links:2602\.12670,[Link](https://arxiv.org/abs/2602.12670)Cited by:[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px3.p2.1)\.
- Luoet al\.\(2025\)Z\. Luo, Z\. Shen, W\. Yang, Z\. Zhao, P\. Jwalapuram, A\. Saha, D\. Sahoo, S\. Savarese, C\. Xiong, and J\. LiMCP\-Universe: Benchmarking Large Language Models with Real\-World Model Context Protocol Servers\.External Links:2508\.14704,[Link](https://arxiv.org/abs/2508.14704)Cited by:[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px3.p1.1)\.
- Merrillet al\.\(2026\)M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi, J\. Y\. Shin, T\. Walshe,et al\.Terminal\-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces\.InICLR 2026 \(poster\),External Links:2601\.11868,[Link](https://arxiv.org/abs/2601.11868)Cited by:[§1](https://arxiv.org/html/2609.23142#S1.p2.1),[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px1.p1.1)\.
- OpenAI \(2024\)OpenAIIntroducing SWE\-bench Verified\.Note:industry blogExternal Links:[Link](https://openai.com/index/introducing-swe-bench-verified/)Cited by:[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px1.p1.1)\.
- Parket al\.\(2026\)D\. Park, M\. Kim, B\. Choi, J\. Kim, K\. Lee, J\. Lee, I\. Park, B\. Lee, J\. Hwang, J\. Ahn,et al\.Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games\.InICLR 2026 poster,External Links:2506\.03610,[Link](https://arxiv.org/abs/2506.03610)Cited by:[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px3.p1.1)\.
- Wu and Barnett \(2025\)S\. Wu and J\. P\. BarnettMCP\-Unity: Protocol\-Driven Framework for Interactive 3D Authoring\.InSIGGRAPH Asia 2025 Technical Communications,External Links:[Document](https://dx.doi.org/10.1145/3757376.3771417),[Link](https://doi.org/10.1145/3757376.3771417)Cited by:[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px3.p1.1)\.
- Wuet al\.\(2025\)Z\. Wu, X\. Liu, X\. Zhang, L\. Chen, F\. Meng, L\. Du, Y\. Zhao, F\. Zhang,et al\.MCPMark: A Benchmark for Stress\-Testing Realistic and Comprehensive MCP Use\.External Links:2509\.24002,[Link](https://arxiv.org/abs/2509.24002)Cited by:[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px3.p1.1)\.
- Xieet al\.\(2024\)T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng,et al\.OSWorld: Benchmarking Multimodal Agents for Open\-Ended Tasks in Real Computer Environments\.InNeurIPS 2024 \(Datasets and Benchmarks Track\),External Links:2404\.07972,[Link](https://arxiv.org/abs/2404.07972)Cited by:[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2024\)J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. PressSWE\-agent: Agent\-Computer Interfaces Enable Automated Software Engineering\.InNeurIPS 2024 \(poster\),External Links:2405\.15793,[Link](https://arxiv.org/abs/2405.15793)Cited by:[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px3.p2.1)\.
- Yinet al\.\(2026\)L\. Yin, W\. Cheng, Z\. Qin, T\. Huang, Y\. Li, and G\. DingAutoUE: Automated Generation of 3D Games in Unreal Engine via Multi\-Agent Systems\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 2341–2364\.External Links:2603\.07106,[Link](https://aclanthology.org/2026.findings-acl.111/)Cited by:[Table 18](https://arxiv.org/html/2609.23142#A7.T18.2.7.1.1.1),[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px2.p2.1)\.
- Zhanget al\.\(2025\)T\. Zhang, K\. Ayyar, M\. Sun, and L\. GongUsing OpenGameEval to Benchmark Agentic AI Assistants for Roblox Studio\.Note:Roblox newsroomExternal Links:[Link](https://about.roblox.com/newsroom/2025/12/opengameeval-benchmark-agentic-ai-assistants-roblox-studio)Cited by:[Table 18](https://arxiv.org/html/2609.23142#A7.T18.2.4.1.1.1),[§1](https://arxiv.org/html/2609.23142#S1.p2.1),[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2026\)Y\. Zhang, J\. Wang, Y\. Ge, W\. Xu, J\. Hamm, and C\. K\. ReddyStop Comparing LLM Agents Without Disclosing the Harness\.External Links:2605\.23950,[Link](https://arxiv.org/abs/2605.23950)Cited by:[§2](https://arxiv.org/html/2609.23142#S2.SS0.SSS0.Px3.p2.1)\.
- Zhouet al\.\(2024\)S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. NeubigWebArena: A Realistic Web Environment for Building Autonomous Agents\.InICLR 2024,External Links:2307\.13854,[Link](https://arxiv.org/abs/2307.13854)Cited by:[§1](https://arxiv.org/html/2609.23142#S1.p2.1)\.

## Appendix AA Worked Example of Task Validation

The glide task in Figure[4](https://arxiv.org/html/2609.23142#A1.F4)asks the agent to slow a falling character while consuming Power, then restore normal falling when Power runs out\. The runtime fixture requests ability activation and observes descent and Power at scheduled checkpoints\. Checking Power alone would miss an implementation that consumes the resource without slowing the character\.

![Refer to caption](https://arxiv.org/html/2609.23142v1/glide_validation_revision.png)Figure 4:Archived development/validation traces for the glide task, separate from the study’s agent runs\. In the shaded post\-request interval, the passing trace slows descent while Power drains; the failing trace drains Power without slowing descent\. The plotted values and verdict labels are transcribed from a retained development capture\. The Manny screenshot is a separate scene illustration, not a frame from either trace\.This example illustrates why task authors check behavior after integration into the engine\. Figure[1](https://arxiv.org/html/2609.23142#S0.F1)in the main paper illustrates tasks across the three deliverable groups\.

## Appendix BExecution Configuration and Submission Handling

The experiments use Unreal Engine 5\.8 on Windows with Claude Code as the agent driver\. Table[5](https://arxiv.org/html/2609.23142#A2.T5)records the requested outer\-agent model identifiers\. Provider\-qualified identifiers are served via the same Claude Code CLI, with its Anthropic base URL pointing to OpenRouter’s Anthropic\-compatible endpoint and the OpenRouter key supplied as the CLI’s bearer token;claude\-sonnet\-5is requested directly from Anthropic\. The analysis manifest retains available model and driver metadata\.

Table 5:Requested agent model identifiers\.#### Tool exposure\.

MCP1 exposes three entry points: catalog discovery, tool description, and a gateway that calls a named operation\. MCP2 exposes 176 entry points, including direct editing and delegated tools\. These names include asset and Blueprint inspection, Blueprint graph editing and compilation, asset creation and editing, text and C\+\+ file editing with in\-editor Python execution, play\-in\-editor control, and more\. A gateway can access many operations, so entry\-point counts do not measure tool coverage\. Table[2](https://arxiv.org/html/2609.23142#S4.T2)gives file and shell permissions\. Initialization failures count as failed attempts\.

#### Timing and comparison scope\.

The runner used a 60\-minute execution ceiling\. Completion required the agent to finish within 40 minutes and its final submission to pass all required checks\. The submitted project is graded afterward\. The C\+\+ baseline uses two hosts, whereas its MCP comparisons use one\. Both hosts use the same hardware configuration, each with an NVIDIA GeForce RTX 5080 GPU\. On BP and Python tasks, MCP1 also retains the native write, edit, and shell tools that MCP2 does not \(Table[2](https://arxiv.org/html/2609.23142#S4.T2)\), and some MCP1 runs on these surfaces issue native file or shell writes, so this permission difference is exercised in the recorded sessions\. Driver versions vary within and across task groups, and some records lack version metadata\. These differences limit attribution of completion or timing differences to MCP access alone\.

Behavior \(L2\) checks launch the editor with\-deterministic \-FPS=60, which fixes the simulation step at 1/60 second from the first frame; a task may declare further legs at other fixed steps, which are replayed in separate sessions and must all pass\. Checkpoints are evaluated against the world’s game time, counted from when play begins rather than from when the test fixture starts\.

#### Agent inputs and testing\.

Each run supplies the task prompt and starting project, specifies where to save changes, and instructs the agent to perform at most one PIE playtest\. The limit is prompt\-level only: the harness does not intercept or refuse play\-start requests during authoring, so additional playtests consume the run’s authoring budget rather than being blocked; however, there is no such request in the current trajectories\. Play\-start requests are instead recovered from the trajectories after the run \(Appendix[D\.1](https://arxiv.org/html/2609.23142#A4.SS1.SSS0.Px1)\)\. The agent receives no human feedback or grading reports during authoring\. The harness grades its saved submission afterward\.

#### Saved submissions\.

The harness collects changed or new files accepted by the starting project’s submission manifest and applies them to a fresh starting project\. The grader applies the task’s additional file and configuration constraints before evaluating the reconstructed project\. This procedure makes the saved submission, rather than unsaved editor state, the object of evaluation\.

### B\.1Access to Evaluation Material

Before authoring, the harness withholds reference implementations, verifier sources, and repository metadata that could expose them\. It also removes other tasks’ scaffolds and replaces runtime test bodies with stubs while retaining the build definitions needed to compile the authoring project\. Withheld material is restored for grading in the clean project\.

The harness monitors the withheld paths and excludes an attempt from scoring if it detects exposed evaluation material\. Task instructions and permitted starting content remain available during authoring\.

### B\.2Release Materials

The release includes the harness, task instructions, starting\-project setup, verifiers, reference implementations, retained submissions, and analysis code\. Run identifiers and source hashes connect the reported panel, check results, and trajectory classifications to their underlying records\. Public trajectories omit credentials and private content while retaining the feedback and submission evidence needed to inspect the analyses\. Users provide an Unreal Engine installation; engine binaries and proprietary MCP2 internals are not redistributed\.

## Appendix CDetailed Results and Scoring Checks

This appendix gives detailed results for the 70\-task suite shown in Figure[5](https://arxiv.org/html/2609.23142#A3.F5)and its ten paired gameplay specifications\. Each record is identified by task, model, and configuration\. The resulting panel has no missing or duplicate combinations\. Completion requiresoverall=PASSand meets the 40\-minute time budget\. Build checks are L1, runtime checks of gameplay are L2, and asset checks of saved structure are L2I\.

Table 6:Full\-task completions within 40 minutes, with counts and percentages shown separately for each surface\. The file/shell baseline is evaluated only on C\+\+\.![Refer to caption](https://arxiv.org/html/2609.23142v1/completion_by_model_40m.png)Figure 5:Task completion within 40 minutes, by model and configuration\. Each model has three stacked bars: Baseline, MCP1, and MCP2\. Colored segment lengths indicate the counts of completed C\+\+, Blueprint, and Python tasks\. Labels inside each segment give completed tasks out of 33 C\+\+, 25 Blueprint, or 12 Python tasks\. The baseline uses file/shell tools and was evaluated only on C\+\+, so its total bar length is not comparable with those of the MCP configurations\. Table[6](https://arxiv.org/html/2609.23142#A3.T6)gives the same counts\.Table 7:Per\-model completion within 40 minutes on the ten paired gameplay specifications\.Table 8:C\+\+ completion within 40 minutes, split by whether the specification also has a BP version\. The split follows the task definitions\. It is not selected from outcomes\.#### BP task groups\.

We group BP tasks by their instructions: three basic graph tasks \(logging, arithmetic, and delayed movement\), two local repairs \(door hitch and inventory stacking\), ten asset\-configuration tasks, and ten paired gameplay tasks\. Asset\-configuration tasks check saved structure; gameplay tasks also check runtime behavior\.

Figure 6:BP completion by task type\. Row labels give the number of tasks and runs per configuration \(seven models per task\)\. Groups follow the task instructions and differ in content, starting state, and required checks\.
#### Counting runtime passes\.

The ten C\+\+/BP gameplay pairs share runtime fixtures; BP also requires asset checks\. Counting recorded runtime passes within 40 minutes leaves all four completion counts unchanged \(Table[9](https://arxiv.org/html/2609.23142#A3.T9)\)\. Every counted runtime pass also passes the build checks\.

Table 9:Completion on the ten paired gameplay tasks, with and without the BP asset checks\. Each task group has 70 attempts per configuration\. All counts require authoring to finish within 40 minutes\. Counting recorded BP runtime passes leaves the full\-task counts and the C\+\+–BP gap unchanged; the original submissions are used in both counts\. Gap is measured in percentage points\.All 20 BP submissions that fail asset checks have runtime reports, including five MCP1 and seven MCP2 submissions returned within 40 minutes\. Five attempts haveFAIL\_NO\_EDITSand no runtime outcome; they remain failures in the denominator\. One MCP1 submission passes runtime but fails asset checks after 52\.1 minutes\. Runtime automation failures remain failures even when individual messages report successful behavior\.

#### Python input bindings\.

The input\-binding task checks action types, six key bindings, and their modifiers across four input assets\. Scores use the engine’s activeDefaultKeyMappings\.Mappingsarray in the retained submissions\. These checks assess saved input configuration without simulating keypresses\.

## Appendix DTrajectory Analysis

### D\.1Inspection and Verification in Agent Trajectories

The trajectory analysis covers 693 C\+\+ transcripts, 350 BP transcripts, and 168 Python transcripts, using the run membership and reported outcomes in Appendix[C](https://arxiv.org/html/2609.23142#A3)\. Counts cover full authoring sessions unless a time limit is stated\. For MCP1, the operation is read from the gateway request’s toolset and operation arguments\. Requests are matched to returns by tool\-use identifier and occurrence; seven requests lack retained returns, and six nested calls in the baseline C\+\+ traces are excluded\. The counts describe visible agent requests and returns; unobserved work inside tool services is outside this analysis\.

#### Asset and PIE state readback\.

An*asset readback*is a returned value from a task\-owned asset, excluding errors, search results, schema descriptions, and object handles without state\. A*PIE state readback*contains a value measured while the game runs in Play\-in\-Editor \(PIE\)\. It may come from a dedicated getter, a property or transform query targeting aUEDPIEobject, an inspected script, a combined action sequence, or a recording\. A*PIE start request*counts the request to start the game, including starts within identified scripts\. A start request alone is insufficient for a state readback\.

Table[4](https://arxiv.org/html/2609.23142#S5.T4)reports these methods for all BP runs; Table[10](https://arxiv.org/html/2609.23142#A4.T10)separates them by model\. Images count separately and require an image block in the agent’s conversation: seven and eight runs qualify, respectively\. Script\-based readbacks require returned measurements rather than a printed success statement\. Compilation counts requests; a return without an error, including a null return, does not establish successful compilation\.

Table 10:BP inspection methods by model, out of 25 runs per configuration\. PIE start requests include identified scripts\. PIE state readbacks require returned measurements from the running game\. An image must appear in the agent’s conversation\. Rows use the definitions in Appendix[D\.1](https://arxiv.org/html/2609.23142#A4.SS1.SSS0.Px1)\.On C\+\+, 151 reviewed shell requests ask for game or engine\-test execution, covering 57/231 MCP1 and 39/231 MCP2 runs\. Two further requests only list available tests and are excluded\. The counts include failed starts and background requests whose completion is unconfirmed\. These shell operations are separate from PIE start requests\.

On Python, ten MCP2 runs across six tasks and five models contain tool/API discovery, a task\-directed modification script, an explicitbSuccess:truereturn, and subsequent asset readback in the task’s namespace\. The readback need not concern the modified field\. MCP1 has a different return protocol, so this sequence is not used to rank script success between configurations\.

#### Graph revisions\.

All 175 MCP1 BP trajectories are screened for direct graph writing\. The 95 qualifying runs contain 454 writes with complete request–return matching; 75 read task assets before their first graph write\. An*accepted graph write*returns without a reported tool error\. A revision changes the arguments for the same graph after the previous write returns\. Six malformed writes have no recoverable graph identity and cannot be paired\.*Immediate*means that the revision is the next agent tool request, rather than a short elapsed time\.

Table[11](https://arxiv.org/html/2609.23142#A4.T11)distinguishes revisions after accepted and rejected writes, together with intervening readbacks and compilation returns\. Revisions can extend functionality as well as correct errors\.

Table 11:Graph revisions in MCP1 BP runs\. Columns cover all direct graph\-writing runs, the seven tasks on which every model directly writes graphs, and sessions ending within 40 minutes\. Rows overlap\. A compilation return need not report successful compilation\. Immediate means the next agent tool request after the preceding write returns; 58 runs contain errors, and 30 revise accepted writes\.Table[12](https://arxiv.org/html/2609.23142#A4.T12)links graph\-writing errors and subsequent accepted revisions to submission outcomes\. Of the 16 on\-time failures after every reported error receives an accepted revision, 15 fail explicit runtime checks, spanning six models and eight gameplay tasks\.

Table 12:Graph\-writing errors and subsequent revisions in MCP1 BP runs\. Rows successively restrict the 175\-run panel\. Accepted revisions return without a reported tool error\. On\-time passes finish authoring within 40 minutes; final passes include later submissions\.

### D\.2Unresolved Resource References

Five MCP1 BP submissions across five tasks and two models contain both an unresolved\-resource warning and a failed runtime check\. Two omit the resource’s owning class; three use a class\-qualified name that the engine cannot resolve\. The warnings establish unresolved references, but need not explain every failed check: the healing warning concerns Health, while the failed check concerns MaxHealth initialization\. The census covers retained runtime reports from 101/105 MCP1 and 103/105 MCP2 runs on the 15 BP tasks that declare runtime checks, including 66/70 and 69/70 paired gameplay runs\.

UE 5\.8 source inspection identifies two supported lookup routes: an owning class plus a member name, or a valid full field path\. No submission was modified to test whether resolving its reference would complete the task\.

### D\.3Runtime Failures After Passing Asset Checks

Of the 140 BP attempts on matched gameplay tasks, runtime reports are retained for 66/70 under MCP1 and 69/70 under MCP2\. The five attempts without reports haveFAIL\_NO\_EDITSoutcomes\. We select submissions that pass asset checks and fail an explicit runtime assertion: 24/70 and 28/70\. One additional MCP1 automation failure has no such assertion and is excluded from this category\.

Failure after passing asset checksMCP1MCP2Initial Health or Power34Required effect or resource use86Pose, display, or object indicator58Restoration or preserved behavior56Poison damage ratio34Total attempts24/7028/70Authoring finished within 40 minutes19/7020/70Table 13:BP submissions that pass asset checks but fail a runtime check\. Each of the 52 selected attempts has one observed failure category; subsequent checks may not execute if a previous check failed\.Selected BP submissionsMCP1MCP2On\-time submissions passing asset checks4540Runtime checks pass2520Explicit runtime assertion failure19/45 \(42\.2%\)20/40 \(50\.0%\)Runtime automation failure10Table 14:Runtime outcomes among BP submissions on the ten gameplay specifications that pass asset checks and finish authoring within 40 minutes\. The last three rows partition this selected subset; automation failures are separate from failed assertions\.Categories group the state measured by failed assertions, with duplicate messages removed within each attempt\. Each of these 52 attempts has one observed category; later checks may not execute\. The reports evaluate the final saved submissions\.

Thirteen effect or resource failures outside poison stacking also show the ability got activated successfully: seven under MCP1 and six under MCP2, across area burn, double jump, glide, and Health operations\. Seven concern gliding; six consume Power before any verifier\-induced exhaustion\. Some reports contain no valid glide sample or no forced\-exhaustion stage, so these observations do not establish a common failure mechanism across all seven submissions, but show how agents might fail when implementing features\. Eight versioned verifier source objects establish how activation and resource use were measured\. C\+\+ failure selection requires passing build checks; BP selection requires passing asset checks\.

## Appendix EAuthoring Time and Cost

For all seven models, median C\+\+ authoring time is longer with either MCP configuration than with the Baseline \(Figure[7](https://arxiv.org/html/2609.23142#A5.F7)\)\. Durations include unsuccessful runs and sessions that exceed the 40\-minute deadline; completion still requires finishing within that deadline\.

![Refer to caption](https://arxiv.org/html/2609.23142v1/cpp_completion_vs_authoring_time.png)Figure 7:C\+\+ completion and median authoring time on the same 33 tasks per configuration\. Markers show median duration on a logarithmic scale; labels give completion counts\. All runs enter the duration calculation\.#### Input context and billed cost\.

Of 231 C\+\+ MCP task–model pairs, 229 have identical retained task prompts and 185 have both first requests linked to billing records\. In these 185 pairs, median first\-request input is 3\.99 times as large under MCP2 as under MCP1, a median difference of 82,354 tokens\. Each configuration retains its own system instructions and tool descriptions\.

Full\-session billing is available for 162 prompt\-matched pairs \(Table[15](https://arxiv.org/html/2609.23142#A5.T15)\)\. The median paired MCP2/MCP1 charge ratio is 0\.21 for DeepSeek and 1\.80–2\.69 for the other five covered models\. Cache use also differs by model: among 19 DeepSeek pairs with complete token records, the median share of input served from cache is 3\.47% under MCP1 and 98\.21% under MCP2; for Sol’s 33 pairs, it is 92\.12% and 93\.17%\.

Table 15:C\+\+ completion and billed inference on the matched subset\. Completion columns count passes within 40 minutes\.*Charge ratio*is the median paired MCP2/MCP1 ratio;*Higher*counts pairs with larger MCP2 charges\. Pair counts give billing coverage out of 33 tasks per model\.
#### Accounting\.

We match generation IDs from CLI summaries and assistant messages to OpenRouter billing records, require the requested model and a numeric charge for every observed generation, deduplicate IDs across four ledgers, and sumcost\_details\.upstream\_inference\_cost\. This excludes unrelated projects in the same ledgers\. On fully linked C\+\+ runs, CLI cost estimates exceed charges by model\-dependent median factors of 1\.25–54\.85; we use the billed amounts directly, without a conversion factor\. Charges cover full agent sessions, including requests after the deadline, and exclude engine compute and unobserved inference inside tools\. BP, Python, and direct\-Anthropic Sonnet runs have no usable billing linkage and are omitted from cost comparisons\.

Input totals add the ledger’s separate uncached, cache\-read, and cache\-write fields; cache shares use that total as the denominator\. The result\-summary input field already includes cached tokens and is never added to its cache fields\.

## Appendix FTask Catalog and Paired Gameplay Specifications

Table[16](https://arxiv.org/html/2609.23142#A6.T16)lists every task in the primary study, its starting project, and its required checks\. The 50 Third Person tasks use Unreal’s standard template; the other 20 use the CraftBench template\. Task names are directory identifiers so that a reader can locate the instructions and verifier directly\. Table[17](https://arxiv.org/html/2609.23142#A6.T17)gives the requested behaviors in the 10 C\+\+/BP pairs\. Their task definitions name the same interface and verifier, while BP additionally requires saved BP graphs rather than C\+\+\.

The task bank also contains 47 additional tasks outside the primary study that were excluded due to the time and budget limit\. They do not enter the reported completion rates or trajectory counts\.

Table 16:The 70 study tasks, with their starting projects and required grading gates\. IDs refer to task directories\. Build, runtime, and inspect denote L1, L2, and L2I;✓\\checkmarkmarks a required check and — marks one not required\. The C\+\+, BP, and Python groups contain 33, 25, and 12 tasks\.RowTask IDStarting projectBuildRuntimeInspectC\+\+ source tasksC01gp\-crafting\-queueCraftBench template✓\\checkmark✓\\checkmark—C02gp\-dot\-aoe\-burn\-cppThird Person✓\\checkmark✓\\checkmark—C03gp\-double\-jump\-stamina\-cppThird Person✓\\checkmark✓\\checkmark—C04gp\-glide\-stamina\-cppThird Person✓\\checkmark✓\\checkmark—C05gp\-harvestable\-regrowCraftBench template✓\\checkmark✓\\checkmark—C06gp\-heal\-over\-time\-cppThird Person✓\\checkmark✓\\checkmark—C07gp\-health\-attribute\-ops\-cppThird Person✓\\checkmark✓\\checkmark—C08gp\-inventory\-stackingCraftBench template✓\\checkmark✓\\checkmark—C09gp\-poison\-dot\-stack\-cppThird Person✓\\checkmark✓\\checkmark—C10gp\-spawner\-populationCraftBench template✓\\checkmark✓\\checkmark—C11t0\-sanity\-log\-on\-beginplayCraftBench template✓\\checkmark✓\\checkmark—C12t1\-data\-asset\-drives\-speedCraftBench template✓\\checkmark✓\\checkmark—C13t1\-datatable\-drives\-valueCraftBench template✓\\checkmark✓\\checkmark—C14t1\-default\-cube\-mesh\-actorCraftBench template✓\\checkmark✓\\checkmark—C15t1\-extraction\-volume\-per\-actor\-triggerCraftBench template✓\\checkmark✓\\checkmark—C16t1\-gameplay\-tag\-gateCraftBench template✓\\checkmark✓\\checkmark—C17t1\-movement\-component\-drives\-actorCraftBench template✓\\checkmark✓\\checkmark—C18t1\-mud\-wade\-cppThird Person✓\\checkmark✓\\checkmark—C19t1\-overlap\-logs\-onceCraftBench template✓\\checkmark✓\\checkmark—C20t1\-overlap\-teleport\-portalThird Person✓\\checkmark✓\\checkmark—C21t1\-physics\-drop\-and\-restCraftBench template✓\\checkmark✓\\checkmark—C22t1\-screen\-tint\-cppThird Person✓\\checkmark✓\\checkmark—C23t2\-gravity\-floating\-pawn\-movementCraftBench template✓\\checkmark✓\\checkmark—C24t2\-homing\-projectileCraftBench template✓\\checkmark✓\\checkmark—C25t2\-hud\-layout\-and\-countdownThird Person✓\\checkmark✓\\checkmark—C26t2\-ladder\-climb\-volumeThird Person✓\\checkmark✓\\checkmark—C27t2\-melee\-ability\-with\-cooldownCraftBench template✓\\checkmark✓\\checkmark—C28t2\-npc\-follows\-playerThird Person✓\\checkmark✓\\checkmark—C29t2\-race\-clock\-cppThird Person✓\\checkmark✓\\checkmark—C30t2\-timeline\-color\-cycleCraftBench template✓\\checkmark✓\\checkmark—C31t2\-weapon\-fire\-animation\-on\-triggerThird Person✓\\checkmark✓\\checkmark—C32t3\-gate\-and\-door\-cppThird Person✓\\checkmark✓\\checkmark—C33tp2\-sprint\-staminaThird Person✓\\checkmark✓\\checkmark—Blueprint/native\-asset tasksB01gp\-additem\-stack\-fix\-bpThird Person✓\\checkmark✓\\checkmark—B02gp\-door\-hitch\-fix\-bpThird Person✓\\checkmark✓\\checkmark—B03gp\-dot\-aoe\-burn\-bpThird Person✓\\checkmark✓\\checkmark✓\\checkmarkB04gp\-double\-jump\-stamina\-bpThird Person✓\\checkmark✓\\checkmark✓\\checkmarkB05gp\-glide\-stamina\-bpThird Person✓\\checkmark✓\\checkmark✓\\checkmarkB06gp\-heal\-over\-time\-bpThird Person✓\\checkmark✓\\checkmark✓\\checkmarkB07gp\-health\-attribute\-ops\-bpThird Person✓\\checkmark✓\\checkmark✓\\checkmarkB08gp\-poison\-dot\-stack\-bpThird Person✓\\checkmark✓\\checkmark✓\\checkmarkB09t0\-sanity\-bp\-log\-on\-beginplayCraftBench template✓\\checkmark✓\\checkmark—B10t1\-blueprint\-event\-to\-actionCraftBench template✓\\checkmark✓\\checkmark—B11t1\-blueprint\-graph\-on\-beginplayCraftBench template✓\\checkmark✓\\checkmark—B12t1\-dawn\-fog\-lighting\-rigThird Person✓\\checkmark—✓\\checkmarkB13t1\-hero\-blueprint\-copy\-with\-flashlightThird Person✓\\checkmark—✓\\checkmarkB14t1\-mud\-wade\-bpThird Person✓\\checkmark✓\\checkmark✓\\checkmarkB15t1\-playable\-level\-bootstrapThird Person✓\\checkmark—✓\\checkmarkB16t1\-screen\-tint\-bpThird Person✓\\checkmark✓\\checkmark✓\\checkmarkB17t1\-third\-person\-chase\-cameraThird Person✓\\checkmark—✓\\checkmarkB18t1\-walk\-animation\-footstep\-cuesThird Person✓\\checkmark—✓\\checkmarkB19t2\-consistent\-enum\-namesThird Person✓\\checkmark—✓\\checkmarkB20t2\-cutscene\-camera\-push\-and\-hero\-riseThird Person✓\\checkmark—✓\\checkmarkB21t2\-datatable\-csv\-exportThird Person✓\\checkmark—✓\\checkmarkB22t2\-race\-clock\-bpThird Person✓\\checkmark✓\\checkmark✓\\checkmarkB23t2\-weapon\-held\-in\-right\-handThird Person✓\\checkmark—✓\\checkmarkB24t3\-gate\-and\-door\-bpThird Person✓\\checkmark✓\\checkmark✓\\checkmarkB25t3\-piercing\-projectileThird Person✓\\checkmark—✓\\checkmarkEditor\-scripting tasksP01kp\-anim\-track\-bakeThird Person✓\\checkmark—✓\\checkmarkP02kp\-blueprint\-actor\-audit\-reportThird Person✓\\checkmark—✓\\checkmarkP03kp\-character\-boom\-and\-movementThird Person✓\\checkmark—✓\\checkmarkP04kp\-config\-source\-auditThird Person✓\\checkmark—✓\\checkmarkP05kp\-derived\-class\-searchThird Person✓\\checkmark—✓\\checkmarkP06kp\-engine\-source\-searchThird Person✓\\checkmark—✓\\checkmarkP07kp\-fog\-and\-postprocess\-rigThird Person✓\\checkmark—✓\\checkmarkP08kp\-motion\-set\-shares\-one\-rigThird Person✓\\checkmark—✓\\checkmarkP09kp\-retarget\-maps\-two\-rigsThird Person✓\\checkmark—✓\\checkmarkP10kp\-routine\-usage\-searchThird Person✓\\checkmark—✓\\checkmarkP11kp\-spawn\-level\-actorsThird Person✓\\checkmark—✓\\checkmarkP12kp\-typed\-input\-bindingsThird Person✓\\checkmark—✓\\checkmarkTable 17:The ten gameplay specifications with C\+\+ and Blueprint counterparts\. Each pair names the same starting project, runtime map, and test class in its task definitions\. Both counterparts require build and runtime checks; the Blueprint counterpart also requires artifact inspection\. These summaries describe the requested behavior; the task prompts specify the acceptance tolerances and deliverables\.
## Appendix GComparison with Game\-Development Benchmarks

Table[18](https://arxiv.org/html/2609.23142#A7.T18)compares what agents are asked to produce and the evidence used to evaluate it\. Different task scopes support different claims: atomic gameplay checks, visual cutscene quality, and complete game generation are not interchangeable outcomes\.

Table 18:Benchmark designs, using the cited releases\. The OpenGameEval count refers to its initial release; its task bank has since expanded\. CraftBench\-UE also includes 10 C\+\+/BP gameplay pairs and evaluates two editor MCP configurations\.