GameASG-Bench：游戏开发中自主软件生成基准测试

# GameASG-Bench: Benchmarking Autonomous Software Generation for Game Development
Source: [https://arxiv.org/html/2609.21293](https://arxiv.org/html/2609.21293)
Yi Chen1Shusheng Xu1Fan Li1Huan Wang1Tongkai Yang1Binhang Yuan1,3Affiliation:1Ant Group,2Beihang University,3HKUST

###### Abstract

Autonomous software generation \(ASG\) aims to turn human requirements into executable applications, but delivering these applications does not necessarily establish that their interacting components satisfy the specified behavioral requirements\. We introduceGameASG\-Bench, a benchmark that makes behavioral testability part of the generation task for game development\. Our design declares an evaluation interface specification before generation, fixing legal starting scenarios, player\-level actions, stable snapshots, rejection behavior, and invariants while leaving private implementations open\. Concretely, we include: \(i\) static L1 checks that assess source\-level compliance; and \(ii\) browser\-executed L2 checks that combine semantic observations with real input and runtime evidence\. We implement this protocol as 47 browser\-native game\-generation tasks spanning 12 primary genres and both 2D and 3D interaction, each with executable checks and an independently verified reference implementation\. Our experiments answer four key questions about end\-to\-end agent performance, tool access and nominal turn budget, reasoning effort, and harness choice\. Across nine agent stacks, the highest observed mean L2 check pass rate is 93\.2%, yet the highest observed strict task success rate, requiring all L1 and applicable L2 prerequisite and core requirement checks, is only 55\.3% \(26/47 tasks\)\. For DeepSeek\-V4\-Flash, full tool access and larger nominal turn budgets yield more strict task successes, while the strict task success rate is not monotonic in reasoning effort\. Both tested harnesses achieve 18 strict task successes, but only ten tasks succeed under both\. These results expose task\-level compliance gaps that high average check pass rates actually obscure\.

## 1Introduction

Autonomous software generation \(ASG\) aims to translate human requirements into complete, executable software artifacts with minimal human intervention, extending the scope of agentic coding from local implementation tasks to end\-to\-end development\[[1](https://arxiv.org/html/2609.21293#bib.bib15)\]\. Modern coding agents embed large language models \(LLMs\) in loops that inspect files, edit code, invoke tools, execute programs, and revise failed implementations\[[2](https://arxiv.org/html/2609.21293#bib.bib1)\]\. Success on bounded programming problems or repository\-level edits, however, does not establish that an agent can deliver an application whose interacting components collectively satisfy the original requirements\[[1](https://arxiv.org/html/2609.21293#bib.bib15),[3](https://arxiv.org/html/2609.21293#bib.bib8)\]\. We therefore ask:how can we systematically evaluate whether frontier LLMs, operating through contemporary coding\-agent loops, can autonomously generate a complete, executable application whose runtime behavior satisfies the specified behavioral requirements?We study browser\-native game generation as a bounded and behavior\-dense proxy for autonomous requirement\-to\-artifact generation\. The resulting benchmark measures compliance with specified gameplay and evaluation requirements\.

Component\-level correctness provides an incomplete view of ASG\[[1](https://arxiv.org/html/2609.21293#bib.bib15),[4](https://arxiv.org/html/2609.21293#bib.bib39)\]\. A generated program may parse and contain plausible mechanics, yet remain unusable because its controls, shared state, rendering, and user\-interaction logic do not work together\[[3](https://arxiv.org/html/2609.21293#bib.bib8),[5](https://arxiv.org/html/2609.21293#bib.bib42)\]\. Complete games make these integration requirements observable: even a compact game must coordinate input handling, spatial behavior, state transitions, scoring or resource changes, failure conditions, and restart within one executable artifact\[[3](https://arxiv.org/html/2609.21293#bib.bib8)\]\. Evaluating these interactions helps distinguish successful delivery from partial implementation and identifies specific behavioral failures\[[6](https://arxiv.org/html/2609.21293#bib.bib10)\]\. By itself, an aggregate “playable” judgment offers limited evidence about which specified behavior failed or why, motivating checks that attribute failures to individual requirements\[[6](https://arxiv.org/html/2609.21293#bib.bib10),[7](https://arxiv.org/html/2609.21293#bib.bib43)\]\.

These integration requirements complicate both generation and evaluation\[[8](https://arxiv.org/html/2609.21293#bib.bib9),[6](https://arxiv.org/html/2609.21293#bib.bib10)\]\. During*generation*, the agent must translate a natural\-language specification into consistent state transitions, interfaces, rendering logic, and player interactions\[[8](https://arxiv.org/html/2609.21293#bib.bib9)\]\. For example, a movement routine can be locally plausible yet update the wrong entity, operate during a locked phase, consume the wrong resource, or fail to affect the rendered game\. During*evaluation*, successful compilation does not establish that the requested mechanics exist, while open\-ended playtesting can miss conditions that require long or precise interaction sequences\[[4](https://arxiv.org/html/2609.21293#bib.bib39),[6](https://arxiv.org/html/2609.21293#bib.bib10)\]\. Controlled state access makes such conditions easier to exercise, and bounded verification can combine visual traces with assertions over runtime state\[[6](https://arxiv.org/html/2609.21293#bib.bib10)\]\. In our setting, evaluator\-facing state is implemented by the generated artifact and must be checked against browser behavior; For example, in Section[4\.6](https://arxiv.org/html/2609.21293#S4.SS6), we illustrate how scenario preparation, semantic observations, and real input can fail to compose\. We therefore design an evaluator around reproducible legal starting conditions, player\-level actions, and evidence connecting state changes to browser input and execution, while preserving freedom in the artifact’s private implementation \(Section[2](https://arxiv.org/html/2609.21293#S2)\)\.

Recent benchmarks address some aspects of this problem\. Function\-level benchmarks test bounded programs\[[9](https://arxiv.org/html/2609.21293#bib.bib2),[10](https://arxiv.org/html/2609.21293#bib.bib3)\]; repository benchmarks evaluate changes to existing software\[[11](https://arxiv.org/html/2609.21293#bib.bib4)\]; and interactive\-agent benchmarks measure operation of already\-built environments\[[12](https://arxiv.org/html/2609.21293#bib.bib12)\]\. Recent game benchmarks move closer to ASG\. GameDevBench\[[13](https://arxiv.org/html/2609.21293#bib.bib5)\]evaluates complex modifications to Godot projects; WebGameBench\[[3](https://arxiv.org/html/2609.21293#bib.bib8)\]evaluates complete browser\-native games through real\-browser interaction; GameCraft\-Bench\[[8](https://arxiv.org/html/2609.21293#bib.bib9)\]evaluates end\-to\-end Godot generation using replay and multimodal evidence; and GameGen\-Verifier\[[6](https://arxiv.org/html/2609.21293#bib.bib10)\]decomposes specifications into bounded precondition\-interaction\-postcondition checks supported by runtime state injection\.

However, how to give executable checks consistent access to heterogeneous generated artifacts remains an underexplored design question\. Thus, we proposeGameASG\-Bench, which explores a complementary approach: the task author specifies behavioral testability*before generation*through a common evaluation interface, legal scenario definitions, stable semantic observations, and explicit invariants\. The coding agent implements the specified interface together with the game, allowing fixed checks to combine semantic observations with real browser input and independent runtime evidence\. Concretely, we make the following key contributions:

Contribution 1\.We designGameASG\-Benchto evaluate requirement\-to\-artifact generation through a pre\-declared evaluation interface specification \(Section[2](https://arxiv.org/html/2609.21293#S2)\)\. Legal starting scenarios, player\-level actions, stable snapshots, rejection behavior, and invariants provide fixed checks with consistent access to different private implementations\. L1 checks assess source\-level compliance; L2 checks exercise the game in headless Chromium using semantic observations, real browser input, and runtime evidence\. We distinguish these evidence layers from requirement priorities and define strict task success as successful delivery and evaluation with every L1 check and every applicable L2 prerequisite and core requirement check passing\.

Contribution 2\.We implement this design as 47 browser\-native game\-generation tasks spanning 12 primary genres and both 2D and 3D interaction \(Section[3](https://arxiv.org/html/2609.21293#S3)\)\. Each task packages a generation prompt, a gameplay design requirement, an evaluation interface specification, and executable checks, with a self\-containedindex\.htmlas the required artifact\. Task authors translate gameplay requirements into scenario, action, observation, and invariant definitions before generation\. Quality control combines independent verification of reference implementations with automated acceptance testing, connecting the shared evaluation protocol to a concrete task corpus\.

Contribution 3\.We conduct experiments to answer four research questions about end\-to\-end performance, tool access and nominal turn budget, reasoning effort, and harness choice \(Section[4](https://arxiv.org/html/2609.21293#S4)\)\. Across nine agent stacks, the highest observed mean L2 check pass rate is 93\.2%, yet the highest observed strict task success rate is 55\.3% \(26/47 tasks\), showing that high average check pass rates can mask unmet task requirements\. For DeepSeek\-V4\-Flash, full tool access yields 18 strict task successes versus 6–9 with limited tool access; the strict task success rate over all 47 planned tasks rises from 7/47 to 18/47 as the nominal turn budget increases from 30 to 120 as more tasks complete evaluation\. High reasoning effort yields 19 strict task successes versus 18 at maximum reasoning effort while using 26\.9% fewer reasoning tokens\. Both tested harnesses achieve 18 strict task successes, but only ten tasks succeed under both\.

## 2Benchmark Design and Evaluation Protocol

The core design ofGameASG\-Benchis to make testability part of the generation task\. The human developer defines an evaluation interface specification before generation, and the coding agent implements the specified interface together with the game\. Fixed checks can then prepare scenarios and exercise player\-level actions across different implementations\. This section describes the shared task structure, evaluation interface, evidence layers, and scoring rule; Section[3](https://arxiv.org/html/2609.21293#S3)explains how that design is instantiated in the benchmark corpus\. The distinguishing design choice is how responsibilities are divided among the human developer, coding agent, and evaluator, building on established ideas of behavioral interface specifications, controllability, and behavioral assertions\[[14](https://arxiv.org/html/2609.21293#bib.bib18),[15](https://arxiv.org/html/2609.21293#bib.bib19),[16](https://arxiv.org/html/2609.21293#bib.bib20),[17](https://arxiv.org/html/2609.21293#bib.bib21)\]\.

### 2\.1Task Definition and Overview

EachGameASG\-Benchtask requires a coding agent to produce a self\-contained browser\-native game inindex\.html\. The game must satisfy both its player\-visible requirements and the evaluation interface specification that makes those requirements testable\. Three documents define the task \(Table[1](https://arxiv.org/html/2609.21293#S2.T1)\): the*generation prompt*specifies the workspace and delivery protocol; the*gameplay design requirement*describes the playable loop, mechanics, feedback, and completion conditions; and the*evaluation interface specification*defines the evaluation interface and its task\-specific semantics\.

Table 1:Task documents and their roles\. All three are available to the coding agent; only the generation prompt is passed directly to the harness\.DocumentFileContentGeneration prompttarget\.mdWorkspace, delivery protocol, and concise gameplay brief\.Gameplay design requirementgame\-spec\.mdPlayer\-visible mechanics and the minimum playable loop\.Evaluation interface specificationtdd\.mdScenario setup, action and observation semantics, rejection behavior, and invariants\.The human developer prepares these documents and executable checks before generation\. Each attempt starts in a clean workspace with a fresh agent session\. The harness passestarget\.mddirectly to the coding agent, which consults the two specifications and implements the game and evaluation interface together\. The evaluator checks delivery, applies fixed L1 and L2 checks, and records itemized outcomes and strict task success\. This division specifies the controllable actions and observable behavior while leaving the private implementation open\. Harness invocations, delivery preflight, and access boundaries appear in Section[4\.1](https://arxiv.org/html/2609.21293#S4.SS1)and Appendix[B](https://arxiv.org/html/2609.21293#A2)\.

### 2\.2Evaluation Interface Specification

Each evaluation interface specification defines four methods exposed throughwindow\.\_\_gameTest:reset,loadScenario,input, andgetSnapshot\. Scenario names, actions, and snapshot fields are task\-specific\. The interface provides two complementary capabilities\.

Reproducible starting conditions\.Theresetoperation restores the initial state and clears transient effects\.loadScenariosets a documented legal state reachable through ordinary play, including rare or late\-game preconditions\. Scenario setup may adjust resources and positions to establish the required starting state, but it must not directly produce the outcome being tested\. A near\-terminal scenario, for example, must still require play to cause victory or defeat\. This constraint preserves the role of the tested action while bounding the cost of reaching its precondition\.

Actions and observations with stable meanings\.Theinputoperation performs player\-level actions such as ordering a unit or selecting a target\.getSnapshotreturns a JSON\-serializable summary of the relevant game state\. The evaluation interface specification fixes the meanings of actions and observations, including rejection behavior and invariants, while permitting different internal representations\. The agent need not expose its private object graph or reproduce a prescribed code layout\.

All four methods operate on the underlying state that drives visible gameplay\. State changes made throughreset,loadScenario, andinputare reflected in the game, whilegetSnapshotreports the corresponding state\. This connection allows the evaluator to prepare scenarios, perform actions, and inspect outcomes within the actual game\.

### 2\.3Layered Evaluation and Behavioral Evidence

Evaluation separates source\-level compliance from executed gameplay behavior\. L1 inspects the delivered artifact; L2 exercises it in a browser\. The layers run in that order, but an L1 failure does not suppress L2, so a report can distinguish a missing source declaration from a runtime failure\.

- •L1: source\-level compliance\.A shared runner inspects the HTML and its inline or directly linked local scripts\. The task’schecks\.jsondeclares structural and syntax checks, regular\-expression assertions, and anti\-pattern rules\. The runner records an outcome and diagnostic hints for each check\. These checks establish evidence of required source patterns or interface declarations, not gameplay correctness: a regular\-expression match, for example, may occur in a comment or string\.
- •L2: controlled behavioral execution\.The evaluator serves the submission locally and executeschecks\.jsin headless Chromium\. Behavioral checks use a prepare\-act\-observe sequence: they can establish a documented scenario, invoke a semantic action or send real browser input, and compare observed state changes with expected outcomes and invariants\. Prerequisite checks may only verify startup and the required interface\. Checks run sequentially, each in a newly created browser page that loads the submitted game\. After a check completes or times out, the runner closes its page before starting the next check\. A shared browser hook records animation frames, input listeners, and Canvas or WebGL activity; checks can also inspect snapshots, rendered output, and runtime errors\. Snapshots report task\-specific game state, while real input and browser observations help assess whether the reported changes are reflected in visible gameplay\. The combination of evidence sources depends on the individual check; not every check uses every signal\. Each check returnsPASS,FAIL, orNOT\_APPLICABLE, together with elapsed time and diagnostic detail\.

### 2\.4Requirement Priorities and Scoring

We use L1 and L2 to specify evidence layers, and P0, P1, and P2 to specify requirement priorities\. Prerequisite checks \(P0\) cover launch, evaluation\-interface availability, and minimum runtime requirements\. Core requirement checks \(P1\) cover required mechanics, interactions, invariants, and supporting interface requirements\. Extended capability checks \(P2\) cover additional mechanics and experience completeness\. Priorities are assigned during test authoring\. L2 checks markedNOT\_APPLICABLEare excluded when calculating applicable\-check pass rates\. The current runner does not enforce a restriction against returning this outcome for required P1 checks\.

For each planned taskgg, letdg,eg∈\{0,1\}d\_\{g\},e\_\{g\}\\in\\\{0,1\\\}indicate valid delivery and completed evaluation, respectively\. Letℒg\\mathcal\{L\}\_\{g\}contain every L1 check, and letℬg\\mathcal\{B\}\_\{g\}contain the applicable L2 P0/P1 checks\. For each checkcc,pg,c=1p\_\{g,c\}=1if the check passes and00otherwise\. Strict task successsgs\_\{g\}and the primary strict task success rate over the set of planned tasks𝒢\\mathcal\{G\}are

sg=dg​eg​∏c∈ℒg∪ℬgpg,c,SR⁡\(𝒢\)=1\|𝒢\|​∑g∈𝒢sg\.s\_\{g\}=d\_\{g\}e\_\{g\}\\\!\\prod\_\{c\\in\\mathcal\{L\}\_\{g\}\\cup\\mathcal\{B\}\_\{g\}\}p\_\{g,c\},\\qquad\\operatorname\{SR\}\(\\mathcal\{G\}\)=\\frac\{1\}\{\|\\mathcal\{G\}\|\}\\sum\_\{g\\in\\mathcal\{G\}\}s\_\{g\}\.\(1\)Thus, strict task success requires valid delivery, completed evaluation, and passing all L1 checks and all applicable L2 P0/P1 checks\. The primary strict task success rate is calculated over all planned tasks, including generation and evaluation failures, and is reported as a percentage\. When reported, conditional strict task success rates use only tasks that complete evaluation as the denominator\.

To characterize partial compliance, we also report check pass rates over tasks that complete evaluation\. Mean L1 and overall L2 check pass rates are calculated by averaging per\-task check pass rates\. For each L2 priority, the pass rate is computed by pooling applicable checks across evaluated tasks and dividing the number passed by the total\. These metrics complement strict task success by showing how many checks pass even when a task does not satisfy every required check\.

## 3Benchmark Construction

We instantiate the design in Section[2](https://arxiv.org/html/2609.21293#S2)through task selection, specification and test authoring, and quality control, yielding 47 tasks, each with executable checks and a reference implementation\. This section describes that construction process and the resulting corpus; Section[4](https://arxiv.org/html/2609.21293#S4)reports reference acceptance and agent performance\.

### 3\.1Scope and Task Selection

Each selected task must support a complete playable loop: the player’s input changes the game state and produces observable progress or a terminal outcome, and the game supports restarting\. Tasks must fit the self\-containedindex\.htmldelivery protocol\. We exclude concepts requiring a backend, user accounts, external databases, paid or private assets, unbounded multiplayer infrastructure, or behavior that cannot be reached and observed within bounded browser execution\. This scope concentrates evaluation on integration among controls, state transitions, rendering, and gameplay logic\.

### 3\.2Specification and Test Authoring

For each selected game concept, human developers write a gameplay design requirement specifying the objective, controls, entity roles, state transitions, scoring or resource effects, player feedback, terminal conditions, and restart behavior\. They then prepare the generation prompt by combining the gameplay brief with the delivery instructions, following the document structure defined in Section[2\.1](https://arxiv.org/html/2609.21293#S2.SS1)\.

Human developers then define task\-specific scenarios, actions, and snapshot fields in the evaluation interface specification\. For each tested behavior, they specify the starting conditions, the player\-level action, the expected observable outcomes, and the invariants that must hold\. For invalid actions, they specify the expected rejection behavior and which parts of the game state must remain unchanged\. Scenario setup and interface behavior follow the constraints described in Section[2\.2](https://arxiv.org/html/2609.21293#S2.SS2)\.

L1 checks are specified inchecks\.json, and L2 behavioral checks are implemented inchecks\.js\. Human developers assign requirement priorities according to Section[2\.4](https://arxiv.org/html/2609.21293#S2.SS4)\.

### 3\.3Quality Control

Quality control combines independent verification of reference implementations with automated acceptance testing\. For each task, the reference implementation is independently verified through manual inspection against the gameplay design requirements and evaluation interface specification\. Verification covers real user interactions, core state transitions, termination and restart, and consistency between the evaluation interface and visible gameplay\. The authored L1 and L2 checks are then run on the verified implementation\. Reference acceptance requires passing all L1 checks and all applicable L2 P0/P1 checks; L2 P2 outcomes are recorded separately\. This procedure uses independently verified implementations as positive controls to assess whether the test suite accepts games that satisfy the core requirements\. Section[4\.1](https://arxiv.org/html/2609.21293#S4.SS1)reports the acceptance results\.

### 3\.4Corpus Composition

The benchmark comprises 47 tasks spanning 12 primary genres, including 32 tasks with 2D environments and 15 with 3D environments\. Task diversity is described along three axes: genre, dimension, and reference technology \(Figure[1](https://arxiv.org/html/2609.21293#S3.F1)\)\.*Genre*describes the repeated core action and objective;*dimension*describes the playable environment’s spatial relations; and*technology*describes the rendering route of the reference implementation\. Most reference implementations use Canvas 2D or Three\.js\. Technology labels characterize the corpus and do not constrain compliant generated artifacts\.

\(a\)Genre\(b\)Dimension\(c\)Reference technology

Figure 1:Composition of the 47 tasks\. Each panel covers all 47 tasks; legend entries give counts and percentages\. RTS and TBS denote real\-time and turn\-based strategy, respectively\. Reference technology identifies the rendering technology used by each reference implementation\.Across the 47 tasks, the test suite contains 336 L1 checks and 885 L2 checks, including 534 L2 core requirement checks \(P1\)\. Table[2](https://arxiv.org/html/2609.21293#S3.T2)summarizes the checks using the layers and priorities defined in Section[2](https://arxiv.org/html/2609.21293#S2)\.

Table 2:Check counts for the 47 tasks used in all reported experiments\. Layer and priority describe different properties of a check\.L1: source\-level checksL2: behavioral checksTool checks43Prerequisite checks \(P0\)102Regular\-expression checks288Core requirement checks \(P1\)534Anti\-pattern checks5Extended capability checks \(P2\)249Total336Total885

## 4Experimental Evaluation

The experimental evaluation is designed to answer four key research questions:

- •\(RQ1\)How well do contemporary agent stacks satisfy end\-to\-end task requirements?
- •\(RQ2\)How do tool access and nominal turn budget affect task performance?
- •\(RQ3\)How does reasoning effort affect task performance and cost?
- •\(RQ4\)How does harness choice affect task performance for a fixed model and reasoning\-effort setting?

We first describe the shared experimental setup and report acceptance results for the reference implementations\. We then address the four research questions in turn, followed by diagnostic case analyses\.

### 4\.1Experimental Setup

Tasks and configurations\.All experiments use the same 47 tasks and task packages\. Each task\-configuration pair is run once from a clean workspace\. Evaluation uses headless Chromium with a fixed1280×8001280\\times 800viewport\. An*agent stack*is a model paired with its coding harness: Claude Code 2\.1\.206 or Codex CLI 0\.153\.4\. RQ1 uses each stack’s maximum reasoning\-effort setting and full tool access\. RQ2–RQ4 reuse DeepSeek\-V4\-Flash with Claude Code, maximum reasoning effort, full tool access, and a nominal turn budget of 120 as the common baseline\. Each subsection specifies the changes to this baseline\.

Evaluation metrics\.The primary metric is the strict task success rate over all 47 planned tasks, as defined in Equation[1](https://arxiv.org/html/2609.21293#S2.E1)\. We also report check pass rates over tasks that complete evaluation\. Mean L1 and overall L2 check pass rates are calculated by averaging per\-task check pass rates; L2 check pass rates for each priority are computed by pooling applicable checks within each priority and dividing the number passed by the total\. We additionally report the number of tasks that deliver artifacts and complete evaluation, along with available measurements of artifact size, token use, and monetary cost\.

Reference\-implementation validation\.Following the verification procedure in Section[3\.3](https://arxiv.org/html/2609.21293#S3.SS3), we evaluated one independently verified core\-compliant reference implementation for each of the 47 tasks\. All 47 passed all L1 checks and all applicable L2 P0/P1 checks\.

### 4\.2RQ1: End\-to\-End Agent Performance

RQ1 asks how reliably contemporary coding agents satisfy the complete task requirements\. We compare nine agent stacks using the same task packages and evaluation protocol\. All nine stacks deliver artifacts and complete evaluation on all 47 tasks\. Table[3](https://arxiv.org/html/2609.21293#S4.T3)reports strict task success and check pass rates, and Table[4](https://arxiv.org/html/2609.21293#S4.T4)reports resource use\.

Table 3:End\-to\-end results at maximum reasoning effort\. Strict task success is reported as the number of successful tasks out of 47 and the corresponding rate \(%\); all other entries are check pass rates \(%\)\. L1 and overall L2 entries are mean per\-task check pass rates; L2 P0–P2 entries pool applicable checks within each priority\. Bold denotes the highest observed value in each column\.ModelHarnessStrict tasksuccessL1L2MeanP0P1P2GPT\-6\-AstraCodex CLI26/47 \(55\.3\)98\.093\.299\.092\.792\.4Claude\-Opus\-5Claude Code24/47 \(51\.1\)99\.690\.491\.289\.991\.1GPT\-5\.6\-SolCodex CLI21/47 \(44\.7\)98\.991\.3100\.089\.791\.9DeepSeek\-V4\-FlashClaude Code18/47 \(38\.3\)99\.489\.398\.086\.790\.2DeepSeek\-V4\-ProClaude Code15/47 \(31\.9\)99\.685\.285\.383\.988\.9Kimi\-K3Claude Code15/47 \(31\.9\)97\.788\.598\.086\.389\.7GLM\-5\.2Claude Code11/47 \(23\.4\)98\.884\.691\.283\.383\.0Hunyuan\-3Claude Code10/47 \(21\.3\)98\.381\.796\.178\.282\.3MiniMax\-M3Claude Code7/47 \(14\.9\)98\.370\.691\.264\.575\.4Table 4:Mean artifact size and reported resource use\. Artifact sizes cover all 47 tasks per stack\. Input tokens combine the reported input, cache\-read, and cache\-write categories\.ModelHarnessArtifact size\(KB\)Input tokens\(k\)Output tokens\(k\)Monetarycost\(USD\)GPT\-6\-AstraCodex CLI78\.82896\.155\.54\.4445Claude\-Opus\-5Claude Code68\.5212,632\.6142\.411\.8500GPT\-5\.6\-SolCodex CLI64\.671,382\.545\.71\.7867DeepSeek\-V4\-FlashClaude Code85\.2113,038\.4203\.60\.2461DeepSeek\-V4\-ProClaude Code90\.3412,944\.3202\.90\.7431Kimi\-K3Claude Code57\.511,263\.972\.32\.5518GLM\-5\.2Claude Code60\.331,067\.147\.00\.6184Hunyuan\-3Claude Code52\.882,474\.588\.90\.1779MiniMax\-M3Claude Code81\.764,918\.971\.10\.4069We identify three key observations:

High source\-level pass rates coexist with incomplete task compliance\.Mean L1 check pass rates range from 97\.7% to 99\.6%, whereas strict task success rates range from 14\.9% to 55\.3% \(Table[3](https://arxiv.org/html/2609.21293#S4.T3)\)\. GPT\-6\-Astra with Codex CLI achieves the highest observed strict task success rate, at 26/47 \(55\.3%\)\. Only two stacks succeed on more than half of the tasks, and even the stack with the highest strict task success rate fails to satisfy all required checks on 21 tasks\. High average source\-level check pass rates therefore do not translate into equally high strict task success rates\.

Average check pass rates and strict task success capture different outcomes\.GPT\-6\-Astra achieves the highest observed mean L2 check pass rate \(93\.2%\) and L2 P1 and P2 check pass rates \(92\.7% and 92\.4%, respectively\), yet its strict task success rate is 55\.3%; only 40 of its 47 artifacts pass every L1 check\. Rankings can also differ: GPT\-5\.6\-Sol has a higher mean L2 check pass rate than Claude\-Opus\-5 \(91\.3% versus 90\.4%\) but fewer strict task successes \(21 versus 24\)\. These results support reporting both check pass rates and strict task success: even a single failed required check prevents an artifact with otherwise high check pass rates from achieving strict task success\.

Larger artifacts and higher token usage do not consistently correspond to more strict task successes\.GPT\-6\-Astra records 55\.5k mean output tokens and achieves 26 strict task successes, compared with approximately 203k output tokens for each DeepSeek variant and 18 and 15 strict task successes for Flash and Pro, respectively\. MiniMax\-M3 produces larger artifacts than Claude\-Opus\-5 \(81\.76 versus 68\.52 KB\) but achieves 7 rather than 24 strict task successes\. DeepSeek\-V4\-Pro produces the largest artifacts \(90\.34 KB\) and achieves 15 strict task successes\. These contrasts show that larger artifacts or higher reported token usage do not consistently correspond to more complete task compliance\.

RQ1 key observation\.Across nine stacks, mean L1 check pass rates range from 97\.7% to 99\.6%, but strict task success rates range from 14\.9% to 55\.3%\. High average check pass rates do not ensure that an artifact satisfies every required check\.

### 4\.3RQ2: Effect of Tool Access and Nominal Turn Budget

RQ2 examines DeepSeek\-V4\-Flash under different levels of tool access and nominal turn budgets\. Both comparisons reuse the Claude Code baseline with full tool access, maximum reasoning effort, and a nominal turn budget of 120\. The tool comparison covers four configurations: no tools \(serialized API\), file read/write only, files \+ syntax checking, and full tool access\. The no\-tools configuration uses serialized API input instead of the file\-based harness workflow\. The budget comparison uses full tool access with nominal turn budgets of 30, 60, and 120\.

Table 5:Tool configurations for DeepSeek\-V4\-Flash\. All 47 tasks complete evaluation in each condition\. Strict task success is reported as the number of successful tasks out of 47 and the corresponding rate \(%\); check pass rates are percentages\. Syntax checking permits Bash fornode \-\-check, but no browser\. The configuration with full tool access uses the nominal 120\-turn baseline\.Tool settingStrict tasksuccessL1L2 P0L2 P1L2 P2No tools \(serialized API\)7/47 \(14\.9\)98\.583\.365\.380\.1File read/write only6/47 \(12\.8\)98\.887\.366\.273\.4Files \+ syntax checking9/47 \(19\.1\)98\.286\.363\.469\.3Full tool access18/47 \(38\.3\)99\.498\.086\.790\.2Full tool access achieves the most strict task successes and the highest check pass rates\.Full tool access achieves 18 strict task successes, compared with 6–9 under restricted tool configurations, including 7 in the no\-tools condition \(Table[5](https://arxiv.org/html/2609.21293#S4.T5)\)\. It also achieves the highest P1 check pass rate, at 86\.7%, while L1 check pass rates remain close to 99% across configurations\. Adding syntax checking to file read/write access increases the number of strict task successes from 6 to 9, while the P1 check pass rate decreases from 66\.2% to 63\.4%\. Thus, improvements in the number of strict task successes and check pass rates do not consistently coincide across the restricted configurations\. Appendix[A\.2](https://arxiv.org/html/2609.21293#A1.SS2)presents examples of tool\-assisted testing and repair during generation\.

Table 6:Nominal turn\-budget comparison for DeepSeek\-V4\-Flash with full tool access\. The primary strict task success rate uses all 47 planned tasks as the denominator; conditional strict task success rates and check pass rates describe only the evaluated subset\. Counts are followed by percentages in parentheses; check pass rates are percentages\.Nominal turnbudgetEvaluated/ plannedStrict task successL1L2 P0L2 P1L2 P2All plannedConditional3010/477/47 \(14\.9\)7/10 \(70\.0\)100\.0100\.094\.097\.66032/4713/47 \(27\.7\)13/32 \(40\.6\)99\.788\.681\.688\.512047/4718/47 \(38\.3\)18/47 \(38\.3\)99\.498\.086\.790\.2#### Larger nominal turn budgets increase evaluation completion and strict task success rates\.

With nominal turn budgets of 30, 60, and 120, respectively, 10, 32, and 47 tasks complete evaluation, including 7, 13, and 18 strict task successes \(Table[6](https://arxiv.org/html/2609.21293#S4.T6)\)\. Over all 47 planned tasks, the strict task success rate therefore increases from 14\.9% to 27\.7% and 38\.3%\. Among the 10, 32, and 47 evaluated tasks, the corresponding conditional strict task success rates are 70\.0%, 40\.6%, and 38\.3%, respectively\. Claude Code stops execution when the configured turn limit is reached\. With smaller nominal turn budgets, more runs terminate before completing generation and delivery, reducing the number of tasks that reach evaluation\. Appendix[A\.3](https://arxiv.org/html/2609.21293#A1.SS3)provides examples of where these runs stop during verification and delivery\.

RQ2 key observation\.Full tool access achieves 18 strict task successes out of 47 tasks, compared with 6–9 under restricted tool configurations\. Increasing the nominal turn budget from 30 to 120 raises the strict task success rate from 14\.9% to 38\.3%, while the number of tasks completing evaluation increases from 10 to 47\.

### 4\.4RQ3: Effect of Reasoning Effort

To explore RQ3, we fix DeepSeek\-V4\-Flash, Claude Code, full tool access, and a nominal turn budget of 120, and compare low, high, and maximum reasoning effort on the same 47 tasks\. We measure computational cost by reasoning\-token use\. Table[7](https://arxiv.org/html/2609.21293#S4.T7)reports strict task success, check pass rates, and mean reasoning\-token use\.

Table 7:Reasoning\-effort comparison for DeepSeek\-V4\-Flash\. Strict task success is reported as the number of successful tasks out of 47 and the corresponding rate \(%\); check pass rates are percentages\. Reasoning\-token counts are averaged across tasks\. Bold denotes the highest observed number of strict task successes or check pass rate\.EffortStrict tasksuccessL1L2 P0L2 P1L2 P2ReasoningtokensLow7/47 \(14\.9\)98\.393\.171\.780\.237,699High19/47 \(40\.4\)99\.497\.182\.788\.9138,232Max18/47 \(38\.3\)99\.498\.086\.790\.2189,210Core check pass rates increase, but the strict task success rate is not monotonic\.The strict task success rate rises from 14\.9% \(7/47 tasks\) at low effort to 40\.4% \(19/47\) at high effort, then decreases to 38\.3% \(18/47\) at maximum effort\. P1 check pass rates increase from 71\.7% to 82\.7% and 86\.7%, a low\-to\-maximum gain of 15\.0 percentage points, the largest among the reported check pass rates\. L1 changes from 98\.3% to 99\.4%, and P0 and P2 also increase\. The principal observed improvement is therefore in core runtime requirements\. Maximum effort achieves the highest P0, P1, and P2 check pass rates, while high effort achieves the most strict task successes\.

High effort achieves more strict task successes with fewer reasoning tokens than maximum effort\.Mean reasoning\-token use increases from 37,699 at low effort to 138,232 at high effort and 189,210 at maximum effort\. High effort uses 3\.67 times as many reasoning tokens as low effort, alongside an increase from 7 to 19 strict task successes\. Compared with maximum effort, high effort uses 26\.9% fewer reasoning tokens while achieving one more strict task success \(19 versus 18\)\. This identifies a useful trade\-off in the recorded runs\.

RQ3 key observation\.High effort achieves the most strict task successes \(19/47\) with 26\.9% fewer reasoning tokens than maximum effort\. Maximum effort achieves the highest P0, P1, and P2 check pass rates but yields 18/47 strict task successes\.

### 4\.5RQ4: Effect of Harness Choice

To answer RQ4, we run DeepSeek\-V4\-Flash at maximum reasoning effort through Claude Code 2\.1\.206 and Codex CLI 0\.153\.4, using the same 47 task packages, evaluation protocol, and full tool access\. We compare the harnesses as complete execution environments, including their system instructions, context management, tool schemas, command execution, and endpoint protocols\.

Table 8:Harness comparison for DeepSeek\-V4\-Flash at maximum reasoning effort\. Strict task success is reported as the number of successful tasks out of 47 and the corresponding rate \(%\); check pass rates are percentages\. Bold denotes the highest observed value in each column\.HarnessStrict tasksuccessL1L2 P0L2 P1L2 P2Claude Code18/47 \(38\.3\)99\.498\.086\.790\.2Codex CLI18/47 \(38\.3\)99\.296\.187\.192\.2Equal numbers of strict task successes conceal different task outcomes\.Both harnesses achieve strict task success on 18/47 tasks \(38\.3%\), but only ten tasks succeed under both\. Eight succeed only under Claude Code, eight only under Codex CLI, and 21 under neither\. The same aggregate count thus arises from different successful task sets\. Check pass rates also differ across the two harnesses \(Table[8](https://arxiv.org/html/2609.21293#S4.T8)\): Claude Code has higher L1 \(99\.4% versus 99\.2%\) and P0 \(98\.0% versus 96\.1%\) pass rates, whereas Codex CLI has higher P1 \(87\.1% versus 86\.7%\) and P2 \(92\.2% versus 90\.2%\) pass rates\.

Resource profiles differ despite equal numbers of strict task successes\.With the same number of strict task successes, Codex CLI produces smaller artifacts on average than Claude Code \(69\.5 versus 85\.2 KB, an 18\.4% reduction\) and records lower mean output\-token use \(112\.8k versus 203\.6k, a 44\.6% reduction\)\.

Matched traces illustrate different repair sequences\.The paired DeepSeek\-V4\-FlashSortietraces show both harnesses using tests to guide repairs\. In Codex CLI, real\-input testing exposes a coordinate offset after evaluation\-interface self\-tests pass\. In Claude Code, self\-tests reveal item states incorrectly preserved across reset, followed by edits to animation and audio initialization\. Both final artifacts achieve strict task success\. Appendix[A\.2](https://arxiv.org/html/2609.21293#A1.SS2)details these repair sequences\.

RQ4 key observation\.Both harnesses achieve 18 strict task successes out of 47 tasks, with only ten tasks achieving strict task success under both\. Equal numbers of strict task successes coexist with different successful task sets, check pass rates, and resource use\.

### 4\.6Diagnostic Analysis

The preceding experiments report strict task success and check pass rates\. We now examine concrete failures to show how real\-input checks and scenario\-based behavioral checks identify violations of task requirements \(Table[9](https://arxiv.org/html/2609.21293#S4.T9)\)\. These cases connect the evaluator’s design to the errors detected in generated artifacts\. Appendix[A\.1](https://arxiv.org/html/2609.21293#A1.SS1)provides the check\-level evidence and supplementary failure analysis\.

Table 9:Diagnostic evidence linking evaluation checks to observed failures\. The GPT\-6\-Astra case uses Codex CLI; the DeepSeek\-V4\-Flash case uses Claude Code with maximum reasoning effort and full tool access\.Artifact / stackObserved evidenceRole of the checksDiner Dasher
GPT\-6\-AstraL1 and L2 P0/P2 pass; L2 P1 mouse and touch drag checks fail because real drags produce no service progress\.Real\-input checks expose a core interaction failure despite other passing checks\.Turbo Smash Beast
DeepSeek\-V4\-FlashReset or scenario loading stops natural\-time simulation; acceleration, coasting, and related real\-input checks fail\.Checks combining scenario preparation and real input expose a failure in their interaction\.Real\-input checks expose failures in player interaction\.In the GPT\-6\-Astra implementation ofDiner Dasher, mouse and touch drags should advance the service workflow\. The artifact passes L1 and L2 P0/P2 checks, but fails the L2 P1 checks for serving customers through real mouse and touch drags: neither interaction produces service progress\. These core requirement checks test whether player actions produce the required gameplay outcome\.

Scenario\-based checks expose failures in combined operations\.In the DeepSeek\-V4\-Flash/Claude Code implementation ofTurbo Smash Beast, resetting or loading a scenario through the evaluation interface suppresses natural\-time simulation, leaving real driving inputs unable to advance the vehicle\. The evaluator records failures of acceleration, coasting, and related real\-input behavior\. The generation traces explain the mismatch: self\-tests through the evaluation interface explicitly advance time, while a successful driving test starts from a freshly loaded page\. Neither exercises the failing combination of scenario preparation, real input, and natural time progression \(Appendix[A\.1](https://arxiv.org/html/2609.21293#A1.SS1)\)\. Executing this combined sequence exposes an interaction failure that the separate self\-tests miss\.

These cases provide empirical support for the evaluator’s design by showing how complementary checks identify concrete violations of task requirements\. Real\-input checks test player interactions, and scenario\-based execution reveals inconsistencies between prepared states and subsequent gameplay\.

## 5Related Work

### 5\.1From Code Generation to Complete Artifacts

Autonomous software generation combines code synthesis with the ability to inspect, execute, and revise an artifact\. The Codex model\[[18](https://arxiv.org/html/2609.21293#bib.bib22)\]and CodeRL\[[19](https://arxiv.org/html/2609.21293#bib.bib23)\]establish language\-model synthesis and execution\-guided learning, while SWE\-Agent\[[2](https://arxiv.org/html/2609.21293#bib.bib1)\], AutoCodeRover\[[20](https://arxiv.org/html/2609.21293#bib.bib24)\], and ChatDev\[[21](https://arxiv.org/html/2609.21293#bib.bib25)\]develop interfaces, program analysis, and role\-based collaboration for longer development workflows\. These systems motivate evaluating the artifact produced by the entire coding\-agent loop, since the delivered artifact reflects both model capabilities and the surrounding development tools\.

Evaluation spans increasingly broad artifact scopes, from bounded programs and repository changes to complete applications\. APPS\[[9](https://arxiv.org/html/2609.21293#bib.bib2)\]and EvalPlus\[[10](https://arxiv.org/html/2609.21293#bib.bib3)\]test bounded programs\. At the repository level, SWE\-bench\[[11](https://arxiv.org/html/2609.21293#bib.bib4)\]and FEA\-Bench\[[22](https://arxiv.org/html/2609.21293#bib.bib11)\]test changes to existing repositories\. CodeFlowBench\[[23](https://arxiv.org/html/2609.21293#bib.bib35)\]and KoCo\-Bench\[[24](https://arxiv.org/html/2609.21293#bib.bib36)\]extend evaluation to reuse, dependencies, and domain knowledge\. At the application level, benchmarks including WebGen\-Bench\[[25](https://arxiv.org/html/2609.21293#bib.bib13)\], E2EDev\[[1](https://arxiv.org/html/2609.21293#bib.bib15)\], RAL\-Bench\[[26](https://arxiv.org/html/2609.21293#bib.bib14)\], and SaaSBench\[[27](https://arxiv.org/html/2609.21293#bib.bib37)\]address broader integration requirements; Vision2Web\[[28](https://arxiv.org/html/2609.21293#bib.bib16)\]and VISTA\[[29](https://arxiv.org/html/2609.21293#bib.bib17)\]also emphasize visual and interactive evidence\.GameASG\-Benchshares this application\-level perspective and concentrates on input, runtime state, rendering, and gameplay rules in a self\-contained browser\-native artifact\.

Beyond the scope of the generated artifact, recent work also studies how to diagnose behavioral failures and express verification requirements\. VideoVIBE\[[7](https://arxiv.org/html/2609.21293#bib.bib43)\]evaluates fine\-grained failure diagnosis from human\-operated recordings of generated webpages, with source code as complementary context\. FlowCheck\[[30](https://arxiv.org/html/2609.21293#bib.bib44)\]expresses user\-visible information\-flow constraints and compiles them into deterministic CodeQL analyses\. This emphasis on diagnosing failures and making verification requirements explicit complements our focus on checking generated games against specified runtime requirements\.

### 5\.2Game Generation and Execution Feedback

Game\-generation systems address two complementary problems: coordinating construction and obtaining useful feedback\. LLMGG\[[31](https://arxiv.org/html/2609.21293#bib.bib26)\]studies joint rule\-and\-level synthesis; GameGPT\[[32](https://arxiv.org/html/2609.21293#bib.bib27)\], ChatGE\[[33](https://arxiv.org/html/2609.21293#bib.bib28)\], and AutoUE\[[34](https://arxiv.org/html/2609.21293#bib.bib29)\]organize development through specialized roles, conversation, or engine\-grounded workflows\. OpenGame\[[35](https://arxiv.org/html/2609.21293#bib.bib7)\]combines agentic game development with execution\-grounded post\-training\.

Execution feedback connects generated code to player\-visible behavior\. CreativeGame\[[36](https://arxiv.org/html/2609.21293#bib.bib30)\]combines mechanic\-guided planning with programmatic rewards and runtime validation; Play2Code\[[37](https://arxiv.org/html/2609.21293#bib.bib31)\]alternates coding and GUI\-agent playtesting; and ALIVE\[[38](https://arxiv.org/html/2609.21293#bib.bib32)\]turns automated play into learning signals\. GameCWM distillation\[[39](https://arxiv.org/html/2609.21293#bib.bib33)\]and The Verifier is the Curriculum\[[40](https://arxiv.org/html/2609.21293#bib.bib34)\]further use verification to support model training\. These approaches use execution and verification feedback to improve generation or learning\.GameASG\-Benchevaluates the resulting artifacts against requirements and an evaluation interface specification fixed before generation, and studies how tool access, nominal turn budget, reasoning effort, and harness choice relate to strict task success\.

### 5\.3Behavioral Evaluation of Generated Games

Game benchmarks differ in both construction scope and the mechanism used to exercise an artifact\. GameDevBench\[[13](https://arxiv.org/html/2609.21293#bib.bib5)\]and GameEngineBench\[[41](https://arxiv.org/html/2609.21293#bib.bib41)\]evaluate scoped development within Godot and Unreal Engine projects\. JamBench\[[42](https://arxiv.org/html/2609.21293#bib.bib40)\]includes theme\-driven project generation and completion at several code granularities, while V\-GameGym\[[43](https://arxiv.org/html/2609.21293#bib.bib6)\]studies text\-to\-Pygame generation\. PlayEval\[[44](https://arxiv.org/html/2609.21293#bib.bib38)\], PlaytestArena\[[37](https://arxiv.org/html/2609.21293#bib.bib31)\], Mage\[[4](https://arxiv.org/html/2609.21293#bib.bib39)\], and OpenGame\-Bench\[[35](https://arxiv.org/html/2609.21293#bib.bib7)\]assess interactive artifacts using combinations of play, runtime, structural, and visual evidence\.

Related evaluators differ in how evaluation criteria are defined and how artifacts are exercised to obtain evidence\. WebGameBench\[[3](https://arxiv.org/html/2609.21293#bib.bib8)\]uses specification\-guided browser interaction and permits candidate state preparation before the final user\-level action\. GameCraft\-Bench\[[8](https://arxiv.org/html/2609.21293#bib.bib9)\]requires complete Godot projects and replayable demonstrations, including scenario initialization, and scores replay evidence against a hidden rubric\. GameGen\-Verifier\[[6](https://arxiv.org/html/2609.21293#bib.bib10)\]extracts precondition\-interaction\-postcondition keypoints and grounds them in each generated implementation through runtime state injection\. GameXpert\-Bench\[[5](https://arxiv.org/html/2609.21293#bib.bib42)\]broadens evaluation to game generation, repair, and iterative refinement\. Its generation track constructs shared event rubrics after generation by pooling events from completed artifacts and incorporating human review, then verifies them through code inspection and live interaction\.

GameASG\-Benchfixes the evaluation interface specification and executable checks before generation\. Task authors define scenario, action, and observation semantics, which agents implement alongside the game\. This enables the same checks to evaluate artifacts with different internal implementations\. Interface\-based state observations, together with real\-input and browser evidence, help assess whether the required behavior is reflected in actual gameplay\.

## 6Conclusion

In this paper, we introduceGameASG\-Bench, which makes behavioral testability part of autonomous game generation through an evaluation interface specification declared before generation\. Its 47 browser\-native tasks combine source\-level checks with browser\-executed checks using semantic observations, real input, and runtime evidence\. Across nine agent stacks, the highest observed mean L2 check pass rate is 93\.2%, yet the highest observed strict task success rate is only 55\.3%, exposing the gap between passing most checks and satisfying every required check\. Comparisons of tool access, nominal turn budget, reasoning effort, and harness choice further reveal configuration\-dependent outcomes\. The benchmark provides a common protocol and task\-level diagnostics for studying requirement compliance\.

## References

- \[1\]J\. Liu, C\. Huang, Z\. Guan, W\. Lei, and Y\. Deng\(2026\)E2EDev: benchmarking large language models in end\-to\-end software development task\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 35032–35068\.Cited by:[§1](https://arxiv.org/html/2609.21293#S1.p1.1),[§1](https://arxiv.org/html/2609.21293#S1.p2.1),[§5\.1](https://arxiv.org/html/2609.21293#S5.SS1.p2.1)\.
- \[2\]J\. Yang, C\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press\(2024\)Swe\-agent: agent\-computer interfaces enable automated software engineering\.Advances in Neural Information Processing Systems37,pp\. 50528–50652\.Cited by:[§1](https://arxiv.org/html/2609.21293#S1.p1.1),[§5\.1](https://arxiv.org/html/2609.21293#S5.SS1.p1.1)\.
- \[3\]W\. Zhang, G\. You, H\. Zhao, T\. Zhu, H\. Wang, X\. Tang, M\. Dai, J\. Gu, D\. Dong, J\. Wu,et al\.\(2026\)WebGameBench: requirement\-to\-application evaluation for coding agents via browser\-native games\.arXiv preprint arXiv:2605\.17637\.Cited by:[§1](https://arxiv.org/html/2609.21293#S1.p1.1),[§1](https://arxiv.org/html/2609.21293#S1.p2.1),[§1](https://arxiv.org/html/2609.21293#S1.p4.1),[§5\.3](https://arxiv.org/html/2609.21293#S5.SS3.p2.1)\.
- \[4\]H\. X\. Liu and K\. Tatar\(2026\)Mage: multi\-axis evaluation of llm\-generated executable game scenes beyond compile\-pass rate\.arXiv preprint arXiv:2605\.07342\.Cited by:[§1](https://arxiv.org/html/2609.21293#S1.p2.1),[§1](https://arxiv.org/html/2609.21293#S1.p3.1),[§5\.3](https://arxiv.org/html/2609.21293#S5.SS3.p1.1)\.
- \[5\]K\. Chen, H\. Hong, P\. Gao, J\. Lin, T\. Luo, Y\. Xie, C\. Liu, J\. He, Z\. Liu, and Z\. Zeng\(2026\)GameXpert\-bench: how far are coding agents from expert game development?\.arXiv preprint arXiv:2608\.21833\.Cited by:[§1](https://arxiv.org/html/2609.21293#S1.p2.1),[§5\.3](https://arxiv.org/html/2609.21293#S5.SS3.p2.1)\.
- \[6\]C\. Jia, R\. Wan, T\. Sun, W\. Tan, B\. Wan, Y\. Tong, G\. Sheng, and H\. Xu\(2026\)GameGen\-verifier: parallel keypoint\-based verification for llm\-generated games via runtime state injection\.arXiv preprint arXiv:2605\.07442\.Cited by:[§1](https://arxiv.org/html/2609.21293#S1.p2.1),[§1](https://arxiv.org/html/2609.21293#S1.p3.1),[§1](https://arxiv.org/html/2609.21293#S1.p4.1),[§5\.3](https://arxiv.org/html/2609.21293#S5.SS3.p2.1)\.
- \[7\]J\. Xu, Y\. Zhou, J\. Liao, Y\. Bai, J\. Zhou, C\. Liu, C\. Yuan, B\. Wang, and Q\. Liu\(2026\)VideoVIBE: a video\-grounded diagnostic benchmark for one\-shot interactive website generation\.arXiv preprint arXiv:2608\.09573\.Cited by:[§1](https://arxiv.org/html/2609.21293#S1.p2.1),[§5\.1](https://arxiv.org/html/2609.21293#S5.SS1.p3.1)\.
- \[8\]T\. Luo, R\. Wang, J\. Bi, C\. Xu, Z\. Tang, J\. Chen, J\. Liang, K\. Ji, S\. Guo, Y\. Du,et al\.\(2026\)GameCraft\-bench: can agents build playable games end\-to\-end in a real game engine?\.arXiv preprint arXiv:2606\.17861\.Cited by:[§1](https://arxiv.org/html/2609.21293#S1.p3.1),[§1](https://arxiv.org/html/2609.21293#S1.p4.1),[§5\.3](https://arxiv.org/html/2609.21293#S5.SS3.p2.1)\.
- \[9\]D\. Hendrycks, S\. Basart, S\. Kadavath, M\. Mazeika, A\. Arora, E\. Guo, C\. Burns, S\. Puranik, H\. He, D\. Song,et al\.\(2021\)Measuring coding challenge competence with apps\.arXiv preprint arXiv:2105\.09938\.Cited by:[§1](https://arxiv.org/html/2609.21293#S1.p4.1),[§5\.1](https://arxiv.org/html/2609.21293#S5.SS1.p2.1)\.
- \[10\]J\. Liu, C\. S\. Xia, Y\. Wang, and L\. Zhang\(2023\)Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation\.Advances in neural information processing systems36,pp\. 21558–21572\.Cited by:[§1](https://arxiv.org/html/2609.21293#S1.p4.1),[§5\.1](https://arxiv.org/html/2609.21293#S5.SS1.p2.1)\.
- \[11\]C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan\(2024\)Swe\-bench: can language models resolve real\-world github issues?\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 54107–54157\.Cited by:[§1](https://arxiv.org/html/2609.21293#S1.p4.1),[§5\.1](https://arxiv.org/html/2609.21293#S5.SS1.p2.1)\.
- \[12\]S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried,et al\.\(2024\)Webarena: a realistic web environment for building autonomous agents\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 15585–15606\.Cited by:[§1](https://arxiv.org/html/2609.21293#S1.p4.1)\.
- \[13\]W\. Chi, Y\. Fang, A\. Yayavaram, S\. Yayavaram, S\. Karten, Q\. A\. Wei, R\. Chen, A\. Wang, V\. Chen, A\. Talwalkar,et al\.\(2026\)Gamedevbench: evaluating agentic capabilities through game development\.arXiv preprint arXiv:2602\.11103\.Cited by:[§1](https://arxiv.org/html/2609.21293#S1.p4.1),[§5\.3](https://arxiv.org/html/2609.21293#S5.SS3.p1.1)\.
- \[14\]B\. Meyer\(1992\)Applying’design by contract’\.Computer25\(10\),pp\. 40–51\.Cited by:[§2](https://arxiv.org/html/2609.21293#S2.p1.1)\.
- \[15\]J\. Hatcliff, G\. T\. Leavens, K\. R\. M\. Leino, P\. Müller, and M\. Parkinson\(2012\)Behavioral interface specification languages\.ACM Computing Surveys \(CSUR\)44\(3\),pp\. 1–58\.Cited by:[§2](https://arxiv.org/html/2609.21293#S2.p1.1)\.
- \[16\]R\. S\. Freedman\(1991\)Testability of software components\.IEEE transactions on Software Engineering17\(6\),pp\. 553–564\.Cited by:[§2](https://arxiv.org/html/2609.21293#S2.p1.1)\.
- \[17\]E\. T\. Barr, M\. Harman, P\. McMinn, M\. Shahbaz, and S\. Yoo\(2014\)The oracle problem in software testing: a survey\.IEEE transactions on software engineering41\(5\),pp\. 507–525\.Cited by:[§2](https://arxiv.org/html/2609.21293#S2.p1.1)\.
- \[18\]M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. D\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§5\.1](https://arxiv.org/html/2609.21293#S5.SS1.p1.1)\.
- \[19\]H\. Le, Y\. Wang, A\. D\. Gotmare, S\. Savarese, and S\. C\. H\. Hoi\(2022\)Coderl: mastering code generation through pretrained models and deep reinforcement learning\.Advances in Neural Information Processing Systems35,pp\. 21314–21328\.Cited by:[§5\.1](https://arxiv.org/html/2609.21293#S5.SS1.p1.1)\.
- \[20\]Y\. Zhang, H\. Ruan, Z\. Fan, and A\. Roychoudhury\(2024\)Autocoderover: autonomous program improvement\.InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis,pp\. 1592–1604\.Cited by:[§5\.1](https://arxiv.org/html/2609.21293#S5.SS1.p1.1)\.
- \[21\]C\. Qian, W\. Liu, H\. Liu, N\. Chen, Y\. Dang, J\. Li, C\. Yang, W\. Chen, Y\. Su, X\. Cong,et al\.\(2024\)Chatdev: communicative agents for software development\.InProceedings of the 62nd annual meeting of the association for computational linguistics \(volume 1: Long papers\),pp\. 15174–15186\.Cited by:[§5\.1](https://arxiv.org/html/2609.21293#S5.SS1.p1.1)\.
- \[22\]W\. Li, X\. Zhang, Z\. Guo, S\. Mao, W\. Luo, G\. Peng, Y\. Huang, H\. Wang, and S\. Li\(2025\)Fea\-bench: a benchmark for evaluating repository\-level code generation for feature implementation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 17160–17176\.Cited by:[§5\.1](https://arxiv.org/html/2609.21293#S5.SS1.p2.1)\.
- \[23\]S\. Wang, Z\. Wang, D\. Ma, Y\. Yu, R\. Ling, Z\. Li, F\. Xiong, and W\. Zhang\(2026\)Codeflowbench: a multi\-turn, iterative benchmark for complex code generation\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 4369–4402\.Cited by:[§5\.1](https://arxiv.org/html/2609.21293#S5.SS1.p2.1)\.
- \[24\]X\. Jiang, G\. Li, J\. Qian, X\. Shi, C\. Li, H\. Zhu, Z\. Wang, J\. Zhang, Z\. Zhao, K\. Zhang,et al\.\(2026\)KOCO\-bench: can large language models leverage domain knowledge in software development?\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 28422–28441\.Cited by:[§5\.1](https://arxiv.org/html/2609.21293#S5.SS1.p2.1)\.
- \[25\]Z\. Lu, Y\. Yang, H\. Ren, H\. Hou, H\. Xiao, K\. Wang, W\. Shi, A\. Zhou, M\. Zhan, and H\. Li\(2026\)Webgen\-bench: evaluating llms on generating interactive and functional websites from scratch\.Advances in Neural Information Processing Systems38\.Cited by:[§5\.1](https://arxiv.org/html/2609.21293#S5.SS1.p2.1)\.
- \[26\]R\. Pan, Y\. Zhang, Q\. Liang, Y\. Zhu, C\. Liu, L\. Zhang, and H\. Zhang\(2026\)Toward functional and non\-functional evaluation of application\-level code generation\.arXiv preprint arXiv:2602\.03462\.Cited by:[§5\.1](https://arxiv.org/html/2609.21293#S5.SS1.p2.1)\.
- \[27\]Q\. Ren, S\. Zou, S\. Huang, Z\. Zhang, K\. Shi, Z\. Fang, Y\. Zhao, Y\. Zeng, Q\. Su, L\. Chen,et al\.\(2026\)SaaSBench: exploring the boundaries of coding agents in long\-horizon enterprise saas engineering\.arXiv preprint arXiv:2605\.17526\.Cited by:[§5\.1](https://arxiv.org/html/2609.21293#S5.SS1.p2.1)\.
- \[28\]Z\. He, W\. Hong, Z\. Yang, Z\. Pan, M\. Liu, X\. Gu, and J\. Tang\(2026\)Vision2web: a hierarchical benchmark for visual website development with agent verification\.arXiv preprint arXiv:2603\.26648\.Cited by:[§5\.1](https://arxiv.org/html/2609.21293#S5.SS1.p2.1)\.
- \[29\]J\. Guo, Y\. Yao, J\. Chen,et al\.\(2026\)VISTA: an end\-to\-end benchmark for visual spec\-to\-web\-app coding agents\.arXiv preprint arXiv:2605\.26144\.Cited by:[§5\.1](https://arxiv.org/html/2609.21293#S5.SS1.p2.1)\.
- \[30\]R\. Vir, L\. Chilton, Z\. Zhang, and E\. Wu\(2026\)FlowCheck: helping end\-users specify and verify intent in vibe\-coded web apps\.arXiv preprint arXiv:2608\.28880\.Cited by:[§5\.1](https://arxiv.org/html/2609.21293#S5.SS1.p3.1)\.
- \[31\]C\. Hu, Y\. Zhao, and J\. Liu\(2024\)Game generation via large language models\.In2024 IEEE conference on games \(CoG\),pp\. 1–4\.Cited by:[§5\.2](https://arxiv.org/html/2609.21293#S5.SS2.p1.1)\.
- \[32\]D\. Chen, H\. Zhang, H\. Wang, Y\. Huo, Y\. Li, and J\. Wang\(2023\)Gamegpt: multi\-agent collaborative framework for game development\.arXiv preprint arXiv:2310\.08067\.Cited by:[§5\.2](https://arxiv.org/html/2609.21293#S5.SS2.p1.1)\.
- \[33\]J\. Hong, H\. Wu, and H\. Zhao\(2025\)Game development as human\-llm interaction\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 4333–4354\.Cited by:[§5\.2](https://arxiv.org/html/2609.21293#S5.SS2.p1.1)\.
- \[34\]L\. Yin, W\. Cheng, Z\. Qin, T\. Huang, Y\. Li, and G\. Ding\(2026\)AutoUE: automated generation of 3d games in unreal engine via multi\-agent systems\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 2341–2364\.Cited by:[§5\.2](https://arxiv.org/html/2609.21293#S5.SS2.p1.1)\.
- \[35\]Y\. Jiang, J\. Hu, Q\. Xiao, Y\. Zheng, R\. Ma, K\. Feng, J\. Han, T\. Peng, K\. Fan, M\. Zhang,et al\.\(2026\)Opengame: open agentic coding for games\.arXiv preprint arXiv:2604\.18394\.Cited by:[§5\.2](https://arxiv.org/html/2609.21293#S5.SS2.p1.1),[§5\.3](https://arxiv.org/html/2609.21293#S5.SS3.p1.1)\.
- \[36\]H\. Ma, H\. Wang, S\. Wang, T\. Yin, Y\. Shi, Y\. Huang, Y\. Zou, M\. Wen, and M\. Yang\(2026\)CreativeGame: toward mechanic\-aware creative game generation\.arXiv preprint arXiv:2604\.19926\.Cited by:[§5\.2](https://arxiv.org/html/2609.21293#S5.SS2.p2.1)\.
- \[37\]Y\. Huang, B\. Li, N\. Li, Z\. Wang, K\. Chen, H\. Ge, Q\. Si, Y\. Shen, R\. Yang, G\. Wang,et al\.\(2026\)GUI agents for continual game generation\.arXiv preprint arXiv:2605\.28258\.Cited by:[§5\.2](https://arxiv.org/html/2609.21293#S5.SS2.p2.1),[§5\.3](https://arxiv.org/html/2609.21293#S5.SS3.p1.1)\.
- \[38\]J\. Zhang, Y\. Jing, Z\. Cui, H\. Zheng, W\. Chen, K\. Li, J\. Yang, T\. Xie, Z\. Ma, T\. Bai,et al\.\(2026\)Bringing code alive: optimizing interactive frontend mini\-games via automated play and reinforcement learning at scale\.InForty\-third International Conference on Machine Learning,Cited by:[§5\.2](https://arxiv.org/html/2609.21293#S5.SS2.p2.1)\.
- \[39\]T\. Serapio, A\. Prakash, H\. Xu, K\. Wang, and A\. Greenwald\(2026\)Distilling game code world model generation into lightweight large language models\.arXiv preprint arXiv:2605\.24375\.Cited by:[§5\.2](https://arxiv.org/html/2609.21293#S5.SS2.p2.1)\.
- \[40\]C\. Zhou, Q\. Jiang, S\. Wu, and X\. Zhou\(2026\)The verifier is the curriculum: precision sets the return on search in code self\-distillation\.Note:arXiv preprint arXiv:2607\.09709External Links:2607\.09709,[Link](https://arxiv.org/abs/2607.09709)Cited by:[§5\.2](https://arxiv.org/html/2609.21293#S5.SS2.p2.1)\.
- \[41\]B\. La, S\. Chang, B\. Kim, J\. Bae, A\. A\. Beg, S\. Chang, G\. Gonzalez\-Pumariega, and K\. Goyal\(2026\)GameEngineBench: evaluating coding agents on real c\+\+ runtime environments\.arXiv preprint arXiv:2607\.03525\.Cited by:[§5\.3](https://arxiv.org/html/2609.21293#S5.SS3.p1.1)\.
- \[42\]J\. Sun, C\. Li, Z\. Li, Y\. Feng, F\. Zhang, Y\. Huang, Y\. Dai, and K\. Zhang\(2026\)JAMER: project\-level code framework dataset and benchmark on professional game engines\.arXiv preprint arXiv:2606\.19830\.Cited by:[§5\.3](https://arxiv.org/html/2609.21293#S5.SS3.p1.1)\.
- \[43\]W\. Zhang, J\. Yang, R\. Tao, L\. Chai, S\. Guo, J\. Wu, X\. Chen, G\. Cui, N\. Ding, X\. Xu,et al\.\(2026\)V\-gamegym: visual game generation for code large language models\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 5613–5641\.Cited by:[§5\.3](https://arxiv.org/html/2609.21293#S5.SS3.p1.1)\.
- \[44\]Z\. Peng, W\. Tao, X\. Yin, C\. Ying, Y\. Luo, and Y\. Guo\(2026\)PlayCoder: making llm\-generated gui code playable\.Proceedings of the ACM on Software Engineering3\(FSE\),pp\. 2003–2026\.Cited by:[§5\.3](https://arxiv.org/html/2609.21293#S5.SS3.p1.1)\.

## Appendix ASupplementary Trace Analysis

This appendix records selected diagnostic examples from evaluation reports and generation traces supporting the experimental discussion\. The cases illustrate failure and repair mechanisms\. Unless otherwise specified, DeepSeek\-V4\-Flash cases use the Claude Code baseline with full tool access, maximum reasoning effort, and a nominal turn budget of 120\. Strict task success has the benchmark\-compliance meaning defined in the main text\.

### A\.1Runtime and Interaction Failures

The following cases supplement the real\-input and scenario\-based diagnostics in Section[4\.6](https://arxiv.org/html/2609.21293#S4.SS6)\. They describe the prepared state, the interaction exercised, and the resulting check outcome\.

Real\-input service progression\.InDiner Dashergenerated by GPT\-6\-Astra with Codex CLI, two L2 P1 checks loadtray\_with\_correct\_item, then obtain the tray item’s and customer’s screen bounds from the snapshot\. One check sends a real mouse drag between their centers; the other sends the corresponding touch sequence\. Each compares the resulting snapshot with the prepared state and requires an increase in service progress or earnings\. Both fail at this progress assertion: neither real drag advances the service workflow\. The checks also test tray consumption and visible rendering changes after successful service, but these runs fail before reaching those assertions\. The artifact passes L1 and L2 P0/P2, showing why exposing a callable interface and a readable playfield does not establish that the required serving interaction works\.

Scenario preparation and natural\-time progression\.In the DeepSeek\-V4\-Flash/Claude Code implementation ofTurbo Smash Beast, reset or scenario loading enables test mode and suppresses natural\-time simulation\. The generation trace exercises two different paths: self\-tests through the evaluation interface advance simulation time explicitly, while a successful real\-input driving test starts from a freshly loaded page\. The benchmark instead prepares a scenario and then exercises real driving input with natural time progression\. In this sequence, the simulation remains stalled, and acceleration, coasting, and related real\-input checks fail\. The failing sequence combines operations that the separate self\-tests do not exercise together\.

### A\.2Execution\-Guided Repairs

These traces supplement the tool comparison in Section[4\.3](https://arxiv.org/html/2609.21293#S4.SS3)and the harness workflow analysis by connecting execution feedback to specific edits and subsequent checks\.

Feedback under different tool settings\.For DeepSeek\-V4\-Flash’sDiner Dasher, the file read/write only configuration supports a correction of touch\-release coordinates through source rereading; the final artifact passes the corresponding mouse and touch checks\. In the run with full tool access, Chromium mouse input exposes a different defect: a hidden completion screen missed by stub\-based tests\. Correcting the screen\-state mapping restores that transition\. These runs illustrate the distinct repairs prompted by source inspection and browser interaction\.

Syntax repair and behavioral verification\.In theTurbo Smash Beastrun with files \+ syntax checking,node \-\-checkcatches an unclosed construct\. Closing it restores parsing, but acceleration and coasting still fail benchmark evaluation\. The syntax feedback resolves a parsing defect while leaving the gameplay failures observable to behavioral checks\.

State\-progression repair\.In theGarden Gulprun with full tool access, a browser\-executed progression probe using the evaluation interface reveals that the hole’s actual size never catches up with its target size\. Repairing the update loop allows the probe to complete the level\. This feedback links an evolving game\-state discrepancy to a concrete update\-loop repair\.

Browser\-specific rendering repair\.In DeepSeek\-V4\-Pro’sNeon Flow, browser execution exposes a negative\-radius Canvas error missed by Node\-based tests\. The agent repairs the rendering path and verifies it again; the final artifact satisfies strict task success\. This sequence illustrates feedback from executing the artifact with the browser’s rendering API\.

Different repairs under two harnesses\.Both maximum\-effort DeepSeek\-V4\-FlashSortieruns achieve strict task success after execution\-guided repairs\. In Codex CLI, 27 self\-tests through the evaluation interface pass before a real mouse drag fails\. Inspection reveals that item positions are rendered relative to a layer below the heads\-up display while the snapshot reports viewport coordinates; correcting this offset is followed by successful real\-input retesting\. In Claude Code, self\-tests expose stale item states surviving reset because layout reconstruction preserves state indiscriminately\. Restricting that preservation to resize is followed by a 148/148 self\-test pass\. Later edits repair the snap animation and adjust audio initialization before the final regression checks\. The paired traces show how both harnesses support testing and repair, with different defects and follow\-up edits in these runs\. The self\-test counts refer to different agent\-authored tests\.

### A\.3Nominal\-Budget Trace Details

Claude Code stops execution when the configured turn limit is reached\. The following cases supplement Section[4\.3](https://arxiv.org/html/2609.21293#S4.SS3)by showing how this cutoff can interrupt verification or finalization before delivery is complete\.

Cutoff before verification executes\.Ancient Beastreaches the cutoff in both shorter\-budget conditions\. One 60\-turn attempt stops immediately after writing a browser harness, before running it\. The trace thus ends after preparing a verification step but before obtaining its execution feedback\.

Cutoff after self\-tests pass\.One 30\-turnBarbie and Ken’s Puzzlesattempt continues with screenshot processing after its self\-tests pass, then reaches the cutoff without completing delivery\. This case shows that passing the agent’s self\-tests does not itself complete the generation and delivery workflow\.

## Appendix BGeneration and Evaluation Implementation Details

Generation execution and delivery preflight\.Each attempt uses the original task documents in a clean workspace and a fresh session\. Claude Code runs in prompt mode with model and turn\-budget arguments and stops when the configured turn limit is reached\. Codex CLI uses an ephemeral session with workspace write access and JSON tracing\. The host records execution traces, exit status, elapsed time, failure reasons, artifact status, and available usage measurements\. The container launcher also monitors elapsed time and periods without trace updates to terminate timed\-out runs\. Evaluation requires a successful generation\-process exit and a regular, non\-symlink, nonemptyindex\.htmlcontaining</html\>\. Artifacts that pass this preflight proceed to L1 and L2 evaluation\.

Task access and evaluation isolation\.The launcher mounts the generation workspace with read/write access and marks the copied task documents read\-only\. The coding agent receives the generation prompt, gameplay design requirement, and evaluation interface specification \(tdd\.md\); executable checks and evaluation runners are not mounted in the generation container\. After generation ends, checks run in a separate evaluation container with the submitted HTML and tests mounted read\-only\.