GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?

# How Far Are Coding Agents from Expert Game Development?
Source: [https://arxiv.org/html/2608.21833](https://arxiv.org/html/2608.21833)
Haorong HongPeizhong GaoAffiliation:Tsinghua UniversityJianfeng LinAffiliation:The Hong Kong University of Science and TechnologyTongxu LuoAffiliation:The Chinese University of Hong Kong, ShenzhenYuxuan XieAffiliation:Lightspeed Studios, TencentChenxu LiuJieling HeAffiliation:Lightspeed Studios, TencentZhongyuan LiuAffiliation:Lightspeed Studios, TencentZeno Zeng\[0\.6em\] Hunyuan TeamTencent\[0\.3em\]Equal contribution\.†\\daggerCorresponding authors\.

###### Abstract

Recent large language models \(LLMs\) can operate as coding agents that build complete games from natural language requests\. Game development is especially demanding because program logic, visual and audio content, interfaces, interaction and playability must function together in one executable artifact\. Measuring this capability therefore requires evaluation of both game product and the development process\. Existing benchmarks often assess the game development capabilities of LLMs by evaluating the final artifact or an isolated development stage\. Our analysis of complete human–agent development trajectories identifies three stages that together span the lifecycle of game development with a coding agent: initial game generation, bug diagnosis and repair, and optimization over multiple turns\. Therefore, we introduceGameXpert\-Bench, which operationalizes the three lifecycle stages as three complementary benchmark tracks\.GameGenevaluates complete game creation from a single request in an empty workspace\.GameFixevaluates diagnosis and repair when defects are reported or left for the agent to discover\.GameOptevaluates cumulative optimization through request chains seeded by real development trajectories between users and agents\. We evaluate each track using live game interaction, deterministic behavioral tests, or final product criteria with regression checks\. The suite contains 97 generation tasks across 11 genres; 100 repair tasks from 50 game levels verified by humans, each with 19–27 injected bugs; and 17 optimization chains with six turns and 102 requests\. Across the three tracks, current agents are more reliable at producing playable foundations and implementing explicit requirements than at discovering defects, verifying runtime behavior, and preserving functionality across changes\. This asymmetry shows that initial generation quality alone is insufficient to characterize an agent’s game development capability\.

![Refer to caption](https://arxiv.org/html/2608.21833v1/fig_banner.png)Figure 1:Overview ofGameXpert\-Benchand its three evaluation tracks\. GameGen evaluates game creation from natural\-language requirements, GameFix evaluates the repair of corrupted reference games, and GameOpt evaluates multi\-turn game optimization based on human feedback\.## 1Introduction

The growing ability of large language models \(LLMs\) to act as coding agents has made game development an increasingly important testbed for assessing their holistic capability\. Recent systems can implement gameplay features in existing projects and game engines\[[5](https://arxiv.org/html/2608.21833#bib.bib2),[15](https://arxiv.org/html/2608.21833#bib.bib3)\]and generate complete games from natural language specifications\[[11](https://arxiv.org/html/2608.21833#bib.bib4),[24](https://arxiv.org/html/2608.21833#bib.bib5),[16](https://arxiv.org/html/2608.21833#bib.bib6)\]\. Yet successful code generation alone does not yield a playable game\. Gameplay logic, rendering, controls, interfaces, audiovisual content, and state transitions must work coherently under player interaction; a failure in any of these elements can degrade the player experience or prevent meaningful play altogether\.

This complexity makes evaluation a central challenge\. A successful build or a plausible screenshot does not establish that controls respond correctly, mechanics remain functional during play, or the completed game satisfies the request\. Recent benchmarks therefore execute generated games and assess their behavior through browser interaction, gameplay traces, and multimodal judgments\[[11](https://arxiv.org/html/2608.21833#bib.bib4),[24](https://arxiv.org/html/2608.21833#bib.bib5),[16](https://arxiv.org/html/2608.21833#bib.bib6),[9](https://arxiv.org/html/2608.21833#bib.bib12)\]\. These methods move game evaluation from static code inspection toward the behavior of the artifact that a player can actually experience\. However, evaluating only the final artifact does not reveal how a coding agent’s capabilities are exercised throughout the sequence of interactions that produces it\. This limitation motivates examining the development process itself, rather than only its endpoint\. A comprehensive evaluation should therefore consider the complete user\-facing game development lifecycle\.

We define the user\-facing game development lifecycle as the sequence of interactions carried out directly through a coding agent, from the initial request to the final playable artifact\. This lifecycle differs in scope from conventional game development frameworks, which include broader organizational phases such as preproduction, production, and postproduction\[[1](https://arxiv.org/html/2608.21833#bib.bib25)\]\. To derive this lifecycle, we conduct a qualitative analysis of complete human–agent game development trajectories\. By classifying interactions that change the game according to their primary intent, we identify three recurring stages: game generation, bug diagnosis and repair, and optimization over multiple turns\. Together, these stages describe how the executable artifact is created, maintained, and improved throughout the observed development process\.

Despite progress in runtime evaluation, existing benchmarks primarily assess the final artifact or an isolated stage of development\. As summarized in[Table1](https://arxiv.org/html/2608.21833#S1.T1), no existing benchmark suite for games jointly evaluates all three stages\. To address this limitation, we introduceGameXpert\-Bench, which connects them within a unified evaluation framework\.

Table 1:Coverage of representative benchmarks across the three stages studied in this paper\. Prior work evaluates several constituent capabilities, whileGameXpert\-Benchplaces game creation, verified repair, and cumulative optimization within one benchmark suite\. Text labels indicate partial coverage or the source of feedback\.The suite contains one track for each stage\.GameGenasks an agent to create a complete game from a single request in an empty workspace, without supplied assets or a prescribed engine\. It contains 97 tasks across 11 genres, including 44 3D games\.GameFixuses 50 confidential game levels verified by human reviewers and injects 19–27 defects into each level through reversible mutations\. Each level is evaluated with the defects reported and with selected defects hidden, yielding 100 repair tasks per run\.GameOptcontains 17 optimization chains with six turns each, for a total of 102 requests\. The chains are seeded by development trajectories between users and agents and completed, where necessary, with requests grounded in the same game state\.

The evaluation protocol for each track reflects its task\.GameGencombines behavioral rubrics, code inspection, live interaction, and human assessment\.GameFixuses deterministic Fail\-to\-Pass and Pass\-to\-Pass probes to verify that a patch repairs the target behavior without introducing regressions\.GameOptevaluates the final game across gameplay, level design, balance, art, interface, and audio, while checking whether earlier requested behavior is preserved\. All three protocols follow the same principle: an implementation receives credit only when the executable game provides evidence of the intended behavior\. Formal task definitions are given in[Section3](https://arxiv.org/html/2608.21833#S3)\.

The three tracks expose related limitations in current coding agents\. InGameGen, agents establish a playable core more reliably than they deliver rich content, robust interfaces, and fully integrated runtime behavior\. InGameFix, the models separate sharply when defects are hidden, and near\-complete repair remains uncommon when a task contains multiple bugs\. On the evaluated ofGameOpt, leading agents often retain requested functionality across six turns, but preservation of the core game loop and balanced improvement across product dimensions are not consistent\. Across the three tracks, agents are more reliable at producing playable foundations and implementing explicit requirements than at discovering defects, verifying runtime behavior, and preserving functionality across changes\. Initial generation quality alone is therefore insufficient to characterize an agent’s game development capability\.

#### Contributions\.

Our contributions can be summarized as follows:

- •We introduceGameXpert\-Bench, a benchmark suite that evaluates game generation, bug diagnosis and repair, and optimization over multiple turns\. These stages are derived from complete human–agent development trajectories and span the coding\-agent game development lifecycle\.
- •We construct 97 diverse generation tasks, 100 controlled repair tasks based on confidential Gold Games and reversible mutations, and 17 optimization chains containing 102 requests\. Their evaluation protocols connect implementation evidence to executable game behavior\.
- •We evaluate current coding agents across all three tracks and identify a common gap between implementing explicit requirements and autonomously discovering, verifying, and controlling the effects of changes to a game\.

## 2Related Work

#### Game generation and runtime\-grounded evaluation\.

Recent benchmarks evaluate coding agents on increasingly realistic game\-development tasks\. GameDevBench\[[5](https://arxiv.org/html/2608.21833#bib.bib2)\]and GameEngineBench\[[15](https://arxiv.org/html/2608.21833#bib.bib3)\]require agents to implement multimodal or runtime behavior within existing game projects\. OpenGame\-Bench\[[11](https://arxiv.org/html/2608.21833#bib.bib4)\]and WebGameBench\[[24](https://arxiv.org/html/2608.21833#bib.bib5)\]instead evaluate the generation of complete, browser\-native games, whereas GameCraft\-Bench\[[16](https://arxiv.org/html/2608.21833#bib.bib6)\]evaluates end\-to\-end game generation in the Godot Engine\. Because many game requirements are observable only during execution, these benchmarks complement static inspection with build validation, browser or engine interaction, replayable traces, and multimodal judging\. Most closely related to our evaluation, GameGen\-Verifier\[[9](https://arxiv.org/html/2608.21833#bib.bib12)\]decomposes specifications into independently verifiable keypoints and injects runtime states to test them through bounded interactions\. Despite their different environments and protocols, these works primarily adopt a*single\-request, final\-artifact*setting\.GameGenretains this setting as the generation track ofGameXpert\-Bench, but starts from a blank workspace without a provided project\. Its evaluation constructs a shared behavioral rubric through cross\-model event analysis, then injects runtime hooks and simulates actions to verify the corresponding game events\.

#### Game bug diagnosis and repair\.

Automated program\-repair benchmarks such as SWE\-bench\[[13](https://arxiv.org/html/2608.21833#bib.bib7)\]evaluate whether agents can resolve reported issues in existing repositories\. For games, VideoGameQA\-Bench\[[20](https://arxiv.org/html/2608.21833#bib.bib8)\]studies visual quality assurance from gameplay images and videos, while GBQA\[[10](https://arxiv.org/html/2608.21833#bib.bib9)\]evaluates whether agents can autonomously discover injected bugs through interactive exploration\. PlayCoder\[[19](https://arxiv.org/html/2608.21833#bib.bib13)\]additionally combines behavioral GUI testing with iterative program repair\. These works capture complementary aspects of testing, bug discovery, and repair, but do not jointly provide a verified clean game, a mechanically reversible mutation, its corresponding gold patch, and regression\-aware repair tests\.GameFixprovides this controlled construction across seven bug dimensions and evaluates agents under both*explicit\-issue*and*self\-discovery*settings\. Repairs must both resolve the injected failure and preserve previously correct behavior, as measured by Fail\-to\-Pass and Pass\-to\-Pass gates\.

#### Human\-guided multi\-turn refinement\.

Interactive coding benchmarks increasingly expose requirements and corrections over multiple turns\. SWE\-Together\[[22](https://arxiv.org/html/2608.21833#bib.bib10)\]reconstructs tasks from real user–agent coding sessions, but evaluates agents through an anchored, state\-conditioned user simulator\. Within game generation, Play2Code\[[8](https://arxiv.org/html/2608.21833#bib.bib11)\]places a coding agent and a GUI playtester agent in a sustained automated feedback loop\.GameOptinstead constructs multi\-turn optimization tasks from real human–agent game\-development trajectories\. Experts calibrate each task’s starting point, quality constraints, difficulty, and evaluation rubric, after which agents respond to successive optimization requests\. The resulting games are evaluated by an evidence\-grounded judge that combines code, image, and rule evidence, targeting improvement beyond repository\-level correctness alone\.

Taken together, prior work largely studies game generation, bug discovery and repair, and interactive refinement in isolation\.GameXpert\-Benchformulates them as three controlled, complementary tracks spanning the game\-development lifecycle: generation→\\rightarrowrepair→\\rightarrowhuman\-guided optimization\.

## 3GameXpert\-Bench Overview

![Refer to caption](https://arxiv.org/arxiv/extracted/7980050/pics/teaser.png)Figure 2:The overall structure ofGameXpert\-Bench\.GameGen starts from a natural language request, GameFix starts from a mutated Gold Game, and GameOpt studies multi\-turn optimization based on real human interaction\.### 3\.1The Game Development Lifecycle with Coding Agents

Conventional game development includes broad organizational phases such as preproduction, production, and postproduction\[[1](https://arxiv.org/html/2608.21833#bib.bib25)\]\.GameXpert\-Benchfocuses on the complete user\-facing artifact lifecycle carried out directly through a coding agent\. The lifecycle begins when a user asks an agent to create a game and ends with the final playable artifact delivered after requested corrections and improvements\.

We derive the lifecycle through a qualitative analysis of complete historical development trajectories between users and coding agents\. We examine each interaction that creates or modifies the executable game and classify it by its primary intent through iterative coding\. Clarification, environment setup, and other operational exchanges are associated with the development task they support but are not treated as separate lifecycle stages\. After resolving ambiguous cases, the analysis identifies three recurring categories that collectively cover the artifact changes observed across the trajectories\.

Definition: Complete User\-Facing Coding\-Agent Game Development LifecycleGeneration→Fix→Optimization\.\\textbf\{Generation\}\\;\\rightarrow\\;\\textbf\{Fix\}\\;\\rightarrow\\;\\textbf\{Optimization\}\.Generationcreates the initial playable game from a user request\.Fixdiagnoses and repairs defects revealed in a generated or existing game\.Optimizationimproves a playable game through successive requests while preserving its established functionality\. Together, the three stages cover the creation, maintenance, and improvement of the executable artifact\.

Figure[3](https://arxiv.org/html/2608.21833#S3.F3)summarizes artifact evolution across the complete trajectories in our analysis\. Source size does not decrease as generation, diagnosis and repair, and optimization requests operate on successive versions of each game\. This pattern is consistent with later development work inheriting and extending the results of earlier interactions rather than beginning from an independent project\.

Figure 3:Artifact growth across complete coding\-agent game development trajectories\.Filled markers denote artifact\-changing requests and colors indicate their lifecycle stage; hollow markers denote continuation requests\. Each trajectory retains or increases its source size across stages\. Game names are reported as broad categories\.The stages describe recurring modes of development rather than a fixed pipeline: diagnosis may trigger further repair, and optimization may expose new defects\. For controlled evaluation,GameXpert\-Benchmeasures the three stages with separate, stage\-specific corpora:GameGenevaluates initial creation,GameFixevaluates diagnosis and repair, andGameOptevaluates iterative optimization\. Together, the tracks measure whether an agent can create, maintain, and improve an executable game\.

### 3\.2Three Benchmark Tracks

#### GameGen: Single\-request game generation\.

GameGenevaluates whether an agent can turn a natural\-language design brief into a complete game\. The agent begins in a blank workspace with no provided template, multimedia assets, or prescribed engine, and is therefore responsible for integrating the mechanics, level logic, interactive interface, and audiovisual elements needed for a playable artifact\. This track measures the ability to realize a high\-level design intent from an initial request\.

#### GameFix: Game bug diagnosis and repair\.

GameFixevaluates repair in controlled game environments constructed from human\-verified Gold Games\. Reversible mutation operators inject defects into the source while preserving an exact gold patch and executable behavioral tests\. Agents are evaluated both when the reported issues are explicitly identified and when they must discover hidden defects from limited symptom descriptions\. A repair succeeds only when it restores the affected behavior and preserves behavior that was correct before the repair\.

#### GameOpt: Human\-guided game optimization\.

GameOptevaluates multi\-turn refinement from a playable game snapshot\. Each task replays a sequence of requests derived from human–agent game development trajectories, and the agent continues from the state it produced in the preceding turn\. The requests span complementary aspects of game quality, including gameplay, level design, balance, art, interface, and audio\. This setting evaluates whether an agent can incorporate successive product\-level requests while maintaining the playability and compatibility of the evolving game\.

### 3\.3Benchmark Organization

All three tracks are evaluated through the game artifact rather than source code alone\.GameGencombines a shared behavioral rubric with runtime verification and human assessment of player\-facing qualities\.GameFixuses deterministic gameplay probes to verify both repaired and preserved behaviors\.GameOptuses evidence\-grounded criteria over the final game, drawing on code and rendered output as appropriate and accounting for regressions\. These protocols connect implementation\-level changes to the game that players can actually run and experience\.

The remainder of the paper presents the three tracks in lifecycle order\.[Section4](https://arxiv.org/html/2608.21833#S4)describes the game\-generation corpus and evaluation,[Section5](https://arxiv.org/html/2608.21833#S5)details the controlled repair setting, and[Section6](https://arxiv.org/html/2608.21833#S6)introduces the trajectory\-based optimization benchmark and its scoring protocol\.

## 4GameGen: Single\-Request Game Generation

Within the AI\-based game\-development lifecycle introduced in[Section3](https://arxiv.org/html/2608.21833#S3),GameGenfocuses on its initial generation stage: translating a high\-level design vision into the first playable game artifact\. In this single\-turn setting, the coding agent acts as the developer\. Given only one natural\-language instruction as the design brief, it must generate a fully functional, browser\-native game\.

Importantly, this generation takes place in a blank workspace without provided game templates or multimedia assets, and the instruction does not prescribe a particular game engine or development toolchain\. The agent is therefore free to decide whether to use an engine and how to construct the game\. This setting captures a broadly accessible form of game creation through general\-purpose coding agents, while avoiding dependence on any particular engine ecosystem\. It also places greater emphasis on the agent’s native coding and game\- engineering capabilities rather than its familiarity with engine\-provided scaffolds\. The agent must build and integrate gameplay mechanics, level logic, interactive UI, and audiovisual content into a cohesive artifact\. This setup makesGameGena rigorous test of end\-to\-end game generation\. Instead of merely evaluating the functional correctness of isolated code snippets, our fine\-grained evaluation criteria go beyond whether the game compiles and runs to assess how faithfully it realizes the intended design\.

### 4\.1Task Formulation and Game Corpus

GameGencomprises 97 distinct games spanning 11 genres, including 44 tasks that require 3D rendering\. Each game is described by a natural\-language brief, and the games are deliberately spread across three difficulty levels, from those built around a single core mechanic to those that require coordinating several interacting mechanics\. This graded design letsGameGenmeasure agents across a wide range of complexity: the easier games test whether an agent can deliver a coherent and playable build at all, while the harder ones expose failures that surface only as games grow more complex, and thus better separate stronger agents from weaker ones\.

#### Formalization\.

Letpip\_\{i\}denote the natural\-language design brief for theii\-th game andAAa coding agent\. The agent produces a browser\-native artifactai=A⁡\(pi\)a\_\{i\}=A\(p\_\{i\}\)from the brief alone; importantly,pip\_\{i\}does not prescribe a game engine or implementation stack\. The benchmark corpus is the collection

𝒟=\{pi\}i=1N\\mathcal\{D\}=\\\{\\,p\_\{i\}\\,\\\}\_\{i=1\}^\{N\}\(1\)over allNNgames\.

### 4\.2Generation Protocol

Building on the task defined in[Section4\.1](https://arxiv.org/html/2608.21833#S4.SS1),GameGenadopts a strictly from\-scratch generation protocol\. Each agent is provided solely with the natural\-language brief and is required to synthesize a complete, browser\-native game without being provided with a game template, a prescribed engine, or pre\-existing assets\. The agent is free to choose its own technical stack, including whether to use an engine or general\-purpose browser libraries, and must produce every source file required by the final artifact\. Generation is carried out by a general\-purpose coding agent,Claude Code, invoked directly under its default configuration; we introduce no task\-specific scaffolding, ensuring that the resulting artifact is attributable to the agent’s intrinsic capability rather than to auxiliary engineering\. Since the brief enumerates the game’s required features explicitly, the protocol further probes the agent’sinstruction\-followingability: the agent is expected to faithfully realize the specified requirements while retaining the latitude to introduce additional mechanics and refinements\. Each task is completed within a single generation session and yields a self\-contained game that executes directly in the browser\.

### 4\.3Evaluation Framework

Our evaluation begins only after all evaluated models have completed generation\. We first construct a singleShared Rubricfor each game throughCross\-Model Events Analysis\. An event\-analysis agent examines the games produced by every model, extracts the gameplay events realized in each artifact, and preliminarily categorizes them as either core or bonus events\. For each game, we pool the events observed across models, merge semantically equivalent descriptions, and remove duplicates\. Human annotators then review the resulting candidate pool and curate a unified checklist, using the observed cross\-model coverage of each event as an important reference\. The resulting Shared Rubric is denoted as

𝒞i=ℛi⊎ℬi,\\mathcal\{C\}\_\{i\}=\\mathcal\{R\}\_\{i\}\\,\\uplus\\,\\mathcal\{B\}\_\{i\},\(2\)where core eventsℛi\\mathcal\{R\}\_\{i\}capture the relatively stable behaviors necessary to realize gameii, while bonus eventsℬi\\mathcal\{B\}\_\{i\}capture additional mechanics and content that vary across models\. Aggregating bonus events across models allows the rubric to distinguish models with different levels of capability without tailoring the evaluation to any single model\. Every artifact generated for the same game is subsequently assessed against the same𝒞i\\mathcal\{C\}\_\{i\}, ensuring a consistent comparison\.

Our evaluation dimensions follow the classical decomposition of a game into its mechanical systems, the resulting gameplay, and the player’s experience\[[25](https://arxiv.org/html/2608.21833#bib.bib14)\]\. We assess each generated artifact across four complementary dimensions\.

#### Completeness \(Automated\)\.

Completeness measures the fulfillment of the core events inℛi\\mathcal\{R\}\_\{i\}, including the fundamental mechanics, interactive UI behaviors, and level logic required by the game\. It therefore captures whether the generated artifact faithfully implements the essential content of the design brief rather than merely compiling into an executable page\.

#### Richness \(Automated\)\.

Richness measures the realization of bonus events inℬi\\mathcal\{B\}\_\{i\}\. It rewards supplementary mechanics, interactive content, level variety, and other functional extensions beyond the core game, thereby distinguishing structurally elaborate games from minimally viable ones\.

#### Visual Quality \(Human\)\.

Visual quality targets aesthetic and spatial properties that require human judgment\. In addition to overall visual presentation, annotators assess whether on\-screen and in\-world elements are positioned coherently\. This includes checking for UI overlaps and occlusions, content overflowing its designated boundaries, and, in 3D environments, geometric interpenetration between objects\. Such defects directly degrade the player\-facing quality of an otherwise functional artifact\.

#### Player Experience \(Human\)\.

Player experience evaluates the holistic playability of the generated game, including the responsiveness of interaction flows and the moment\-to\-moment feel of engaging with the game\. It complements event\-level functional evaluation by capturing qualities that emerge only through actual play\.

#### Hybrid Automated Scoring\.

To compute completeness and richness, the evaluation agent combines static source\-code analysis with dynamic runtime validation\. It first inspects the generated codebase to locate the programmatic logic corresponding to each item in the Shared Rubric\. It then interacts with the live game to reach the relevant states, trigger the targeted events, and verify that the observed runtime behavior matches the criterion\. A checklist item is credited only when its intended effect is validated during execution; this constraint prevents syntactically plausible yet functionally inert code from yielding false\-positive scores\. The criterion\-level outcomes are then aggregated into completeness and richness scores and combined with the human\-assessed dimensions to provide a holistic evaluation of the generated artifact\. All four dimension scores are normalized to\[0,100\]\[0,100\]and weighted equally\. LetSd​\(A\)S\_\{d\}\(A\)denote the benchmark\-level score of modelAAon dimensiondd, averaged over allNNgames\. The overall score is computed as the unweighted mean

Soverall​\(A\)=14​\(CLOSEScomp​\(A\)\+Srich​\(A\)OPEN\+Sexp​\(A\)\+Svis​\(A\)\)\.\\begin\{split\}S\_\{\\mathrm\{overall\}\}\(A\)=\\frac\{1\}\{4\}\\bigl\(&S\_\{\\mathrm\{comp\}\}\(A\)\+S\_\{\\mathrm\{rich\}\}\(A\)\\\\ &\+S\_\{\\mathrm\{exp\}\}\(A\)\+S\_\{\\mathrm\{vis\}\}\(A\)\\bigr\)\.\\end\{split\}\(3\)

### 4\.4Experimental Results and Analysis

#### Experimental Setup\.

We evaluate 15 representative model variants: Claude\-Opus\-5\[[3](https://arxiv.org/html/2608.21833#bib.bib15)\], Claude\-Fable\-5\[[3](https://arxiv.org/html/2608.21833#bib.bib15)\], Claude\-Opus\-4\.8\[[3](https://arxiv.org/html/2608.21833#bib.bib15)\], Claude\-Opus\-4\.7\[[3](https://arxiv.org/html/2608.21833#bib.bib15)\], Kimi\-K3\[[14](https://arxiv.org/html/2608.21833#bib.bib16)\], GPT5\.6\-sol\[[18](https://arxiv.org/html/2608.21833#bib.bib17)\], GPT5\.5\[[18](https://arxiv.org/html/2608.21833#bib.bib17)\], GLM5\.2\[[23](https://arxiv.org/html/2608.21833#bib.bib18)\], GLM5\.1\[[23](https://arxiv.org/html/2608.21833#bib.bib18)\], DeepSeek\-V4\-Flash\[[6](https://arxiv.org/html/2608.21833#bib.bib19)\], Hy3\[[21](https://arxiv.org/html/2608.21833#bib.bib20)\], Gemini\-3\.5\-flash\[[7](https://arxiv.org/html/2608.21833#bib.bib21)\], MiniMax\-M3\[[17](https://arxiv.org/html/2608.21833#bib.bib22)\], Qwen3\.7\-Max\[[2](https://arxiv.org/html/2608.21833#bib.bib23)\], and Seed\-2\.1\-pro\[[4](https://arxiv.org/html/2608.21833#bib.bib24)\]\. Every model is evaluated on all 97 game briefs under the same generation protocol described in[Section4\.2](https://arxiv.org/html/2608.21833#S4.SS2): the model receives the same blank workspace and natural\-language brief, with no provided template, assets, or prescribed game engine\. We report scores aggregated over the full benchmark\. The report\-level diagnostic analyses below cover all 15 models and 97 games, totaling15×97=1,45515\\times 97=1\{,\}455model–game runs and 43,081 event\-level outcomes\.

Table 2:Main results onGameGen\. All scores are reported out of 100, with the Overall score computed as the average of Completeness, Richness, Player Experience, and Visual Quality\. Values are rounded to one decimal place\. The best and second\-best results in each column are shown in bold and underlined, respectively\.
#### Overall Performance\.

[Table2](https://arxiv.org/html/2608.21833#S4.T2)presents the mainGameGenleaderboard\. Claude\-Opus\-5 ranks first with an overall score of 79\.7, exceeding Claude\-Fable\-5 by 3\.9 points and Kimi\-K3 by 8\.4 points\. It leads in completeness \(94\.4\), richness \(72\.0\), and visual quality \(80\.4\), while Claude\-Fable\-5 achieves the best player experience \(74\.4\)\. Thus, Claude\-Opus\-5 combines the broadest functional coverage with the strongest visual score, whereas Claude\-Fable\-5 retains an advantage in moment\-to\-moment player experience\. Kimi\-K3 and Claude\-Opus\-4\.8 follow with 71\.3 and 69\.9 overall, respectively\. A further 6\.5\-point drop separates Claude\-Opus\-4\.8 from GPT5\.6\-sol, after which GPT5\.6\-sol, GLM5\.2, DeepSeek\-V4\-Flash, Hy3, and Claude\-Opus\-4\.7 form a dense middle group between 62\.0 and 63\.4\.

#### Dimension\-wise Performance\.

Across the 15 evaluated models, completeness averages 77\.5, substantially higher than richness at 46\.1; player experience and visual quality average 61\.5 and 68\.2, respectively\. Richness is lower than completeness for every model, revealing a persistent gap between constructing the essential playable core and extending it with diverse bonus mechanics and content\. The leading models also exhibit complementary profiles\. Claude\-Opus\-5 leads three of the four dimensions, including visual quality, whereas Claude\-Fable\-5 remains strongest on player experience \(74\.4\) and ranks second on visual quality \(76\.8\)\. These differences show that a single notion of “playability” cannot capture game\-generation quality: functional breadth and player\-facing polish remain related but distinct capabilities\.

#### Progress across Model Versions\.

Within model families represented by multiple generations, Claude\-Opus\-4\.8 improves over Claude\-Opus\-4\.7 by 7\.9 overall points, and Claude\-Opus\-5 adds a further 9\.8 points\. The latter gain is largest in richness \(\+22\.4\), followed by completeness \(\+10\.0\) and visual quality \(\+6\.4\), while player experience changes only slightly \(\+0\.4\)\. This pattern suggests that the principal improvement from Claude\-Opus\-4\.8 to Claude\-Opus\-5 lies in broader functional scope, accompanied by a substantial gain in presentation quality\. GLM5\.2 similarly improves over GLM5\.1 by 5\.4 overall points, with gains across all four dimensions\.

Figure 4:Functional coverage across all 15 models\. Average completeness and richness are reported on\[0,10\]\[0,10\], and event pass rate is multiplied by 10 for comparison\. Models follow the main\-leaderboard order\.Figure 5:Composition of event\-level outcomes across all 15 models, including checklist events that are missing from the generated report\. Each stacked bar sums to 100%, and models follow the main\-leaderboard order\.
#### Functional Coverage and Event Outcomes\.

[Figures4](https://arxiv.org/html/2608.21833#S4.F4)and[5](https://arxiv.org/html/2608.21833#S4.F5)provide an event\-level view of the functional scores\. Completeness and event pass rate move together: Claude\-Opus\-5 passes more than 80% of tested events, whereas the lowest\-ranked model passes fewer than half\. More importantly, richness remains below completeness for every model, including the leaders\. Current agents therefore tend to prioritize the required gameplay skeleton before adding optional mechanics, levels, and interactions\. The stacked outcome distribution further shows that the separation between models is driven mainly by how much probability mass is converted from failed or unresolved events into fully passed events, rather than by a uniform change in every outcome category\. This result complements the aggregate leaderboard by showing that stronger models not only attempt broader functionality but also execute a larger fraction of tested events successfully\.

Figure 6:Performance gaps between the 2D and 3D subsets ofGameGen\.\(a\)Difference in overall score for each model, computed as the 3D score minus the 2D score\. The dashed line marks the cross\-model mean gap\.\(b\)Cross\-model mean gaps across the four evaluation dimensions\. All scores are reported on a\[0,100\]\[0,100\]scale, and negative values indicate lower performance on the 3D subset\. Models follow the main\-leaderboard order\.
#### 2D versus 3D Game Generation\.

We further disaggregate the results by game dimensionality\. Averaged across the 15 evaluated models, the overall score decreases from 65\.9 on the 53 2D games to 60\.1 on the 44 3D games, corresponding to an average drop of 5\.8 points\. As shown in[Figure6](https://arxiv.org/html/2608.21833#S4.F6), 14 of the 15 models perform worse on the 3D subset, with Kimi\-K3 being the only exception \(\+1\.8\)\. The largest mean degradation occurs in completeness \(−8\.8\-8\.8points\), followed by player experience \(−6\.0\-6\.0\), richness \(−5\.9\-5\.9\), and visual quality \(−2\.4\-2\.4\)\. These results indicate that 3D game generation poses a greater challenge to current agents under our benchmark\. Notably, the performance gap is driven primarily by reduced functional completeness rather than visual quality alone\. Full subset\-level leaderboards are provided in the appendix\.

Figure 7:Player\-facing and runtime failure diagnostics\.\(a\)Share of model–game runs whose evaluation reports contain evidence of UI overlap or misalignment\.\(b\)Share of events recognized as implemented in code but assigned a runtime outcome ofFailorPartial\.\(c\)Attribution of the 2,293 implementation\-gap events\.
#### UI Misalignment Is a Universal Failure Mode\.

We retrieve thereasoningfields of UI\-related events from each evaluation report and search for a fixed multilingual keyword set covering misalignment, overlap, occlusion, covering, overflow, and offset, together with the English termsoverlap,clipped,off\-screen, andoverflow\. A model–game run is marked as affected if any of these terms is matched\. As shown in[Figure7](https://arxiv.org/html/2608.21833#S4.F7), 221 of the 1,455 runs \(15\.2%\) are flagged\. Every evaluated model exhibits this problem: the rate ranges from 7\.2% for Claude\-Fable\-5 and 10\.3% for Claude\-Opus\-5 to 19\.6% for GLM5\.1 and 22\.7% for Seed\-2\.1\-pro\. Although the ordering is not strictly monotonic, UI layout defects are generally more prevalent among lower\-performing models\. These results reveal a widespread gap between generating individually plausible interface components and composing them into a stable layout across runtime states\. We provide a per\-game breakdown in the appendix\.

#### Implemented Does Not Mean Functional\.

Finally, we examine events for which static inspection reportsimplementation\_status == implemented, but dynamic evaluation assignsFailorPartial\. Such cases correspond to functionality that appears in the code but cannot be triggered reliably, produces insufficient feedback, or transitions to an incorrect state at runtime\. Across 43,081 assessed event instances, 2,293 \(5\.32%\) fall into this category\. As shown in[Figure7](https://arxiv.org/html/2608.21833#S4.F7), the rate varies sharply across models, from 0\.4% for Claude\-Fable\-5 and 0\.9% for Claude\-Opus\-5 to 10\.1% for Qwen3\.7\-Max, 10\.8% for Gemini\-3\.5\-flash, and 17\.3% for Seed\-2\.1\-pro\.

The attribution panel further shows that load or crash failures account for the majority of these cases \(56\.0%\)\. Incorrect state transitions contribute 16\.0%, unclear or missing feedback 13\.1%, and missing visual responses 11\.9%, with the remaining 2\.9% assigned to other causes\. Thus, most implementation gaps arise not because the relevant code is entirely absent, but because it is not integrated into a robust end\-to\-end execution path\. This finding directly motivates our hybrid evaluator: static code inspection is useful for locating candidate implementations, but runtime interaction is necessary to prevent inert or incorrectly wired functionality from receiving credit\.

Figure 8:Model\-level association between automated functional evaluation and human\-perceived quality\. Each point represents one of the 15 evaluated models\. The automated functional score is the mean of Completeness and Richness, while the human perceptual score is the mean of Player Experience and Visual Quality; all component scores are on a\[0,100\]\[0,100\]scale\. The solid line denotes the ordinary least\-squares fit, and the shaded region denotes its 95% confidence band for the mean response\. GPT5\.6\-sol and GLM5\.2 have identical aggregate coordinates and therefore overlap\.
#### Automated Functional Scores Are Strongly Associated with Human\-Perceived Quality\.

We examine whether models rated as functionally stronger by the automated evaluator also tend to produce games that receive higher human judgments\. For each modelAA, we define an automated functional score and a human perceptual score as

Sauto​\(A\)\\displaystyle S\_\{\\mathrm\{auto\}\}\(A\)=Scomp​\(A\)\+Srich​\(A\)2,\\displaystyle=\\frac\{S\_\{\\mathrm\{comp\}\}\(A\)\+S\_\{\\mathrm\{rich\}\}\(A\)\}\{2\},\(4\)Shuman​\(A\)\\displaystyle S\_\{\\mathrm\{human\}\}\(A\)=Sexp​\(A\)\+Svis​\(A\)2\.\\displaystyle=\\frac\{S\_\{\\mathrm\{exp\}\}\(A\)\+S\_\{\\mathrm\{vis\}\}\(A\)\}\{2\}\.Across the 15 models, the two aggregates exhibit a strong positive linear association \(Pearson’sr=0\.898r=0\.898,p=5\.38×10−6p=5\.38\\times 10^\{\-6\}\) and a strong rank association \(Spearman’sρ=0\.832\\rho=0\.832,p=1\.20×10−4p=1\.20\\times 10^\{\-4\}\)\. As shown in[Figure8](https://arxiv.org/html/2608.21833#S4.F8), ordinary least\-squares regression yieldsShuman=0\.643​Sauto\+25\.082S\_\{\\mathrm\{human\}\}=0\.643S\_\{\\mathrm\{auto\}\}\+25\.082withR2=0\.807R^\{2\}=0\.807\. Thus, agents that realize more required and bonus functionality generally also deliver games with better player experience and visual quality\. The slope below one further indicates that human\-perceived scores vary less across models than automated functional scores\. Nevertheless, the two signals are not interchangeable: Claude\-Opus\-4\.8 lies above the fitted trend, whereas DeepSeek\-V4\-Flash lies below it\. These deviations reinforce the need for our hybrid protocol, as functional coverage and human\-perceived quality capture complementary aspects of generated games rather than duplicate judgments of the same criteria\.

## 5GameFix: Game Bug Diagnosis and Repair

GameFixis the bug\-repair track ofGameXpert\-Bench\. Following the SWE\-bench paradigm\[[12](https://arxiv.org/html/2608.21833#bib.bib1)\], we give an agent a real HTML5/JavaScript web game broken by automatically injected bugs and ask it to diagnose and repair the source in a network\-disabled sandbox\. Each task is graded by a deterministic probe that launches the repaired game in headless Chromium through Playwright and executes behavioral tests\. Unlike rubric\-based coding benchmarks, this directly verifies game behavior, making the benchmark reproducible, verifiable, inexpensive to extend, and free of LLM\-judge bias\. We describe the task setting and Gold Games \(§[5\.1](https://arxiv.org/html/2608.21833#S5.SS1)\), bug construction \(§[5\.2](https://arxiv.org/html/2608.21833#S5.SS2)\), evaluation protocol and metrics \(§[5\.3](https://arxiv.org/html/2608.21833#S5.SS3)\), and results on 17 models \(§[5\.4](https://arxiv.org/html/2608.21833#S5.SS4)\)\. Additional examples, test definitions, and complete results are in Appendix[B](https://arxiv.org/html/2608.21833#A2)\.

### 5\.1Task setting and gold games

GameFixis a controlled benchmark built from a curated collection ofGold Games: complete, human\-verified, fully playable web games\. Bugs are injected through a reversible mutation pipeline \(§[5\.2](https://arxiv.org/html/2608.21833#S5.SS2)\)\. Each defect is produced by a mechanically invertible edit whose inverse is the exact gold patch, giving every task fixed ground truth without human annotation and enabling deterministic executable grading\.

The closed set contains50 independently playable, separately reviewed internal game levelswith distinctive gameplay and control mechanics unavailable on the internet\. Each level receives 19–27 bugs at once, so repair requires localisation, prioritisation, and regression avoidance across multiple independent defects within a single session\. Combined with the two query modes in §[5\.3](https://arxiv.org/html/2608.21833#S5.SS3), this yields100 evaluation tasksper run\. Because the pipeline requires only a gold game and its executable tests, it can be extended to additional games without re\-annotation\.

#### Provenance and anti\-contamination\.

Proprietary games with novel mechanics reduce the risk of dataset contamination\. Classic and popular titles are common in pre\-training corpora, allowing frontier models to potentially “repair” them by recalling memorised source rather than reasoning from observed behavior\. In a pilot of50 open\-source games, we observed signs of such contamination\. Repository quality was also inconsistent, making even “correct” behavior difficult to define reliably\. Our Gold Games instead use gameplay largely unavailable elsewhere and high player\-skill difficulty, making memorisation shortcuts unlikely and requiring the model to reason about playability\. Each game is admitted only aftermore than 24 hours of reviewand adouble sign\-offfrom a game\-design specialist and an AI researcher\. For confidentiality, the closed set remains internal\.

### 5\.2Bug construction

Each Gold Game is broken at code sites drawn from a taxonomy of7 dimensions and 61 subcategories, distilled by our game\-development team from production experience\. The dimensions cover the major ways a game can fail:

- •Core Gameplay: core mechanics, including controls, combat, AI, physics, and win/lose rules;
- •Meta: long\-term systems, including progression, economy, achievements, and multiplayer;
- •UI Design: HUD, menus, text, layout, tutorials, and accessibility;
- •Art Design: visual and audio presentation, including animation, effects, camera, and sound;
- •Test: runtime issues, including crashes, freezes, loading failures, and performance;
- •Level Design: maps, object placement, level progression, and procedural generation;
- •Balance Design: numerical balancing of combat, difficulty, rewards, and economy\.

Within a chosen subcategory, a bug is created by a small, mechanically reversible mutation operator whose inverse is the gold fix\. The pipeline therefore emits both amutation\.patchand its exact inverse,gold\.patch\. Subcategories specify what kind of behavior breaks, while operators are the low\-level edits that cause the breakage\. Reversibility provides exact ground truth and makes repair outcomes unambiguous\. Worked mutation↔\\leftrightarrowgold examples are given in Appendix[B](https://arxiv.org/html/2608.21833#A2)\.

#### Why many bugs per task\.

Real games rarely present 19–27 defects at once; this count is a deliberate evaluation design rather than a claim about realism\. Its purpose is to evaluate long\-horizon agentic behavior\. In one session, the agent repeatedly performs Diagnose→\\rightarrowEdit→\\rightarrowTest, and we measure how many independent bug sites it can localise, repair, and verify before stopping\. This coverage reflects both its understanding of the game and its ability to resolve multiple defects efficiently\. Individual bugs are typically one\- or two\-line edits whose effects often appear only during gameplay\. In*Self\-Discovery*, most symptom descriptions are withheld, further requiring the model to discover bug sites autonomously\. Together, these settings evaluate the six agentic abilities defined in Appendix[B\.2](https://arxiv.org/html/2608.21833#A2.SS2)\.

### 5\.3Evaluation protocol

#### Two query modes\.

Each task is evaluated under two settings that differ only in how much the prompt discloses:

- •*Explicit Issue*\(“list”\): the user provides a numbered checklist of all unreasonable behaviours and asks the agent to repair them\. The task is to localise and fix each named bug\.
- •*Self\-Discovery*\(“minimal”\): the prompt reveals only subjective, presentation\-level or taste\-dependent symptoms \(e\.g\.,art\_designchoices\)\. Objective defects, such as reversed controls, impassable obstacles, or unplayable movement speeds, are withheld\. The agent is told that additional strange behaviours remain and must discover and repair them autonomously\. The scoring denominator remains the full bug set, so every undiscovered objective bug counts as a failure\.

The gap between the two modes directly measures a model’sself\-discoveryability and is a central axis of our analysis\.

#### Executable grading: F2P and P2P\.

Each candidate patch is graded by a deterministic program that executes the repaired game\. In Node\.js, Playwright launches headless Chromium and serves the game locally\. A virtual clock advances time in fixed steps, synthetic inputs are dispatched at stage coordinates, and a read\-only JSON state snapshot is collected after each step\.

Each test case is an executable behavioral check over this state: the evaluator resets the level, advances the clock, dispatches inputs, and evaluates a boolean condition\. Parameterized tests must pass for every value in their parameter set\. Following SWE\-bench\[[12](https://arxiv.org/html/2608.21833#bib.bib1)\], tests are divided intoFail\-to\-Pass \(F2P\), which capture behaviors broken by injected bugs and must pass after repair, andPass\-to\-Pass \(P2P\), which ensure previously correct behavior does not regress\. Each task is evaluated in a single probe session, with every test run from a fresh reset\. Tests are grouped by bug site, and the task score is the percentage of bug sites repaired\.

#### Scoring\.

A bug counts as*fixed*only when its Fail\-to\-Pass assertion passes and every associated Pass\-to\-Pass assertion still passes, ensuring the intended behaviour is restored without regression\. Because each task contains 19–27 bugs, we verify at authoring time that no injected bug masks another bug’s F2P or breaks its P2P \(Appendix[B\.1](https://arxiv.org/html/2608.21833#A2.SS1.SSS0.Px1)\)\. A P2P failure at grading time therefore reflects a regression introduced by the candidate patch\. A task’s score is the fraction of bugs fixed,

score=100⋅\#​fixed bugs\#​bugs∈\[0,100\]\.\\text\{score\}\\;=\\;100\\cdot\\frac\{\\\#\\text\{fixed bugs\}\}\{\\\#\\text\{bugs\}\}\\;\\in\\;\[0,100\]\.\(5\)

#### From scores to metrics\.

For each runrrwe form a*survival curve*Sr​\(τ\)S\_\{r\}\(\\tau\): for an integer thresholdτ∈\[0,100\]\\tau\\in\[0,100\],Sr​\(τ\)S\_\{r\}\(\\tau\)is the percentage of the 100 tasks whose score is at leastτ\\tau\. We report the average of the three curves,S⁡\(τ\)=13​∑rSr​\(τ\)S\(\\tau\)=\\tfrac\{1\}\{3\}\\sum\_\{r\}S\_\{r\}\(\\tau\)\(Fig\.[10](https://arxiv.org/html/2608.21833#S5.F10)\)\. Each metric is the mean height ofSSover a threshold band\[a,b\]⊆\[0,100\]\[a,b\]\\subseteq\[0,100\],

S¯​\[a,b\]=1b−a\+1​∑τ=abS⁡\(τ\),a,b∈ℕ\.\\bar\{S\}\[a,b\]\\;=\\;\\frac\{1\}\{b\-a\+1\}\\sum\_\{\\tau=a\}^\{b\}S\(\\tau\),\\qquad a,b\\in\\mathbb\{N\}\.\(6\)
Our primary metric,Strict, is the mean height over the near\-perfect bandτ∈\[90,100\]\\tau\\in\[90,100\],

Strict=S¯​\[90,100\]=111​∑τ=90100S⁡\(τ\)\.\\textsc\{Strict\}\\;=\\;\\bar\{S\}\[90,100\]\\;=\\;\\frac\{1\}\{11\}\\sum\_\{\\tau=90\}^\{100\}S\(\\tau\)\.\(7\)Strictis high only when a model repairs nearly every bug in a task across runs, making it sensitive to self\-discovery and difficult to saturate\.

AlongsideStrictwe report theCliff, our measure of the self\-discovery deficit\. Taking a model’s macro average@3 repair score under each of the two query modes, the Cliff is the drop from*Explicit Issue*to*Self\-Discovery*\. We report only this difference, not the two absolute scores: under*Explicit Issue*every bug is named, so absolute scores there are compressed near the top of the range and separate models poorly, whereas the drop to*Self\-Discovery*isolates exactly the ability we care about\. All models run with each vendor’s maximum reasoning effort in a network\-disabled sandbox under the Claude Code agent framework; the two GPT models are additionally run under Codex to separate framework effects from model behavior\.

### 5\.4Experimental results and analysis

Table[3](https://arxiv.org/html/2608.21833#S5.T3)gives the full 17\-model leaderboard\. We draw four observations from it\.

Table 3:GameFixleaderboard, sorted by the primaryStrictmetric \(mean height of the average@3 survival curve over the bandτ∈\[90,100\]\\tau\\in\[90,100\]\)\. Cliff is the drop in macro average@3 repair score from*Explicit Issue*to*Self\-Discovery*\(we report this difference only, not the two absolute scores; see §[5\.3](https://arxiv.org/html/2608.21833#S5.SS3)\)\. All values are 3\-run average@3\. In each score columnboldmarks the best value andunderlinethe worst \(Cliff is lower\-is\-better, so bold is the smallest cliff and underline the largest\)\.†models are analysed at the trajectory level in Appendix[B](https://arxiv.org/html/2608.21833#A2)\.Figure 9:Survival curvesof the average@3 pass rate versus the all\-solved thresholdτ\\taufor all 17 models \(eight highlighted\); each run contributes its own curve and the three are averaged\. The shaded bandτ∈\[90,100\]\\tau\\in\[90,100\]is the region whose mean curve height definesStrict\.Figure 10:Geometric meaning ofStrict\.Left: for the highlighted models, the survival curve over theτ∈\[90,100\]\\tau\\in\[90,100\]band;Strictis the mean height of that curve across the band \(dashed line\), equivalently the shaded area divided by the band width\. Right:Strictfor all 17 models, sorted descending\.Figure 11:The*Explicit Issue*→\\rightarrow*Self\-Discovery*Cliff\.Left: each model’s macro average@3 under*Explicit Issue*versus*Self\-Discovery*; a steeper drop is weaker self\-discovery\. Right \(inset\): the Cliff shrinks asStrictrises\.#### The benchmark is far from saturated\.

Even the strongest model, Claude Opus 5, reaches onlyStrict=39\.0=39\.0out of 100, while the median model sits near 14\. Near\-perfect multi\-bug repair remains rare, leaving substantial headroom\.

#### Models separate under*Self\-Discovery*, not*Explicit Issue*\.

With the full checklist the 17 models are compressed into a span of about 13 points, suggesting that localise\-and\-fix is close to solved*when every bug is named*\. Once the bugs must be discovered autonomously that span widens to about 38 points, nearly three times wider\.Strictreflects this distinction directly, since reaching the≥90\\geq 90tail requires recovering nearly all un\-hinted bugs \(Fig\.[9](https://arxiv.org/html/2608.21833#S5.F9)\)\.

#### The*Explicit Issue*→\\rightarrow*Self\-Discovery*Cliff narrows as models get stronger\.

The Cliff ranges from 7\.6 for Opus 5 to 32\.8 for Hy\-3 and decreases asStrictrises \(Fig\.[11](https://arxiv.org/html/2608.21833#S5.F11)\)\. A large Cliff indicates that a model can repair bugs once pointed to them but struggles to discover and resolve them autonomously\.

#### Six agentic abilities behind the gap\.

The multi\-bug setting requires more than code editing\. We identify six interacting abilities:*self\-discovery*,*behavioural verification*,*value recovery*,*multi\-bug coverage and planning*,*regression control*, and*stopping criterion*\. Most become substantially more demanding under*Self\-Discovery*, where the checklist no longer supplies bug locations or a clear completeness target\. Appendix[B\.2](https://arxiv.org/html/2608.21833#A2.SS2)defines these axes and provides trajectory\-level comparisons across representative models\.

#### Two primary mechanisms produce the Cliff\.

*\(i\) Discovery deficit\.*Weaker models fail to find many un\-hinted bugs, causing their*Self\-Discovery*score to fall sharply; for example, Hy\-3 drops from 96\.2 to 34\.6 on one instance when the checklist is reduced\.*\(ii\) Fix authorization\.*Some models identify additional bugs but decline to repair them because they interpret the checklist as a scope boundary\. Claude Opus 4\.8, for example, rationalises injected defects as intentional design \(100→\\rightarrow59\.3 despite similar localisation\), while GLM 5\.2 identifies a dead enemy subsystem but leaves it as beyond “minimal changes” \(100→\\rightarrow26\.9\)\. In these cases, “not listed” is effectively treated as “not authorised\.”

#### Verification and stopping further limitStrict\.

Discovery alone is insufficient: models must also infer uncertain values, verify the resulting behaviour, and decide when the task is complete\. The sandbox provides no gold reference, so free constants such as physics parameters must often be reconstructed from gameplay constraints or local evidence\. Weaker models more often rely on static inspection or plausibility\-based checks, while stronger models derive values from in\-game invariants and validate them through execution\.

#### Other recurring failure modes\.

We also observe*rabbit\-holing*, where an agent spends excessive effort on one confusing artefact while leaving other bugs untouched, and occasional*multi\-agent conflict*, where parallel workers overwrite or misinterpret one another’s edits\. These cases are less frequent, but show that long\-horizon repair introduces planning and orchestration failures beyond individual bug localisation\.

#### Why stronger models keep the Cliff small\.

Stronger models continue searching beyond the disclosed hints, pursue coverage until they have an objective completeness signal, recover missing values from game invariants, and verify repairs against actual execution\. Weaker models perform this end\-to\-end audit less consistently\. The resulting reliability gap across discovery, verification, planning, regression control, and stopping is whatStrictis designed to expose\. More trajectory analysis and worked repair cases are provided in Appendix[B](https://arxiv.org/html/2608.21833#A2)\.

## 6GameOpt: Human\-Guided Game Optimization

Game development rarely ends once a game becomes executable\. A high\-quality game also requires multiple rounds of optimization to enhance its mechanics, level layout, numerical balance, visual language, interface, and audio feedback\. Unlike bug fixing, these changes generally admit multiple valid implementations and cannot be specified by a single reference patch\. Therefore, a benchmark for evaluating multi\-turn, open\-ended game development\.

We introduceGameOpt, which is the third stage of our benchmark suite, following game generation and game bug fixing\. It asks a single question: given a real game and a trajectory\-grounded sequence of product requests, can a coding agent turn a working prototype into a better product without breaking what already worked? The games and starting trajectories originate from historical human–agent co\-creation\. Requests retain human requirements where suitable history exists and use snapshot\-grounded synthesis to complete missing design dimensions; every model receives the same fixed replay inputs\.

### 6\.1Task Formulation

AGameOptinstance consists of an initial game repositoryG\(0\)G^\{\(0\)\}and an ordered sequence of natural\-language requestsU=\(u1,…,uT\)U=\(u\_\{1\},\\ldots,u\_\{T\}\)\. At turntt, the coding agent receives the current repository and the new request, and produces a set of editsΔt\\Delta\_\{t\}:

Δt=𝒜⁡\(G\(t−1\),ut,h<t\),G\(t\)=Apply⁡\(G\(t−1\),Δt\)\\Delta\_\{t\}=\\mathcal\{A\}\\\!\\left\(G^\{\(t\-1\)\},u\_\{t\},h\_\{<t\}\\right\),\\qquad G^\{\(t\)\}=\\operatorname\{Apply\}\\\!\\left\(G^\{\(t\-1\)\},\\Delta\_\{t\}\\right\)\(8\)The repository is never reset between turns, so the chain is a genuine long\-horizon process rather than six independent edit tasks: the state on which requestutu\_\{t\}is served is whatever the agent itself produced in the firstt−1t\-1turns\.

Each turn targets one of six design dimensions,

𝒟=\{gameplay,level,balance,art,ui,audio\},\|𝒟\|=T,\\mathcal\{D\}=\\\{\\textsc\{gameplay\},\\,\\textsc\{level\},\\,\\textsc\{balance\},\\,\\textsc\{art\},\\,\\textsc\{ui\},\\,\\textsc\{audio\}\\\},\\qquad\|\\mathcal\{D\}\|=T,\(9\)
and a chain\-specific bijectionπc:\{1,…,T\}→𝒟\\pi\_\{c\}:\\\{1,\\dots,T\\\}\\\!\\to\\\!\\mathcal\{D\}assigns one dimension to each turn\. Conceptually, the agent is asked to solve

max⁡∑d∈𝒟G\(T\)⁡αd​Qd​\(G\(T\)\)−λ​ℛ​\(G\(0\),G\(T\)\)\\max\_\{G^\{\(T\)\}\}\\sum\_\{d\\in\\mathcal\{D\}\}\\alpha\_\{d\}Q\_\{d\}\\\!\\left\(G^\{\(T\)\}\\right\)\-\\lambda\\,\\mathcal\{R\}\\\!\\left\(G^\{\(0\)\},G^\{\(T\)\}\\right\)\(10\)whereQdQ\_\{d\}is not assumed to be a unique ground\-truth quality function; different implementations may realize the same user intent\. In the benchmark, it is operationalized by task\-specific acceptance criteria and observable evidence\. The regression termℛ\\mathcal\{R\}captures the equally important requirement that the final game remain buildable, playable, and compatible with its original core loop\.

The target use case is live human–agent collaboration, in which a person plays or inspects successive versions and expresses the next need from a player or product perspective\. For controlled model comparison, however,GameOptinstantiates this setting as an offline replay: all models receive the same ordered requests and the same starting snapshot\. This design retains the cumulative nature of human\-guided optimization while removing variation caused by different users choosing different follow\-up requests\.

### 6\.2Trajectory\-Based Benchmark Construction

#### Source trajectories\.

The data originate from historical game co\-creation trajectories between users and coding agents\. Each trajectory records a sequence of user requests together with the code artifact produced after each request\. We select a playable intermediate version as the starting snapshot: it must be sufficiently complete to admit meaningful optimization, while the requested improvements must still be observable or implementable from that snapshot\. Only this selected snapshot is exposed during evaluation; later historical versions are not provided to the tested model\.

Each benchmark chain contains six turns, one for every dimension in Eq\. \([10](https://arxiv.org/html/2608.21833#S6.E10)\)\. When a historical trajectory contains a suitable request, the user request is retained and lightly normalized for clarity\. Dimensions absent from the trajectory are completed with strong\-model\-generated requests grounded in concrete deficiencies of the same snapshot\. Synthetic requests are therefore used for coverage rather than as generic, repository\-independent instructions\. The resulting task prompt preserves intentional product\-level ambiguity \(e\.g\., “make the encounter more tense”\), whereas a normalized intent and the acceptance criteria are kept private from the tested model\.

#### Dataset composition\.

The currentGameOptevaluation collection contains 17 self\-contained JavaScript games, each with one six\-turn chain, for 102 optimization turns in total\. Its 701 acceptance criteria comprise 604 positive rubric items and 97 regression checks\. Table[4](https://arxiv.org/html/2608.21833#S6.T4)reports the composition; the complete chain inventory appears in Appendix[C\.1](https://arxiv.org/html/2608.21833#A3.SS1)\.

Table 4:Composition of the currentGameOptevaluation collection\.
#### Evidence\-oriented rubrics\.

For each turn, the benchmark defines a hidden set of independently testable rubric items\. A positive item is either a*requirement*, which captures the user’s stated intent, or a*challenge*, which checks integration and edge cases that are necessary for a robust implementation\. A*regression*item instead detects functionality present inG\(0\)G^\{\(0\)\}that is broken inG\(T\)G^\{\(T\)\}\. Each item specifies its priority, provenance, evaluation modality \(code, rendered output, or both\), and the evidence required for a judgment\. Compound conditions follow an all\-conditions\-required rule: an item passes only when every explicit condition is established\. Comments, unused configuration, dead code, and the model’s own description of its changes are not accepted as evidence\.

The current bundle contains 604 positive items \(392 requirements and 212 challenges\) and 97 regression checks\. Of all 701 items, 281 are judged from code, 122 require both code and rendered evidence, and 298 use rendered evidence\. Thirty\-three items are explicitly marked as proxies for properties such as visual coherence or game feel\. This marker is important: the presence of an implementation path can support a proxy criterion, but does not by itself prove that the resulting experience is aesthetically superior\.

### 6\.3Evaluation Protocol and Scoring

#### Replay\.

For every model and chain, the harness copiesG\(0\)G^\{\(0\)\}into an isolated workspace, records file hashes, and runs a preflight check whose findings are logged as pre\-existing conditions and are never attributed to the model\. At turnttonly the promptutu\_\{t\}is released; the model continues from its ownG\(t−1\)G^\{\(t\-1\)\}with no access to the criteria, to a reference implementation, or to any later human version\. After the sixth turn the workspace is frozen and the complete diff againstG\(0\)G^\{\(0\)\}is recorded\.

#### Final\-product\-only judging\.

All six turns are evaluated on the frozenG\(T\)G^\{\(T\)\}\. A positive criterioniiis binary, with no partial credit, and passes only when every explicit condition in its description is established by admissible evidence:

xm,k,i=∏j∈𝒞i\[condition​j​holds in​G\(T\)\]⋅\[evidence for​j​is admissible\]\.x\_\{m,k,i\}=\\prod\_\{j\\in\\mathcal\{C\}\_\{i\}\}\\mathbf\{1\}\\\!\\left\[\\text\{condition \}j\\text\{ holds in \}G^\{\(T\)\}\\right\]\\cdot\\mathbf\{1\}\\\!\\left\[\\text\{evidence for \}j\\text\{ is admissible\}\\right\]\.\(11\)Herexm,k,i∈\{0,1\}x\_\{m,k,i\}\\in\\\{0,1\\\}is the judgment for modelmm, full\-corpus roundkk, and positive itemii\. Admissible evidence is a code location on a reachable call path, a runtime log, a screenshot, or an audio trace; comments, unused configuration, dead code, and the model’s own description are not admissible, and insufficient evidence resolves toxm,k,i=0x\_\{m,k,i\}=0\.

#### Difficulty\-leaning discrimination weights\.

LetK=3K=3and𝒫\\mathcal\{P\}be the 604 positive items pooled across all 17 chains\. For itemii, letnin\_\{i\}be its number of judgments andnipassn\_\{i\}^\{\\mathrm\{pass\}\}its number of passes\. We assign a common weight

pi=nipassni,wi=pi​\(1−pi\)2\.p\_\{i\}=\\frac\{n\_\{i\}^\{\\mathrm\{pass\}\}\}\{n\_\{i\}\},\\qquad w\_\{i\}=p\_\{i\}\(1\-p\_\{i\}\)^\{2\}\.\(12\)Thepi​\(1−pi\)p\_\{i\}\(1\-p\_\{i\}\)factor rewards items that discriminate among systems, while the additional\(1−pi\)\(1\-p\_\{i\}\)factor tilts weight toward harder items\. Items passed by everyone or no one receive zero weight\. A model’s positive\-item score in roundkkis the pooled weighted pass rate

Sm,k\+=100​∑i∈𝒫wi​xm,k,i∑i∈𝒫wi\.S^\{\+\}\_\{m,k\}=100\\,\\frac\{\\sum\_\{i\\in\\mathcal\{P\}\}w\_\{i\}x\_\{m,k,i\}\}\{\\sum\_\{i\\in\\mathcal\{P\}\}w\_\{i\}\}\.\(13\)Pooling is performed over the full item bank rather than by averaging a score for each game first; consequently, chains containing more discriminative hard items contribute more weight\. Letℛ\\mathcal\{R\}be the applicable regression rows,rm,k,j∈\{0,1\}r\_\{m,k,j\}\\in\\\{0,1\\\}indicate whether regressionjjis triggered, andqj<0q\_\{j\}<0be its predefined penalty\. The reported score is

Sm,k=Sm,k\+\+∑j∈ℛqj​rm,k,j\.S\_\{m,k\}=S^\{\+\}\_\{m,k\}\+\\sum\_\{j\\in\\mathcal\{R\}\}q\_\{j\}r\_\{m,k,j\}\.\(14\)Thus, requirement and challenge items contribute positive credit, while triggered regression rows deduct points\. Priority\-specific, dimensional, and turn\-specific positive\-item values use the same weighted\-pass formula after restricting𝒫\\mathcal\{P\}to the corresponding subset\. There is no P0 gate or runtime cap in the reported leaderboard\. Build/start failures, dead products, and core\-loop failures are also reported as separate diagnostics\.

#### Result aggregation\.

The reported overall, dimension\-specific, and turn\-specific scores use the same unified result summary for every model\. We retain the three round values to expose run\-to\-run variation; chain\-level statistical comparisons should treat the chain, rather than an individual rubric item, as the independent unit\.

### 6\.4Experimental Results and Analysis

#### Setup\.

We report 15 model variants on the 17\-game JavaScript collection, with six turns per game and three full\-corpus evaluation rounds\. This yields 102 turns per model and 765 model–game runs in total\. Overall, dimension\-specific, and turn\-specific results use the same 15\-model cohort\.

Table 5:Overall and six\-dimensionalGameOptscores for 15 model variants on 17 JavaScript games\. All columns use the pooled discrimination\-weighted result summary\.
#### Overall results\.

Claude\-Opus\-5 ranks first at 93\.96, followed by Claude\-Fable\-5 at 89\.31 and Kimi\-K3 at 84\.66\. Across all 15 models, the median is 74\.24, and the 58\.07\-point range from Claude\-Opus\-5 to Seed\-2\.1\-Pro shows that the difficulty\-leaning weighting separates the field substantially\. Round stability also varies sharply\. GPT\-5\.6\-sol spans only 0\.13 points across rounds, whereas Gemini\-3\.5\-Flash and MiniMax\-M3 span 36\.12 and 33\.67 points\. The six\-dimensional columns additionally expose whether similar overall scores arise from balanced capability or from sharply different strengths across gameplay, level design, balance, art, UI, and audio\.

![Refer to caption](https://arxiv.org/html/2608.21833v1/gameopt_sixdim.png)Figure 12:Six\-dimensional capability profiles\.Rows denote the six optimization dimensions and columns denote models\. Each cell reports the final\-product rubric score; black outlines mark the best\-performing model in each dimension\. The right column reports means over all 15 models, while the bottom row reports the leaderboard Overall score\.
#### Six\-dimensional capability\.

Figure[12](https://arxiv.org/html/2608.21833#S6.F12)reveals structure hidden by the overall score\. Across all 15 models, numeric balance \(66\.11\) and level design \(66\.36\) have the lowest means, whereas audio is highest \(81\.97\)\. Numeric balance is the weakest dimension for 7/15 models and level design for 3/15; audio is the strongest for 10/15\. High overall performance does not imply an even profile: Claude\-Opus\-5 ranges from 80\.80 in balance to 100\.00 in art, and GPT\-5\.6\-sol spans 29\.19 points between UI and audio\. These profiles are descriptive rather than causal, but they show why the pooled Overall score should be accompanied by dimension\-specific results\.

Figure 13:Performance across six optimization turns\.Each value is the discrimination\-weighted final fulfillment score for requests introduced at the corresponding turn\. Thin gray curves show individual models and the green curve is their macro\-average\.
#### Multi\-turn behavior\.

The model macro\-average follows a pronounced non\-monotonic pattern: 77\.30, 73\.50, 63\.85, 65\.48, 81\.27, and 81\.97 from turns one through six\. Every model reaches its lowest value at turn two, three, or four \(3, 7, and 5 models, respectively\), and none is weakest at turn one, five, or six\. This rules out a simple monotonic turn\-wise decay in the current data, but it does not establish long\-context stability\. Turn and content are confounded—for example, audio is always turn six, while balance is concentrated at turn three and art at turn four—so the rebound at turns five and six cannot be attributed to accumulated context\. Longer, counterbalanced trajectories are needed to isolate retention from request and dimension difficulty\.

![Refer to caption](https://arxiv.org/html/2608.21833v1/gameopt_cases.png)

Figure 14:Aggregate integration outcomes and representative extremes and case study\.
#### Case study: authored features versus integrated products\.

Figure[14](https://arxiv.org/html/2608.21833#S6.F14)places the paired examples in their aggregate context by separating final products into fully integrated, playable\-but\-incomplete, and unplayable outcomes\. The examples then control the requested change by contrasting two models on the same first\-turn interface optimization\. kimi\-K3’s final product makes the redesigned selection screen, runtime HUD, and settlement state all reachable, demonstrating that the interface change remains connected to the game loop\. MiniMax\-M3 also authors the requested arcade\-style CSS, but a single structural omission—the closing</style\>tag—changes how the browser parses the entire document\. The body has no rendered children,main\.jsnever executes, and the same blank frame is observed at the menu, selection, HUD, and settlement probes\. Thus, every requested dimension becomes unreachable even though the feature code is present\. This paired case explains the mechanism behind the tails of the aggregate distribution: optimization quality depends on an executable, end\-to\-end product, not merely on locally plausible edits\.

## 7Conclusion

We introducedGameXpert\-Bench, an execution\-grounded benchmark for evaluating coding agents across game generation, diagnosis and repair, and human\-guided optimization\. Across all three stages, the central finding is that producing an apparently plausible implementation is easier than delivering a rich, verified, and regression\-free game\. By evaluating not only what agents write but also what their games actually do,GameXpert\-Benchprovides a foundation for developing agents that can participate reliably in the full game\-development lifecycle rather than only produce its first playable draft\.

## References

- \[1\]S\. Aleem, L\. F\. Capretz, and F\. Ahmed\(2016\)Game development software engineering process life cycle: a systematic review\.Journal of Software Engineering Research and Development4\(1\),pp\. 6\.Cited by:[§1](https://arxiv.org/html/2608.21833#S1.p3.1),[§3\.1](https://arxiv.org/html/2608.21833#S3.SS1.p1.1)\.
- \[2\]Alibaba Cloud\(2026\)Qwen3\.7\-Max model information\.Note:Alibaba Cloud Model Studio documentationLast updated July 24, 2026; accessed 2026\-08\-10External Links:[Link](https://www.alibabacloud.com/help/en/model-studio/qwen3-7-max)Cited by:[§4\.4](https://arxiv.org/html/2608.21833#S4.SS4.SSS0.Px1.p1.1)\.
- \[3\]Anthropic\(2026\)Choosing the right Claude model\.Note:Claude Platform documentationCovers Claude Opus 5, Claude Fable 5, Claude Opus 4\.8, and Claude Opus 4\.7; accessed 2026\-08\-10External Links:[Link](https://platform.claude.com/docs/en/about-claude/models/choosing-a-model)Cited by:[§4\.4](https://arxiv.org/html/2608.21833#S4.SS4.SSS0.Px1.p1.1)\.
- \[4\]ByteDance Seed Team\(2026\)Seed2\.1 officially released: advancing AI productivity\.Note:Official ByteDance Seed technical blogReleased June 23, 2026; covers Seed2\.1 Pro; accessed 2026\-08\-10External Links:[Link](https://seed.bytedance.com/en/blog/seed2-1-officially-released-advancing-ai-productivity)Cited by:[§4\.4](https://arxiv.org/html/2608.21833#S4.SS4.SSS0.Px1.p1.1)\.
- \[5\]W\. Chi, Y\. Fang, A\. Yayavaram, S\. Yayavaram, S\. Karten, Q\. A\. Wei, R\. Chen, A\. Wang, V\. Chen, A\. Talwalkar,et al\.\(2026\)Gamedevbench: evaluating agentic capabilities through game development\.arXiv preprint arXiv:2602\.11103\.Cited by:[Table 1](https://arxiv.org/html/2608.21833#S1.T1.7.1.2.1),[§1](https://arxiv.org/html/2608.21833#S1.p1.1),[§2](https://arxiv.org/html/2608.21833#S2.SS0.SSS0.Px1.p1.1)\.
- \[6\]DeepSeek\-AI\(2026\)DeepSeek\-V4\-Flash update\.Note:DeepSeek API documentation change logReleased July 31, 2026; accessed 2026\-08\-10External Links:[Link](https://api-docs.deepseek.com/updates/)Cited by:[§4\.4](https://arxiv.org/html/2608.21833#S4.SS4.SSS0.Px1.p1.1)\.
- \[7\]Google DeepMind\(2026\)What’s new in Gemini 3\.5 Flash\.Note:Google AI for DevelopersLast updated July 30, 2026; accessed 2026\-08\-10External Links:[Link](https://ai.google.dev/gemini-api/docs/whats-new-gemini-3.5)Cited by:[§4\.4](https://arxiv.org/html/2608.21833#S4.SS4.SSS0.Px1.p1.1)\.
- \[8\]Y\. Huang, B\. Li, N\. Li, Z\. Wang, K\. Chen, H\. Ge, Q\. Si, Y\. Shen, R\. Yang, G\. Wang,et al\.\(2026\)GUI agents for continual game generation\.arXiv preprint arXiv:2605\.28258\.Cited by:[§2](https://arxiv.org/html/2608.21833#S2.SS0.SSS0.Px3.p1.1)\.
- \[9\]C\. Jia, R\. Wan, T\. Sun, W\. Tan, B\. Wan, Y\. Tong, G\. Sheng, and H\. Xu\(2026\)GameGen\-verifier: parallel keypoint\-based verification for llm\-generated games via runtime state injection\.arXiv preprint arXiv:2605\.07442\.Cited by:[§1](https://arxiv.org/html/2608.21833#S1.p2.1),[§2](https://arxiv.org/html/2608.21833#S2.SS0.SSS0.Px1.p1.1)\.
- \[10\]S\. Jiang, C\. Chen, and Z\. Chen\(2026\)GBQA: a game benchmark for evaluating llms as quality assurance engineers\.arXiv preprint arXiv:2604\.02648\.Cited by:[Table 1](https://arxiv.org/html/2608.21833#S1.T1.7.1.7.1),[§2](https://arxiv.org/html/2608.21833#S2.SS0.SSS0.Px2.p1.1)\.
- \[11\]Y\. Jiang, J\. Hu, Q\. Xiao, Y\. Zheng, R\. Ma, K\. Feng, J\. Han, T\. Peng, K\. Fan, M\. Zhang,et al\.\(2026\)Opengame: open agentic coding for games\.arXiv preprint arXiv:2604\.18394\.Cited by:[Table 1](https://arxiv.org/html/2608.21833#S1.T1.7.1.4.1),[§1](https://arxiv.org/html/2608.21833#S1.p1.1),[§1](https://arxiv.org/html/2608.21833#S1.p2.1),[§2](https://arxiv.org/html/2608.21833#S2.SS0.SSS0.Px1.p1.1)\.
- \[12\]C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. Narasimhan\(2024\)SWE\-bench: can language models resolve real\-world github issues?\.InThe Twelfth International Conference on Learning Representations,Cited by:[§5\.3](https://arxiv.org/html/2608.21833#S5.SS3.SSS0.Px2.p2.1),[§5](https://arxiv.org/html/2608.21833#S5.p1.1)\.
- \[13\]C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan\(2024\)Swe\-bench: can language models resolve real\-world github issues?\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 54107–54157\.Cited by:[§2](https://arxiv.org/html/2608.21833#S2.SS0.SSS0.Px2.p1.1)\.
- \[14\]Kimi Team\(2026\)Kimi K3: open frontier intelligence\.Note:Official Kimi technical blogAccessed 2026\-08\-10External Links:[Link](https://www.kimi.com/blog/kimi-k3)Cited by:[§4\.4](https://arxiv.org/html/2608.21833#S4.SS4.SSS0.Px1.p1.1)\.
- \[15\]B\. La, S\. Chang, B\. Kim, J\. Bae, A\. A\. Beg, S\. Chang, and G\. Gonzalez\-Pumariega\(2026\)GameEngineBench: evaluating coding agents on real c\+\+ runtime environments\.arXiv preprint arXiv:2607\.03525\.Cited by:[Table 1](https://arxiv.org/html/2608.21833#S1.T1.7.1.3.1),[§1](https://arxiv.org/html/2608.21833#S1.p1.1),[§2](https://arxiv.org/html/2608.21833#S2.SS0.SSS0.Px1.p1.1)\.
- \[16\]T\. Luo, R\. Wang, J\. Bi, C\. Xu, Z\. Tang, J\. Chen, J\. Liang, K\. Ji, S\. Guo, Y\. Du,et al\.\(2026\)GameCraft\-bench: can agents build playable games end\-to\-end in a real game engine?\.arXiv preprint arXiv:2606\.17861\.Cited by:[Table 1](https://arxiv.org/html/2608.21833#S1.T1.7.1.6.1),[§1](https://arxiv.org/html/2608.21833#S1.p1.1),[§1](https://arxiv.org/html/2608.21833#S1.p2.1),[§2](https://arxiv.org/html/2608.21833#S2.SS0.SSS0.Px1.p1.1)\.
- \[17\]MiniMax\(2026\)MiniMax M3: frontier coding, 1m context, native multimodality—all in one model\.Note:Official MiniMax research blogReleased June 1, 2026; accessed 2026\-08\-10External Links:[Link](https://www.minimax.io/blog/minimax-m3)Cited by:[§4\.4](https://arxiv.org/html/2608.21833#S4.SS4.SSS0.Px1.p1.1)\.
- \[18\]OpenAI\(2026\)GPT\-5\.6 Sol and GPT\-5\.5 model documentation\.Note:OpenAI API documentation[https://developers\.openai\.com/api/docs/models/gpt\-5\.6\-sol](https://developers.openai.com/api/docs/models/gpt-5.6-sol)and[https://developers\.openai\.com/api/docs/models/gpt\-5\.5](https://developers.openai.com/api/docs/models/gpt-5.5); accessed 2026\-08\-10Cited by:[§4\.4](https://arxiv.org/html/2608.21833#S4.SS4.SSS0.Px1.p1.1)\.
- \[19\]Z\. Peng, W\. Tao, X\. Yin, C\. Ying, Y\. Luo, and Y\. Guo\(2026\)PlayCoder: making llm\-generated gui code playable\.Proceedings of the ACM on Software Engineering3\(FSE\),pp\. 2003–2026\.Cited by:[Table 1](https://arxiv.org/html/2608.21833#S1.T1.7.1.8.1),[§2](https://arxiv.org/html/2608.21833#S2.SS0.SSS0.Px2.p1.1)\.
- \[20\]M\. R\. Taesiri, A\. Ghildyal, S\. Zadtootaghaj, N\. Barman, and C\. Bezemer\(2026\)Videogameqa\-bench: evaluating vision\-language models for video game quality assurance\.Advances in Neural Information Processing Systems38\.Cited by:[§2](https://arxiv.org/html/2608.21833#S2.SS0.SSS0.Px2.p1.1)\.
- \[21\]Tencent Hunyuan Team\(2026\)Hy3: a reasoning and agent model\.Note:Official model repository and model cardAccessed 2026\-08\-10External Links:[Link](https://github.com/Tencent-Hunyuan/Hy3)Cited by:[§4\.4](https://arxiv.org/html/2608.21833#S4.SS4.SSS0.Px1.p1.1)\.
- \[22\]Y\. Wu, Z\. Zhao, S\. Li, H\. H\. Lee, J\. Zhu, S\. Wu, T\. Yu, S\. Li, L\. Zhang, X\. Fan,et al\.\(2026\)SWE\-together: evaluating coding agents in interactive user sessions\.arXiv preprint arXiv:2606\.29957\.Cited by:[Table 1](https://arxiv.org/html/2608.21833#S1.T1.7.1.9.1),[§2](https://arxiv.org/html/2608.21833#S2.SS0.SSS0.Px3.p1.1)\.
- \[23\]Z\.AI\(2026\)GLM\-5\.2 and GLM\-5\.1 model documentation\.Note:Z\.AI Developer Documentation[https://docs\.z\.ai/guides/llm/glm\-5\.2](https://docs.z.ai/guides/llm/glm-5.2)and[https://docs\.z\.ai/guides/llm/glm\-5\.1](https://docs.z.ai/guides/llm/glm-5.1); accessed 2026\-08\-10Cited by:[§4\.4](https://arxiv.org/html/2608.21833#S4.SS4.SSS0.Px1.p1.1)\.
- \[24\]W\. Zhang, G\. You, H\. Zhao, T\. Zhu, H\. Wang, X\. Tang, M\. Dai, J\. Gu, D\. Dong, J\. Wu,et al\.\(2026\)WebGameBench: requirement\-to\-application evaluation for coding agents via browser\-native games\.arXiv preprint arXiv:2605\.17637\.Cited by:[Table 1](https://arxiv.org/html/2608.21833#S1.T1.7.1.5.1),[§1](https://arxiv.org/html/2608.21833#S1.p1.1),[§1](https://arxiv.org/html/2608.21833#S1.p2.1),[§2](https://arxiv.org/html/2608.21833#S2.SS0.SSS0.Px1.p1.1)\.
- \[25\]R\. Zubek\(2020\)Elements of game design\.MIT Press\.Cited by:[§4\.3](https://arxiv.org/html/2608.21833#S4.SS3.p2.1)\.

## Appendix AAdditional Details of GameGen

This appendix provides additional details of theGameGencorpus and its evaluation\. We report the category distribution, representative full prompts, the complete Shared Rubric for one example game, complete 2D and 3D subset leaderboards, and per\-game event pass rates\. All source material originally written in Chinese is translated into English\.

### A\.1Corpus Statistics and Representative Prompts

#### Category statistics\.

GameGencontains 97 games from 11 categories, comprising 53 2D games and 44 3D games\. As shown in[Table6](https://arxiv.org/html/2608.21833#A1.T6), most categories contain both 2D and 3D tasks\. Simulation and Management is the only category without a 3D task in the current corpus\.

Table 6:Category distribution of theGameGencorpus\. The Share column is computed over all 97 games\.
#### Representative full prompts\.

Below, we provide one complete 2D prompt and one complete 3D prompt from each category\. Since Simulation and Management contains no 3D task, only its 2D prompt is reported\. The prompts are translated without omitting any requested gameplay feature\.

Table 7:Representative full prompts fromGameGen\(Part I of III\)\.Table 8:Representative full prompts fromGameGen\(Part II of III\)\.Table 9:Representative full prompts fromGameGen\(Part III of III\)\.

### A\.2Example Shared Rubric: 2048

To illustrate the event\-level evaluation used byGameGen, we provide the complete Shared Rubric for the 2048 task\. Core events determine Completeness, whereas bonus events determine Richness\. The two components are reported separately in[Tables10](https://arxiv.org/html/2608.21833#A1.T10)and[11](https://arxiv.org/html/2608.21833#A1.T11)\.

Table 10:Completeness checklist for the 2048 task\.Table 11:Richness checklist for the 2048 task\.
### A\.3Complete 2D and 3D Leaderboards

We partition the 97GameGentasks into 53 2D games and 44 3D games and report the complete subset\-level leaderboards in[Tables12](https://arxiv.org/html/2608.21833#A1.T12)and[13](https://arxiv.org/html/2608.21833#A1.T13)\. All scores are reported on a\[0,100\]\[0,100\]scale\. Overall scores are computed from unrounded dimension scores; consequently, averaging the displayed one\-decimal values can differ from the reported Overall score by 0\.1 point\.

Table 12:Results on the 53\-game 2D subset ofGameGen\. The best and second\-best results in each column are shown in bold and underlined, respectively\.Table 13:Results on the 44\-game 3D subset ofGameGen\. The best and second\-best results in each column are shown in bold and underlined, respectively\.
### A\.4Per\-Game Event Pass Rates

[Figure15](https://arxiv.org/html/2608.21833#A1.F15)reports the event\-level pass rate for every game–model pair\. The heatmap exposes substantial variation across games that is hidden by aggregate model\-level scores: even strong models encounter isolated difficult games, while lower\-ranked models occasionally perform well on particular tasks\.

![Refer to caption](https://arxiv.org/arxiv/extracted/7980050/pics/gamegen_diagnostics/per_game_pass_rate_heatmap.png)Figure 15:Per\-game event pass rates across all 97GameGentasks and 15 evaluated models\. Rows correspond to games, columns correspond to models, and each cell reports the fraction of evaluated checklist events that pass\. Green indicates a higher pass rate and red indicates a lower pass rate\.

## Appendix BAdditional Details of GameFix

This section expands §[5](https://arxiv.org/html/2608.21833#S5)with the F2P/P2P test definitions and multi\-bug isolation \(§[B\.1](https://arxiv.org/html/2608.21833#A2.SS1)\), the six agentic abilities measured by the benchmark \(§[B\.2](https://arxiv.org/html/2608.21833#A2.SS2)\), the complete secondary metrics for all 17 models \(§[B\.3](https://arxiv.org/html/2608.21833#A2.SS3)\), and the reasoning\-level trajectory analysis \(§[B\.4](https://arxiv.org/html/2608.21833#A2.SS4)\)\.

### B\.1F2P and P2P test definitions

A test case is*not*an equality check against the gold constant: it is a hand\-written boolean predicate over the running game’s observable state, expressed as abehavioural tolerance band\(e\.g\.,a value must land in\(0, 1000\], a displacement must exceed a threshold, or a velocity must have the correct sign\)\. The gold patch is used only at*authoring*time to certify that the tests fail on the mutated game, pass on the gold game, and remain stable across repeated gold runs\. It plays no role when grading a candidate repair: any repair whose behaviour falls inside the intended band is credited\.

- •Fail\-to\-Pass \(F2P\): a predicate on the intended behaviour broken by the injected bug\. It fails on the mutated game and must pass after a correct repair\.
- •Pass\-to\-Pass \(P2P\): a predicate on behaviour that must not regress\. It passes on the gold game and must remain passing after repair, guarding against over\-editing\.

Each injected bug is counted as fixed only when its F2P test passes and all associated P2P tests remain passing\. The task score is then the percentage of injected bugs that satisfy this criterion\.

#### Multi\-bug isolation: no cross\-bug test contamination\.

Because each task co\-injects 19–27 bugs into one game, a naive assertion set admits two silent failure modes: a bug’s F2P could be*masked*green by a neighbour’s injection \(a free pass for an unfixed bug\), or its P2P could be*broken*by one \(a spurious regression charged to an otherwise\-correct repair\)\. We exclude both with an authoring\-time gate\. For each candidate task we run three configurations of the game and require, per bug sitess:

- •gold\-green: with the gold patch applied \(all bugs fixed\), every F2P ofsspasses; the assertion is satisfiable\.
- •coupled\-base\-red: with all bugs co\-injected \(the base the candidate is judged on\),ss’s F2P is red; no neighbour masks it into a free pass\.
- •single\-bug\-red: with onlyssinjected,ss’s F2P is red; the bug genuinely causes the symptom, not a merge artefact\.
- •P2P survives the merged base:ss’s P2P passes on gold*and*stays green on the fully\-injected base; no co\-injected neighbour can break it\. A P2P any neighbour could break is rejected before shipping\.

together with a stability check \(gold→\\toall P2P green forN=5N\{=\}5consecutive runs, zero flake\)\. Bugs failing any check are dropped and replaced from the pool under a line\-disjointness constraint \(no two bugs edit overlapping lines,±\\pma window\), and the gate re\-runs until≥20\\geq 20isolated bugs ship\. The four booleans are stored per task; e\.g\.,avalanche\-l5recordsgold\_green = coupled\_base\_red = indep\_base\_red\_all = trueandp2p\_stable\_runs = 5\. This gate has a direct consequence at grading time\. Because the merged base is provably red on every F2P and green on every P2P, a P2P failure on a candidate patch must be a regression the candidate itself introduced, not contamination from an unfixed neighbour, so the P2P gate also serves as our regression detector\. Onavalanche\-l5/*Self\-Discovery*, for instance, Hy3’s fix to the cable\-car trigger drops a guard the gold patch keeps, so theinvuln\-stunP2P \(“player can move forward”\) fails on a bug whose own F2P passes: a self\-inflicted regression, not a neighbour’s contamination\.

### B\.2The six agentic abilities the benchmark measures

The multi\-bug, small\-edit design \(§[5\.2](https://arxiv.org/html/2608.21833#S5.SS2)\) does not test a single “can it edit code” ability\. Each individual mutation is invisible in the source and breaks only when the game is played, and a 19–27\-bug task forces several distinct abilities to act together\. We separate the demanded skill into six axes, shown in Table[14](https://arxiv.org/html/2608.21833#A2.T14), contrasting Hy3 \(the weakest model we analyse\) with three stronger models, each with a verbatim trajectory quote\.

Table 14:The six agentic ability axes, and how three models place on each\. Each cell carries a representative verbatim agent quote; quotes marked \[tr\.\] are translated from the agent’s original Chinese\.Two points about the table\. First, the axes are not independent:*behavioural verification*and*regression control*are two sides of whether the model runs the game to completion, the first being the ability to build a real test harness and the second being what that harness then catches\. Second, five of the six axes appear mainly under*Self\-Discovery*\. Under*Explicit Issue*the symptom list supplies the localisation, the completeness target, and \(per §[B\.1](https://arxiv.org/html/2608.21833#A2.SS1.SSS0.Px1)\) a guard against silent regression, so*Explicit Issue*collapses several axes into “follow the list\.” Removing the list, as*Self\-Discovery*does, turns each axis into a separate, discriminating demand, so the*Explicit Issue*→\\to*Self\-Discovery*cliff measures the six axes rather than a single coding skill\.

### B\.3Full leaderboard with secondary metrics

Table[15](https://arxiv.org/html/2608.21833#A2.T15)reports, for all 17 models: pass@3 \(any of 3 runs solves the task\), pass3\(all 3 runs solve it\) and average@3 counts at score thresholds 100/95/90; mean agent turns per clean\-completed task; and mean per\-task uncached input / output tokens\. Blank token/turn tasks indicate a framework whose usage stream did not expose the field\. All are derived from the same live report module as the main leaderboard\.

Table 15:Secondary metrics for all 17 models\. Counts are out of 100 tasks \(pass@3, pass3\) or expected tasks \(avg@3\)\. Models evaluated under two agent frameworks are listed once per framework, given in parentheses; all other models are run under Claude Code\. The Hy3 figures are from the self\-deployed model \(see §[5\.4](https://arxiv.org/html/2608.21833#S5.SS4)\)\.
### B\.4Reasoning\-level trajectory analysis

We analyse agent trajectories only within the information exposed by each model provider and framework\. Frontier models generally do not provide raw chain\-of\-thought traces, so our analysis does not rely on hidden reasoning\. Instead, we examine the available reasoning summaries or planning preambles, the model’s visible outputs, tool\-use sequences, code edits, and final task outcomes\. For each representative model, we inspect these observable signals across selected tasks and relate failure cases to the benchmark taxonomy\. This allows us to compare search, repair, verification, and stopping behaviour without requiring access to private chain\-of\-thought\. Table[16](https://arxiv.org/html/2608.21833#A2.T16)summarises the dominant failure mechanism for each analysed model\.

Table 16:The seven trajectory\-analysed models \(marked†in Table[3](https://arxiv.org/html/2608.21833#S5.T3)\), their profile and dominant cliff mechanism\. GPT\-5\.6\-solStrictis shown as Codex / Claude Code\. Cliff is the drop from*Explicit Issue*to*Self\-Discovery*; all values 3\-run average@3\.

## Appendix CGameOptDetails

This appendix provides supplementary details for the 17\-chain JavaScript collection evaluated in §[6\.4](https://arxiv.org/html/2608.21833#S6.SS4)\. We first summarize the chain inventory and rubric composition, then provide three request chains and one complete hidden rubric\. Each starting snapshot runs in the offline browser sandbox, so acceptance criteria may require rendered output as evidence\.

### C\.1Track Overview

Table 17:JavaScript chain inventory\.Table[17](https://arxiv.org/html/2608.21833#A3.T17)covers 17 chains, 102 turns, and 701 criteria\. Snapshot size counts authored source inG\(0\)G^\{\(0\)\}, excluding vendored engine builds andnode\_modules; difficulty is rated on a 1–5 scale\. A*first version*start denotes the first recorded version, whereas other rows use the named version from a co\-creation history\. The intended budget is one P0 and one regression check per turn\. Fourteen chains meet the P0 budget exactly, but*Street Racer*has no regression check\. Three representative chains are expanded in Table[20](https://arxiv.org/html/2608.21833#A3.T20)\.

Table 18:Acceptance\-criterion composition\.Percentages in Table[18](https://arxiv.org/html/2608.21833#A3.T18)use all criteria as the denominator\.diff\_analysiscriteria come from changes between adjacent human versions;real\_user\_followupcriteria restate later complaints from the same user\.

Table 19:Dimension order by turn\.TurnDimension123456Gameplay10241––Level–10–43–Balance221111–Art11294–UI42–29–Audio–––––17Each chain covers every dimension exactly once\. The ordering is only partially counterbalanced:audiois always turn six, so turn position and dimension cannot be fully separated, as noted in §[6\.4](https://arxiv.org/html/2608.21833#S6.SS4)\.

### C\.2Request Chains

The requests below are translated from the Chinese originals shown to the model\. The translations preserve register and deliberate vagueness; resolving phrases such as “make these boundaries more clearly visible” is part of the task\.

Table 20:Representative request chains\.\#DimensionRequest shown to the modelNP0Img\.Street Racer—v0001; 9 files; 92 kchar; difficulty 3; 39 criteria1UIUnify the whole game’s UI into a dark\-background street\-arcade look: near\-black translucent panels, bright\-yellow primary buttons and outlines, one self\-drawn icon set and colour code \(green = score / steady, orange = combo / heat, white = time and speed digits\)\. Replace the default system controls, and keep information at the screen edges so it does not block the driving view\. Cover the main menu, the car\-select page, the in\-game HUD and the results page\. Core requirements: \(1\) all four screens share the dark arcade style and self\-drawn controls, with a clear primary/secondary button hierarchy; \(2\) structure the in\-game HUD — minimap and settings entry top\-left, wanted/heat bar, score and time \(mm:ss\) top\-centre, combo plate top\-right, a virtual stick bottom\-left that actually drives the car, speed digits plus handbrake/reset/horn buttons bottom\-right; \(3\) complete the feedback — floating score on smashing props, a large combo readout whose multiplier changes colour, and a score tick\. The results page presents the run, highlights a new record, and uses the same primary/secondary buttons\.8182GameplayWhile driving, the car can enter or even pass straight through walls over a large area\. Please prevent this: the car should not be able to get inside walls or buildings\.6153ArtIn some places there are invisible air walls\. I would like these boundaries to be delimited and made more clearly visible\.6164LevelOn the screen where the game starts, let the player freely choose the map, and add more maps to choose from\.7175BalanceAmong the six cars, the supercar “Ghost X” has nearly the highest top speed, acceleration and handling — it is an all\-rounder, so there is no reason to pick any of the others\. Please rebalance the cars’ numbers: give every car a clear strength and a clear weakness so that each has a distinct role\. Also make the “weight” attribute actually affect how driving feels — for instance a heavy car is more stable and takes hits better but steers more sluggishly, while a light car is more agile but is knocked around more easily\. Do not let any single car be the best on every attribute\.6116AudioThe sound effects feel good overall, but there are two problems I want solved\. First, there is nowhere to turn the sound off — after playing a while with headphones I cannot lower the volume or mute, so I want a volume/mute control in the pause panel, and the setting should be remembered\. Second, when I smash a whole row of props the breaking sounds pile up in an instant into a loud crackling mush; please rein in this kind of high\-frequency effect so they do not all cram into the same moment\.611Deck\-Climb Roguelike—v0004; 9 files; 76 kchar; difficulty 3; 43 criteria1BalanceAdjust the game balance to the following number system: starting health 70, 3 energy per turn, a base hand of 5 cards; a basic attack costs 1 energy and deals about 6 damage, a defence card grants about 5 shield; normal enemies have 30–50 health, elites 80–120, and the boss about 200–300; after a fight the player picks 1 card out of 3, and may skip so the deck does not get too thick; the shop can sell cards and relics or remove a card, and a rest site restores about 30% of max health\.8142LevelChange the levels into a vertical branching tower\-climb map with combat, elite, event, rest, shop and treasure nodes\. Keep early enemy mechanics simple, introduce status ailments and multi\-enemy fights in the middle, and use elites and the boss late to test the strength of the deck; different routes should carry different risk and reward\.8153GameplayThe cards are currently not strategic enough to beat the final boss\. Please improve three things: first, there is no way to heal during a fight, so add one; second, the boss has too much health, so lower it somewhat; third, there are too few strategic card types — for example there is no “dodge” card that completely avoids the damage of one enemy attack, so add cards of that kind\.7144UIPlease add deck editing and a card compendium\.7155ArtRework the UI into the same fresh hand\-drawn campus style as the game: panels, buttons and cards should systematically carry a hand\-drawn quality \(outlines / texture / a rounded sticker feel\) across the main menu, the map, the combat HUD and the results screen, staying with the existing warm cream / grass / red\-blue\-yellow palette\. While you are at it, fix the places where controls overlap or sit too close together and end up occluding key information \(health, energy, gold, enemy health bars, the hand\)\.7256AudioThe sound is a bit muddy right now: many actions sound the same — playing a card, taking a hit, getting a debuff and losing are often the same sound — and drawing a card makes no sound at all\. On top of that, when I play cards quickly in a row, or a pile of enemies acts at once, the sounds all stack up and it gets very loud, almost clipping\. Could you separate the sounds for these key actions, fill in the ones that are silent, and give me a mute/volume control?611Hero Arena—v0425; 49 files; 546 kchar; difficulty 3; 40 criteria1BalanceThe talent level cap is 20\. Please make this cap explicit and enforce it in the system\.6142GameplayWhen each wave’s timer ends, first bring every character and monster in the game to a halt; then delete all the monsters’ sprite frames and entities, and only enter the shop screen once the deletion has finished\. The deletion can be done with a spin\-and\-shrink tween\.7173ArtAdd suitable sound effects for button presses, ranged weapon fire, melee weapon swings and thrusts, and shell explosions\.7304UIChange the way the Extreme difficulty is entered: clicking the Extreme option should no longer go straight into the game, but show an Extreme leaderboard first, with a Start button below it; only pressing Start enters the Extreme game scene\.7155LevelMonster spawn positions in the arena are too random — several groups in a row often come from the same direction, so the player only has to keep moving the other way\. Please stagger each group’s spawn direction around the player so monsters close in from different sides; at the same time keep the existing no\-spawn\-in\-your\-face distance, and make sure spawn points stay inside the arena bounds\.6136AudioFrom the main town to combat and on to the victory/defeat results, the background music is one and the same loop — the mood never changes; and the moment I win or lose it is dead silent\. I want combat to have its own battle music, and the victory/defeat results screen to switch to a matching victory or defeat track \(or sting\)\. Also, when health is nearly gone, give me a continuous low\-health warning sound so I know I am about to die\. Do not break the existing fire, hit and button sounds\.710Table[20](https://arxiv.org/html/2608.21833#A3.T20)reports all six requests for each example, along with the total, P0, and rendered\-output criterion counts\. The examples span interface, gameplay, level design, balance, art, and audio optimization\.*Deck\-Climb Roguelike*turn 1 is a fully quantified balance request, while*Hero Arena*turn 3 is expanded below\.

### C\.3Rubric Example

Table[21](https://arxiv.org/html/2608.21833#A3.T21)gives the complete hidden rubric for*Hero Arena*, turn 3\. Its visible request asks only for suitable sound effects for buttons, ranged fire, two melee attacks, and shell explosions\. All seven items use code evidence, so the table omits a redundant evidence column\.

Table 21:Example hidden rubric:*Hero Arena*, turn 3\.Three of the seven criteria are P0\. Two of them \(R1andR5\) come fromreal\_user\_followup: later reports that the buttons remained silent and that no output was audible motivate an explicit trigger\-to\-output condition\.R6adds throttling from the adjacent human\-version diff, although the request does not mention it\.R7awards no positive credit and can only subtract\. This example illustrates the provenance result in §[6\.4](https://arxiv.org/html/2608.21833#S6.SS4): satisfying the stated request is only the first layer of evaluation\.