A2Z GameSpec-Bench: How Faithfully Can Coding Agents Generate Games from Game Design Specifications?

Hugging Face Daily Papers Papers
coding-agents benchmark game-development specification-following evaluation software-engineering

Summary

论文提出 A2Z GameSpec-Bench，一个包含 100 份长篇游戏设计文档（GDD）的基准，用于评估编码代理在端到端游戏开发中遵循互锁需求的能力。结果显示现有代理在同时满足代码实现与实际游玩的联动需求上表现不佳，而需求级反馈可在两轮修订后将 GDD 忠实度提升 10.9%。

Delegating complete application development to coding agents requires preserving the intended design rather than simply producing plausible outputs through naive prompting. Game development provides a demanding testbed, as long-form Game Design Documents (GDDs) describe requirements that must work together across game logic, visual rendering, and player interactions. However, existing game-development benchmarks typically use compact specifications and provide limited support for evaluating interdependent requirements across these aspects in long-form GDDs. We introduce A2Z GameSpec-Bench, a benchmark of 100 long-form GDDs for evaluating end-to-end game development by agents. We measure faithfulness by checking whether the game satisfies the GDD requirements and preserves the relationships among them. Each GDD is turned into a dependency-aware contract that contains rules, constraints, and prerequisite relations. Following game-development practices, we combine source-code inspection with agent-generated test policies for scenario-based replay and adaptive playtesting. The contract remains fixed across agents and revision rounds, while judgments and evidence linked to the same requirements support consistent comparison and failure detection. Our evaluations show that current agents struggle to jointly satisfy interdependent requirements across code implementation and actual play. Requirement-specific feedback improves GDD Fidelity by 10.9% relative to self-revision after two rounds. A2Z GameSpec-Bench assesses end-to-end specification-following ability beyond implementation judgments and provides targeted feedback to support more faithful game development. Code and datasets are available at https://a2z-gamespec-bench.github.io.
Original Article
View Cached Full Text

Cached at: 10/01/26, 04:21 AM

Paper page - A2Z GameSpec-Bench: How Faithfully Can Coding Agents Generate Games from Game Design Specifications?

Source: https://huggingface.co/papers/2609.39564

Abstract

Delegatingcompleteapplicationdevelopmenttocodingagentsrequirespreservingtheintendeddesignratherthansimplyproducingplausibleoutputsthroughnaiveprompting.Gamedevelopmentprovidesademandingtestbed,aslong-formGameDesignDocuments(GDDs)describerequirementsthatmustworktogetheracrossgamelogic,visualrendering,andplayerinteractions.However,existinggame-developmentbenchmarkstypicallyusecompactspecificationsandprovidelimitedsupportforevaluatinginterdependentrequirementsacrosstheseaspectsinlong-formGDDs.WeintroduceA2ZGameSpec-Bench,abenchmarkof100long-formGDDsforevaluatingend-to-endgamedevelopmentbyagents.WemeasurefaithfulnessbycheckingwhetherthegamesatisfiestheGDDrequirementsandpreservestherelationshipsamongthem.EachGDDisturnedintoadependency-awarecontractthatcontainsrules,constraints,andprerequisiterelations.Followinggame-developmentpractices,wecombinesource-codeinspectionwithagent-generatedtestpoliciesforscenario-basedreplayandadaptiveplaytesting.Thecontractremainsfixedacrossagentsandrevisionrounds,whilejudgmentsandevidencelinkedtothesamerequirementssupportconsistentcomparisonandfailuredetection.Ourevaluationsshowthatcurrentagentsstruggletojointlysatisfyinterdependentrequirementsacrosscodeimplementationandactualplay.Requirement-specificfeedbackimprovesGDDFidelityby10.9%relativetoself-revisionaftertworounds.A2ZGameSpec-Benchassessesend-to-endspecification-followingabilitybeyondimplementationjudgmentsandprovidestargetedfeedbacktosupportmorefaithfulgamedevelopment.Codeanddatasetsareavailableathttps://a2z-gamespec-bench.github.io.

View arXiv pageView PDFProject pageGitHub2Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.39564 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.39564 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.39564 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?

arXiv cs.AI

The paper introduces GameXpert-Bench, a benchmark with three tracks to evaluate coding agents' game development capabilities across creation, bug repair, and optimization. It finds that current agents are better at initial generation than at defect discovery and multi-turn optimization.

SWE-Game: Can Coding Agents Build the Games We Want?

Hugging Face Daily Papers

We introduce SWE-Game, a benchmark of 247 tasks grounded in 41 executable Godot games covering five game-development task types. Across six models, Opus5 leads but best scores remain below 60/100, highlighting gaps in requirement fulfillment and gameplay logic.