JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines
Summary
JamSet and JamBench are introduced as a dataset and benchmark for project-level game code generation on the Godot engine, derived from Game Jam projects, with evaluation showing a capability cliff for AI models as project scale increases.
View Cached Full Text
Cached at: 06/20/26, 02:29 PM
Paper page - JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines
Source: https://huggingface.co/papers/2606.19830
Abstract
Game development frameworks and benchmarks were created using data from game jam competitions to evaluate code generation and project-level programming capabilities.
Current AI-drivengame developmenthas made substantial progress in asset generation, gameplay design, and web-based game coding, yet project-level code engineering on professional game engines remains largely unexplored due to the absence of large-scale datasets and deterministic evaluation methods. We present JamSet and JamBench, the first project-level game code framework dataset and benchmark built on a professional game engine. Our key insight is thatGame Jam competitions, community events where developers build complete games under tight time constraints, yield thousands of open-source projects suitable for this purpose. Building on theGodot engine’s text-based format andheadless executionmode, we design adeterministic verificationpipeline from file integrity to runtime behavior collection, distilling 8,133 verified projects from over 240,000 repositories. Of these, 300 manually verified projects form JamBench; the rest constitute JamSet. JamBench defines theme-driven generation and code completion tasks, evaluated through a pipeline combiningcompilation pass rates,Structural Completeness Score(SCS), andBehavioral Alignment Score(BAS). Evaluation of 9 frontier models reveals a capability cliff as project scale increases, with runtime pass rates dropping from 80.4% on small projects to 5.7% on large ones (Task2a). Code Agents improve compilation rates yet yield no gains in runtime behavioral quality, indicating that the bottleneck lies in architectural design rather than syntactic correctness. Experiments validate JamSet as effective training data. All data and code are publicly available.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2606\.19830
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.19830 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.19830 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.19830 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?
GameCraft-Bench is a benchmark for evaluating AI coding agents on end-to-end game generation from natural language descriptions using the Godot engine. The strongest agent achieves only 41.46%, showing the task remains highly challenging.
ProgramBench (5 minute read)
ProgramBench is a new benchmark that evaluates AI agents' ability to reconstruct complete software projects from compiled binaries and documentation without access to source code or decompilation tools.
BEAMS: Benchmarking and Evaluating AI for Modeling and Simulation
The BEAMS Initiative presents a benchmark suite for evaluating AI tools in modeling and simulation, focusing on human-centered and responsible AI practices. Tests reveal variability across LLM-based engines, with better performance in qualitative tasks than causal reasoning.
3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code
This paper introduces 3DCodeBench, a benchmark for evaluating vision-language models on procedural 3D modeling via code, and 3DCodeArena, a ranking platform based on pairwise human preferences.
Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack
This paper introduces BenchJack, an automated red-teaming system that systematically audits AI agent benchmarks by identifying reward-hacking exploits. It applies BenchJack to 10 popular benchmarks, surfacing 219 distinct flaws and demonstrating that evaluation pipelines lack an adversarial mindset, with the system reducing hackable-task ratios from near 100% to under 10% on four benchmarks.