Metraipolis › archive › dc_78256f124afe

GameASG-Bench: Benchmarking Autonomous Software Generation for Game Development — arXiv:2609.21293

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.21293
License (as archived)
CC0-1.0
Archived
2026-10-10T01:05:11.534Z
SHA-256 of the archived text
ec13cc1525f7f2baaf55182e94e4354a25cbba0ac078a86fd077fa918d7878c9
Stable URL of this page
https://metraipolis.com/documento/dc_78256f124afe

How to cite

GameASG-Bench: Benchmarking Autonomous Software Generation for Game Development — arXiv:2609.21293 — arXiv (en), https://arxiv.org/abs/2609.21293 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_78256f124afe (Metraipolis archive dc_78256f124afe, sha256 ec13cc1525f7f2baaf55182e94e4354a25cbba0ac078a86fd077fa918d7878c9)
BibTeX
@misc{dc78256f124afe,
  title = {GameASG-Bench: Benchmarking Autonomous Software Generation for Game Development — arXiv:2609.21293},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.21293},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_78256f124afe; sha256 ec13cc1525f7f2baaf55182e94e4354a25cbba0ac078a86fd077fa918d7878c9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.21293. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Autonomous software generation (ASG) aims to turn human requirements into executable applications, but delivering these applications does not necessarily establish that their interacting components satisfy the specified behavioral requirements. We introduce GameASG-Bench, a benchmark that makes behavioral testability part of the generation task for game development. Our design declares an evaluation interface specification before generation, fixing legal starting scenarios, player-level actions, stable snapshots, rejection behavior, and invariants while leaving private implementations open. Concretely, we include: (i) static L1 checks that assess source-level compliance; and (ii) browser-executed L2 checks that combine semantic observations with real input and runtime evidence. We implement this protocol as 47 browser-native game-generation tasks spanning 12 primary genres and both 2D and 3D interaction, each with executable checks and an independently verified reference implementation. Our experiments answer four key questions about end-to-end agent performance, tool access and nominal turn budget, reasoning effort, and harness choice. Across nine agent stacks, the highest observed mean L2 check pass rate is 93.2%, yet the highest observed strict task success rate, requiring all L1 and applicable L2 prerequisite and core requirement checks, is only 55.3% (26/47 tasks). For DeepSeek-V4-Flash, full tool access and larger nominal turn budgets yield more strict task successes, while the strict task success rate is not monotonic in reasoning effort. Both tested harnesses achieve 18 strict task successes, but only ten tasks succeed under both. These results expose task-level compliance gaps that high average check pass rates actually obscure.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-21 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2609.21293 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T00:23:42.445Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).