Metraipolis › archive › dc_960251c9019f

OpenGameEval: Benchmarking Agentic Programming and Exploration in a Stateful Game Engine — arXiv:2610.02563

Source
arXiv
Original
https://arxiv.org/abs/2610.02563
License (as archived)
CC0-1.0
Archived
2026-10-05T22:50:07.447Z
SHA-256 of the archived text
5f826f63f16d49241d3a4ab47a07bc2d98ca03813c49614bd4dc3b23ab8dfaf7
Stable URL of this page
https://metraipolis.com/documento/dc_960251c9019f

How to cite

OpenGameEval: Benchmarking Agentic Programming and Exploration in a Stateful Game Engine — arXiv:2610.02563 — arXiv, https://arxiv.org/abs/2610.02563 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_960251c9019f (Metraipolis archive dc_960251c9019f, sha256 5f826f63f16d49241d3a4ab47a07bc2d98ca03813c49614bd4dc3b23ab8dfaf7)
BibTeX
@misc{dc960251c9019f,
  title = {OpenGameEval: Benchmarking Agentic Programming and Exploration in a Stateful Game Engine — arXiv:2610.02563},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.02563},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_960251c9019f; sha256 5f826f63f16d49241d3a4ab47a07bc2d98ca03813c49614bd4dc3b23ab8dfaf7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.02563. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We present OpenGameEval, a benchmark and evaluation framework for agentic game development inside Roblox Studio. It runs language models as agents in reproducible, stateful game-engine sessions and scores each run with executable checks, both on the edited scene and in a simulated play session. Most agentic coding benchmarks require exploration but score only final task success. OpenGameEval separates observation tools from editing tools in its eight-tool action space, so exploration can be measured directly. We measure the pass rates and exploration behavior of 13 frontier models on 84 human-curated core tasks, with 16 attempts per task. The tasks are hard for current models. The best model solves 51.7% of tasks on a single attempt and 39.4% five times out of five, and no tested model solves six of the tasks. Models at the frontier reach similar pass rates by solving different tasks: splitting tasks by the kind of work they require spreads the top five by 5.0pp on script-authoring tasks and 12.5pp on scene-change tasks. Exploration behavior predicts whether a run succeeds. Holding task and model fixed, a run that inspects every object a reference solution touches before acting on it passes 13.4pp more often than a run that inspects none of them on scene-only tasks, and 9.8pp more often on script-only tasks. We release the task suite, its place files, the per-task annotations, a plugin that runs the tasks inside Roblox Studio, and an updated leaderboard under the MIT license at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.02563 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-05T22:45:55.773Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).