Metraipolis › archive › dc_ccf5dc43be02

Rome Wasn't Built in One Turn: Replay-Grounded Reward Signals for Long-Horizon Freeciv Agents — arXiv:2610.41264

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.41264
License (as archived)
CC0-1.0
Archived
2026-11-03T07:41:25.729Z
SHA-256 of the archived text
66b8cae14c4362d9eea47aa75c7f1ce97eeb38a8a094ae809111bb0549c14ca7
Stable URL of this page
https://metraipolis.com/documento/dc_ccf5dc43be02

How to cite

Rome Wasn't Built in One Turn: Replay-Grounded Reward Signals for Long-Horizon Freeciv Agents — arXiv:2610.41264 — arXiv (en), https://arxiv.org/abs/2610.41264 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ccf5dc43be02 (Metraipolis archive dc_ccf5dc43be02, sha256 66b8cae14c4362d9eea47aa75c7f1ce97eeb38a8a094ae809111bb0549c14ca7)
BibTeX
@misc{dcccf5dc43be02,
  title = {Rome Wasn't Built in One Turn: Replay-Grounded Reward Signals for Long-Horizon Freeciv Agents — arXiv:2610.41264},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41264},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ccf5dc43be02; sha256 66b8cae14c4362d9eea47aa75c7f1ce97eeb38a8a094ae809111bb0549c14ca7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41264. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Training language-model agents to play long-horizon strategy games is hard because deterministic feedback is sparse and often arrives only at the end of a game. In-game scores can offer dense signals, but they do not reliably predict future outcomes: in 8-68% of position pairs across eight strategy games, the higher-scoring position leads to the worse outcome. We term this phenomenon scoreboard traps. In Freeciv, an open-source turn-based empire-building game where several players manage units, cities, and diplomacy over hundreds of turns, we look into better reward signals for gameplay agents. To facilitate this, we build CivHarness, an open system that saves current game states, branches on different decisions, and replays the game to future turns to provide measurable long-horizon feedback from saved positions. We use CivHarness to train CivTelescope, an LLM Judge that predicts which one of two game positions will yield the better outcome. On scoreboard-trap pairs, CivTelescope is 1.3-2.0 times more accurate than off-the-shelf language models, and it generalizes to new game starts, later turns, and unseen rules. We then train a Freeciv gameplay agent with CivTelescope as a dense reward, and call this recipe CivMarshmallow. When used in the middle stage of the game where scoreboard traps are common, CivMarshmallow reaches an average training phase gain of 8.94 points, 77% more than the best baseline, the end-of-game reward. Over the full 120-turn game, CivMarshmallow uses a hybrid reward that starts from the scoreboard or the end-of-game reward and shifts to CivTelescope as the game advances; this hybrid reward scores highest among the rewards we compare.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.41264 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T07:39:37.277Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).