Metraipolis › archive › dc_69361b264378

Code Owns the Simulation, Jev Owns the Evaluation — arXiv:2610.01834

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.01834
License (as archived)
CC0-1.0
Archived
2026-10-06T12:06:21.401Z
SHA-256 of the archived text
a0eb443dd4b986ca87af4188f2335bb6c13314aa6dea80a04f0ed303e5aef708
Stable URL of this page
https://metraipolis.com/documento/dc_69361b264378

How to cite

Code Owns the Simulation, Jev Owns the Evaluation — arXiv:2610.01834 — arXiv (en), https://arxiv.org/abs/2610.01834 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_69361b264378 (Metraipolis archive dc_69361b264378, sha256 a0eb443dd4b986ca87af4188f2335bb6c13314aa6dea80a04f0ed303e5aef708)
BibTeX
@misc{dc69361b264378,
  title = {Code Owns the Simulation, Jev Owns the Evaluation — arXiv:2610.01834},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.01834},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_69361b264378; sha256 a0eb443dd4b986ca87af4188f2335bb6c13314aa6dea80a04f0ed303e5aef708},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.01834. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Judgment models such as \jev{} return, in a single call and without reasoning text, a probability for each described option. This makes them attractive as an agent's action-selection layer, but it is unclear which decisions they can be trusted with. We test \jev{} on reflection tests, one-shot matrix games, the text game ALFWorld and robot control, and find a sharp boundary. \jev{} succeeds when the right option can be judged from what the input describes, which we call \emph{evaluation}. Specifically, it solves 99\% of the counterintuitive Cognitive Reflection Test questions. However, it fails when the right option depends on \emph{simulation} (i.e., predicting something not in the input), such as the opponent's action or the subgoal that must come first. In games, \jev{} plays suboptimally as if its rational opponent acted at random, because the opponent's action is not given. In ALFWorld, \jev{} favors commands that mention an object or place named in the task description. For example, given the task ``put a clean knife in the drawer'', \jev{} carries an unwashed knife straight to the drawer instead of first washing it at the sink. Surprisingly, many of these failures are not due to a lack of knowledge. Asked separately what the opponent will do, \jev{} usually answers correctly, and it responds well given the opponent's action. It fails when one call must both perform the simulation and evaluate based on it. This suggests letting code make the prediction or simulation. When code supplies it, such as a lookahead in ALFWorld and physics simulation in robot control, \jev{} becomes an expert controller through its general evaluation ability.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.01834 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T10:21:27.980Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).