Metraipolis › archive › dc_15dba9651dfe

System1-Arena: Which Decisions Can Be Left to System 1? — arXiv:2610.21244

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.21244
License (as archived)
CC0-1.0
Archived
2026-10-16T09:14:13.074Z
SHA-256 of the archived text
80b0cb81088f3bc23d8f592955631487724f87e8801b9890e00d9b0a95d93226
Stable URL of this page
https://metraipolis.com/documento/dc_15dba9651dfe

How to cite

System1-Arena: Which Decisions Can Be Left to System 1? — arXiv:2610.21244 — arXiv (en), https://arxiv.org/abs/2610.21244 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_15dba9651dfe (Metraipolis archive dc_15dba9651dfe, sha256 80b0cb81088f3bc23d8f592955631487724f87e8801b9890e00d9b0a95d93226)
BibTeX
@misc{dc15dba9651dfe,
  title = {System1-Arena: Which Decisions Can Be Left to System 1? — arXiv:2610.21244},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.21244},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_15dba9651dfe; sha256 80b0cb81088f3bc23d8f592955631487724f87e8801b9890e00d9b0a95d93226},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.21244. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human. System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. Which decisions can System-1 models handle, and which require generative reasoning? We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.21244 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T08:44:17.924Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).