Metraipolis › archive › dc_fc428e328d83

OptionWorld: A Benchmark for Diagnosing Failures of LLMs as Long-Term Personal Assistants — arXiv:2610.28491

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.28491
License (as archived)
CC0-1.0
Archived
2026-10-21T07:09:10.646Z
SHA-256 of the archived text
c71b1057e41b44eebfad598e5306f16a23da188327ccce962c2491d76c492680
Stable URL of this page
https://metraipolis.com/documento/dc_fc428e328d83

How to cite

OptionWorld: A Benchmark for Diagnosing Failures of LLMs as Long-Term Personal Assistants — arXiv:2610.28491 — arXiv (en), https://arxiv.org/abs/2610.28491 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fc428e328d83 (Metraipolis archive dc_fc428e328d83, sha256 c71b1057e41b44eebfad598e5306f16a23da188327ccce962c2491d76c492680)
BibTeX
@misc{dcfc428e328d83,
  title = {OptionWorld: A Benchmark for Diagnosing Failures of LLMs as Long-Term Personal Assistants — arXiv:2610.28491},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.28491},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fc428e328d83; sha256 c71b1057e41b44eebfad598e5306f16a23da188327ccce962c2491d76c492680},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.28491. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Long-term assistants must follow unfinished tasks across days as information and circumstances change. Finding information or starting an action, however, does not ensure that the task gets completed. We introduce OptionWorld, a benchmark that traces information acquisition, judgments, actions, and outcomes in simulated daily-life tasks to identify where current LLMs fall short as long-term personal assistants. In runs with archived decision inputs, the key facts remain in the model's last input before the deadline in 35.5% of failed instances. Failure analysis identifies the three most frequent problems: needed information is not obtained or combined, available opportunities to act go unused, and agents impose restrictions that the task does not require. Three follow-up experiments show that supplying missing facts restores task success in some cases; action reminders can restart work while subsequent steps still miss the deadline; and correcting a supplied note that wrongly forbids contact restores permitted inquiry. These findings locate shortcomings in maintaining task state: connecting facts to unfinished work, following through before deadlines, and separating actual restrictions from the agent's assumptions.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.28491 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-21T07:05:58.156Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).