Metraipolis › archive › dc_0fab770be489

StreamDecisionBench: Evaluating Decisions in Force on Evolving Language Streams — arXiv:2609.38612

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.38612
License (as archived)
CC0-1.0
Archived
2026-10-07T13:13:26.780Z
SHA-256 of the archived text
3bfcd437df623f2590dbf4608de9029b2bea0b6e7f7e74d276381a8621c274d3
Stable URL of this page
https://metraipolis.com/documento/dc_0fab770be489

How to cite

StreamDecisionBench: Evaluating Decisions in Force on Evolving Language Streams — arXiv:2609.38612 — arXiv (en), https://arxiv.org/abs/2609.38612 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0fab770be489 (Metraipolis archive dc_0fab770be489, sha256 3bfcd437df623f2590dbf4608de9029b2bea0b6e7f7e74d276381a8621c274d3)
BibTeX
@misc{dc0fab770be489,
  title = {StreamDecisionBench: Evaluating Decisions in Force on Evolving Language Streams — arXiv:2609.38612},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.38612},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0fab770be489; sha256 3bfcd437df623f2590dbf4608de9029b2bea0b6e7f7e74d276381a8621c274d3},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38612. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Language models increasingly make real-time decisions in applications that apply the latest answer until a newer one arrives. A late answer can prolong an outdated decision, such as a call recorder still running while a customer reads out card details, an error offline accuracy misses. We make three contributions. First, we release StreamDecisionBench (SDB), a dataset of eight streaming scenarios in four application families, with executable reference decisions derived from public rules. Second, we propose an evaluation protocol and a metric, in-force accuracy: the share of time the applied decision is correct across update intervals of 0.5-8 s. It reflects accuracy and latency jointly, attributing each error to judgment, latency or both. Third, we evaluate thirteen single-model settings, and this attribution separates speed-limited from judgment-limited models: slower, more accurate models lose 42-51% of the time to outdated answers, a fast model 34% to wrong ones. We therefore test hybrids in which a slow model corrects a fast one; with the right pairing and configuration, a hybrid outperforms every single model. However, even the best evaluated system keeps a correct decision in force only about two-thirds of the time, leaving a substantial gap for real-time use.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.38612 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:44:51.741Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).