Metraipolis › archive › dc_0fab770be489
StreamDecisionBench: Evaluating Decisions in Force on Evolving Language Streams — arXiv:2609.38612
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.38612
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T13:13:26.780Z
- SHA-256 of the archived text
3bfcd437df623f2590dbf4608de9029b2bea0b6e7f7e74d276381a8621c274d3- Stable URL of this page
https://metraipolis.com/documento/dc_0fab770be489
How to cite
StreamDecisionBench: Evaluating Decisions in Force on Evolving Language Streams — arXiv:2609.38612 — arXiv (en), https://arxiv.org/abs/2609.38612 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0fab770be489 (Metraipolis archive dc_0fab770be489, sha256 3bfcd437df623f2590dbf4608de9029b2bea0b6e7f7e74d276381a8621c274d3)
BibTeX
@misc{dc0fab770be489,
title = {StreamDecisionBench: Evaluating Decisions in Force on Evolving Language Streams — arXiv:2609.38612},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.38612},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0fab770be489; sha256 3bfcd437df623f2590dbf4608de9029b2bea0b6e7f7e74d276381a8621c274d3},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38612. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Language models increasingly make real-time decisions in applications that apply the latest answer until a newer one arrives. A late answer can prolong an outdated decision, such as a call recorder still running while a customer reads out card details, an error offline accuracy misses. We make three contributions. First, we release StreamDecisionBench (SDB), a dataset of eight streaming scenarios in four application families, with executable reference decisions derived from public rules. Second, we propose an evaluation protocol and a metric, in-force accuracy: the share of time the applied decision is correct across update intervals of 0.5-8 s. It reflects accuracy and latency jointly, attributing each error to judgment, latency or both. Third, we evaluate thirteen single-model settings, and this attribution separates speed-limited from judgment-limited models: slower, more accurate models lose 42-51% of the time to outdated answers, a fast model 34% to wrong ones. We therefore test hybrids in which a slow model corrects a fast one; with the right pairing and configuration, a hybrid outperforms every single model. However, even the best evaluated system keeps a correct decision in force only about two-thirds of the time, leaving a substantial gap for real-time use. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.38612 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T12:44:51.741Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).