Metraipolis › archive › dc_b80c5d3bbb2b

Do System One Decisions Add Up? A Study of Probabilistic Coherence — arXiv:2609.33971

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.33971
License (as archived)
CC0-1.0
Archived
2026-10-07T21:53:14.911Z
SHA-256 of the archived text
c93b9135d90fdf353285796b93fade1c8c7f5b2273b4c6f4c8cbb948dee43954
Stable URL of this page
https://metraipolis.com/documento/dc_b80c5d3bbb2b

How to cite

Do System One Decisions Add Up? A Study of Probabilistic Coherence — arXiv:2609.33971 — arXiv (en), https://arxiv.org/abs/2609.33971 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b80c5d3bbb2b (Metraipolis archive dc_b80c5d3bbb2b, sha256 c93b9135d90fdf353285796b93fade1c8c7f5b2273b4c6f4c8cbb948dee43954)
BibTeX
@misc{dcb80c5d3bbb2b,
  title = {Do System One Decisions Add Up? A Study of Probabilistic Coherence — arXiv:2609.33971},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.33971},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b80c5d3bbb2b; sha256 c93b9135d90fdf353285796b93fade1c8c7f5b2273b4c6f4c8cbb948dee43954},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33971. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A decision model can give probabilities that sum to one for every question yet disagree with itself when the same decision is broken into smaller steps. We study this form of probabilistic coherence in Jev and the English Laya checkpoint, using 2,500 matched examples per system across TREC, CLINC150, and MASSIVE. Across 72,000 classification questions, we compare direct fine-label predictions with broad-category probabilities and predictions reconstructed through those categories. Both systems show substantial disagreement: mean category-level total variation ranges from 0.219 to 0.349 for Jev and from 0.424 to 0.689 for Laya, on a scale where zero means exact agreement. The consequences differ sharply. On CLINC150, reconstruction reduces Jev's accuracy by 22.9 percentage points (paired 95% bootstrap interval: [-24.9, -20.9]) and improves Laya's by 21.3 points ([18.0, 24.5]). The same directions hold across all three datasets, with all six unadjusted accuracy-change intervals excluding zero. Improved accuracy can also accompany less reliable confidence: on MASSIVE, Laya gains 9.2 accuracy points while its expected calibration error rises from 0.046 to 0.124. Error analysis identifies both broad-category mistakes and within-category confusions. These findings show why decision systems need joint evaluation of accuracy, confidence calibration, and probability coherence in the workflow used by an application.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.33971 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T21:27:08.572Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).