Metraipolis › archive › dc_d791bb17cc29
Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms? — arXiv:2609.33209
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.33209
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:01:09.732Z
- SHA-256 of the archived text
bfb0d43d1502011eda04732d45f79d02967fc41b18b10a207f149ad021dcbdd9- Stable URL of this page
https://metraipolis.com/documento/dc_d791bb17cc29
How to cite
Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms? — arXiv:2609.33209 — arXiv (en), https://arxiv.org/abs/2609.33209 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d791bb17cc29 (Metraipolis archive dc_d791bb17cc29, sha256 bfb0d43d1502011eda04732d45f79d02967fc41b18b10a207f149ad021dcbdd9)
BibTeX
@misc{dcd791bb17cc29,
title = {Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms? — arXiv:2609.33209},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.33209},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d791bb17cc29; sha256 bfb0d43d1502011eda04732d45f79d02967fc41b18b10a207f149ad021dcbdd9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33209. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Typed-decision models such as TypeSafe's Jev answer a declared yes/no or multiple-choice question about a state with a probability instead of text, and their evaluations report accuracy and calibration. Neither requires that the probabilities a model gives to logically related questions fit together item by item, which is what a system that acts on those probabilities needs. We test this property, coherence, with a battery of logically linked questions that needs no labels. For 160 items from ChaosNLI and PubMedQA, each with three mutually exclusive labels, we ask whether the label is X, whether it is not X, whether it is one of the other two labels, and which label applies. On 480 negation pairs, Jev's probabilities for "the label is X" and "the label is not X" miss summing to one by 0.064 on average (95% CI 0.055 to 0.072). Qwen3.8-27B, run from its official BF16 weights, misses by 0.293 with first-token probabilities and by 0.122 with verbalized probabilities. The gap to the first-token readout persists on pairs where both systems give similar probabilities, without double-negation labels, and after averaging Jev's repeated calls. Jev is not coherent either: its violations are about five times its repeat noise, and it over-endorses statements about single labels, so that its three single-label probabilities sum to 1.14 on average. The two systems also fail differently. Qwen3.8-27B's first-token readout under-endorses the complement of a label whether or not the question contains "not", rejecting both a statement and its negation in 196 of 480 pairs, and it does not become more coherent where it is more confident, whereas Jev's violations concentrate where its answer is uncertain. Because the checks need no labels, they expose biases that appear only when question forms are compared, and inconsistencies within items. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.33209 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:02.583Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).