Metraipolis › archive › dc_44abebdf7073

Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration — arXiv:2609.35342

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.35342
License (as archived)
CC0-1.0
Archived
2026-10-07T20:58:44.600Z
SHA-256 of the archived text
b250b6450cea4254313afab60e9cf52876c1b5a5afc2870f0a0874a30b6a7d01
Stable URL of this page
https://metraipolis.com/documento/dc_44abebdf7073

How to cite

Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration — arXiv:2609.35342 — arXiv (en), https://arxiv.org/abs/2609.35342 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_44abebdf7073 (Metraipolis archive dc_44abebdf7073, sha256 b250b6450cea4254313afab60e9cf52876c1b5a5afc2870f0a0874a30b6a7d01)
BibTeX
@misc{dc44abebdf7073,
  title = {Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration — arXiv:2609.35342},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.35342},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_44abebdf7073; sha256 b250b6450cea4254313afab60e9cf52876c1b5a5afc2870f0a0874a30b6a7d01},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35342. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The appearance of Jev marked the era of System One Models, foundation models that return structured decisions with probability distributions rather than text. Aside from low cost and great speed, Jev's central promise is that these probabilities are calibrated: such claim is not backed by any public test and available external benchmarks evaluate confidence calibration, not whether every returned option probability has the right numerical meaning. To tackle this issue, we introduce Sys1Cal-v1, a dataset of True/False questions about a proposition $A$ for which the exact probability $P(A)$ is known by construction. Each item is queried through the three Jev primitives - Noul, Choice and Score - and evaluated by total variation distance from the ground-truth distribution, which can be used to estimate a soft accuracy of System One Models. We showcase the utility of Sys1Cal-v1 as a benchmark dataset by evaluating Jev and SemIf, an open-source Choice-style baseline. In this work, however, we focus even more deeply on Jev, by studying the calibration of its Score and Choice answers. In particular, we discover a peculiar behaviour that can be explained by assuming that Jev suppresses a third truth value, going beyond True and False. In other words, in \texttt{Choice} answers, $P(A)$ and $P(\neg A)$ are presented as if $P(A)+P(\neg A)=1$, while a term $P(U)\neq0$ is missing in the sum. Recovering $P(U)$ leads to an improvement of median soft accuracy in \texttt{Choice} answers from $0.771$ to $0.978$, suggesting that, even in binary decisions, Jev wants to answer with a third option:``I don't know''.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Logic in Computer Science (cs.LO); Systems and Control (eess.SY).
Pagina del lavoro: https://arxiv.org/abs/2609.35342 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:01.841Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).