Metraipolis › archive › dc_8e8c19c8f148

Laya as a Typed Probabilistic Assessor: An Independent Reproduction and a Preregistered Study of Calibration and Selective Escalation — arXiv:2609.33843

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.33843
License (as archived): CC0-1.0
Archived: 2026-10-07T20:57:13.105Z
SHA-256 of the archived text: 1ec9d06f7163336166bf47091218fd053fefe697ef0a5b94072f1cb1537af14a
Stable URL of this page: https://metraipolis.com/documento/dc_8e8c19c8f148

How to cite

Laya as a Typed Probabilistic Assessor: An Independent Reproduction and a Preregistered Study of Calibration and Selective Escalation — arXiv:2609.33843 — arXiv (en), https://arxiv.org/abs/2609.33843 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8e8c19c8f148 (Metraipolis archive dc_8e8c19c8f148, sha256 1ec9d06f7163336166bf47091218fd053fefe697ef0a5b94072f1cb1537af14a)

BibTeX

@misc{dc8e8c19c8f148,
  title = {Laya as a Typed Probabilistic Assessor: An Independent Reproduction and a Preregistered Study of Calibration and Selective Escalation — arXiv:2609.33843},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.33843},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8e8c19c8f148; sha256 1ec9d06f7163336166bf47091218fd053fefe697ef0a5b94072f1cb1537af14a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33843. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The shipped Laya Typed-Decisions checkpoint, a 421M-parameter ModernBERT-large assessor that answers typed choice/noul/score questions over workflow state, is uniformly under-confident. The signed confidence-accuracy gap is $-0.214$, every occupied reliability bin's accuracy exceeds its confidence, and that sign uniformity collapses every binned ECE variant to the same value, $0.214$. The card frames the risk as over-confidence; the measured direction is the opposite, and the direction decides which way a confidence-gated cascade fails. A single disjointly fitted temperature ($T=0.469$, sharpening) removes most of the miscalibration (held-out ECE $0.204$ to $0.037$) and outperforms the shipped per-option-count table. The frozen selection rule instead chose isotonic regression, which overfit and failed its held-out NLL contrast on both tracks, so hypothesis H2 is not supported. Re-running the released checkpoint on its full official test split reproduces the card's headline accuracy ($0.767$ vs. $0.766$). The retrospective E1 reproduction preceded the analysis freeze; E2-E8 were prospectively preregistered, and 20 of 22 executed confirmatory tests reject under Benjamini-Hochberg FDR at $q=0.05$ (two descoped). The frozen gate beats random escalation but misses its 10% accepted-set error target on both tracks, an exploratory out-of-distribution probe finds no zero-shot transfer (accuracy $0.617$), and every score measures agreement with a synthetic teacher whose self-agreement ceiling ($0.735$) the specialist exceeds. Per-decision predictions, run manifests, and the frozen preregistration are in the ancillary files. The author has no affiliation with the model's publisher, the dataset's publisher, or TypeSafe.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.33843 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:01.376Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).