Metraipolis › archive › dc_23cf905ec04f
Known Good, Known Bad: A Ground-Truth Calibration Protocol for AI Judgment in Science — arXiv:2611.11673
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.11673
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-10T18:46:54.121Z
- SHA-256 of the archived text
6462518d19bb811353641b84213017bf931ce260c505bf7ce3f4a827dc288a02- Stable URL of this page
https://metraipolis.com/documento/dc_23cf905ec04f
How to cite
Known Good, Known Bad: A Ground-Truth Calibration Protocol for AI Judgment in Science — arXiv:2611.11673 — arXiv (en), https://arxiv.org/abs/2611.11673 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_23cf905ec04f (Metraipolis archive dc_23cf905ec04f, sha256 6462518d19bb811353641b84213017bf931ce260c505bf7ce3f4a827dc288a02)
BibTeX
@misc{dc23cf905ec04f,
title = {Known Good, Known Bad: A Ground-Truth Calibration Protocol for AI Judgment in Science — arXiv:2611.11673},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.11673},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_23cf905ec04f; sha256 6462518d19bb811353641b84213017bf931ce260c505bf7ce3f4a827dc288a02},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.11673. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Scientific use of generative AI requires evaluation against cases with independently checkable answers. We present a ground truth calibration protocol that compares a verified reference case with reference deviant variants through neutral evaluation, an authority framed recheck, and a line by line audit. Human adjudication distinguishes deviation detection, premise accommodation, unsupported doubt, and correctness of the resulting calculation. An instructor generated organic chemistry demonstration comprises six conversations and 18 responses from an unidentified backend served through the free logged out ChatGPT interface in July 2026. Neutral evaluation detected three of five intended deviations. Both initial misses, involving oxidant stoichiometry and experimental scale, were detected at the authority framed second turn; all five detections persisted at the third turn audit. The fixed sequence supports a descriptive account of recovery across turns. In the scale case, the model initially produced a locally coherent peroxide limited yield. After recognizing the unit problem, it incorrectly retained that limiting reagent conclusion for the corrected scale. Detection and numerical repair therefore require separate assessment. The verified reference attracted speculative concerns while retaining its correct yield calculation. The protocol connects documentary verification of scientific claims and citations with empirical verification of laboratory interpretations. It supplies inspectable transcripts, reference calculations, behavior codes, and a reusable calibration architecture for AI literacy instruction. The pilot demonstrates the protocol's measures in one case family; replicated model evaluation and learner assessment are the next validation steps. Tema degli esploratori: governance digitale e beni comuni (elenco arXiv cs.CY, nuovi lavori del giorno). Categorie arXiv: Computers and Society (cs.CY). Pagina del lavoro: https://arxiv.org/abs/2611.11673 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-10T18:38:52.431Z dalla pagina https://arxiv.org/list/cs.CY/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).