Metraipolis › archive › dc_562dc3fa3df5

Four Ledgers, Not One Score: Responsible Communication of LLM-Judge Calibration in Biomedical ML — arXiv:2609.15015

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.15015
License (as archived): CC0-1.0
Archived: 2026-10-15T22:47:45.731Z
SHA-256 of the archived text: a24079e815bf02762fcbb58743c624db2930123382f3e01411fc57ec065a5b1d
Stable URL of this page: https://metraipolis.com/documento/dc_562dc3fa3df5

How to cite

Four Ledgers, Not One Score: Responsible Communication of LLM-Judge Calibration in Biomedical ML — arXiv:2609.15015 — arXiv (en), https://arxiv.org/abs/2609.15015 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_562dc3fa3df5 (Metraipolis archive dc_562dc3fa3df5, sha256 a24079e815bf02762fcbb58743c624db2930123382f3e01411fc57ec065a5b1d)

BibTeX

@misc{dc562dc3fa3df5,
  title = {Four Ledgers, Not One Score: Responsible Communication of LLM-Judge Calibration in Biomedical ML — arXiv:2609.15015},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.15015},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_562dc3fa3df5; sha256 a24079e815bf02762fcbb58743c624db2930123382f3e01411fc57ec065a5b1d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.15015. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Synthetic perturbations appear to offer inexpensive calibration data for LLM evaluators in biomedical ML, where expert review is scarce. Yet a planted mutation key is neither a detector output nor automatically human ground truth. We formalize four distinct ledgers: planted perturbations, independent detector outputs, source-linked human dispositions, and human-added discoveries. We then audit the evaluation design, scoring code, read paths, and current human records of a private synthetic Japanese care-handoff workflow. The factory stored 69 planted error cards across 47 targets. Final review covers 22 targets and contains 22 confirmed imported proposals, 9 rejected proposals, and 79 human-added cards; only 3 reviewed targets are double annotated. Passing imported plant keys to a generic detector scorer yields 22/(22+9)=0.710 and 22/(22+79)=0.218. A direct audit identity shows that these values are proposal-confirmation yield and submitted-ledger composition, not judge precision and recall, because no independent detector realization was preserved for the audited proposals in the available records. The audit also finds source-name collisions, row shadowing, forced severity, vacuous ratio defaults, and unsupported zero-support field weights. We contribute a provenance-aware claim audit, a storage contract, and a minimum calibration gate for responsibly communicating biomedical ML capability claims. This single-workflow forensic case is an existence proof of a failure mode, not an estimate of its prevalence: existing human work supports an exploratory audit of synthetic proposals, but not LLM-judge operating characteristics, clinical validity, corpus prevalence, or robust inter-annotator agreement.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.15015 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T22:29:03.254Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).