Metraipolis › archive › dc_cb7b814dcdb9
Item Response Theory for AI Safety — arXiv:2608.05086
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.05086
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T11:13:13.756Z
- SHA-256 of the archived text
1dd3496e7b4bf737a67b8a544598508636cd98373f07166b854e94f25e992af5- Stable URL of this page
https://metraipolis.com/documento/dc_cb7b814dcdb9
How to cite
Item Response Theory for AI Safety — arXiv:2608.05086 — arXiv (en), https://arxiv.org/abs/2608.05086 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cb7b814dcdb9 (Metraipolis archive dc_cb7b814dcdb9, sha256 1dd3496e7b4bf737a67b8a544598508636cd98373f07166b854e94f25e992af5)
BibTeX
@misc{dccb7b814dcdb9,
title = {Item Response Theory for AI Safety — arXiv:2608.05086},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.05086},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cb7b814dcdb9; sha256 1dd3496e7b4bf737a67b8a544598508636cd98373f07166b854e94f25e992af5},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.05086. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Language models differ in how safely they behave and these differences are measured by safety benchmarks. But aggregated benchmark scores are hard to trust and interpret, because benchmarks duplicate one another, correlate heavily, and models may sandbag when they detect evaluation. To address these issues, we draw on Item Response Theory (IRT), a statistical toolkit for measuring these latents from performance on items with inferred psychometric properties. We fit IRT models to eight safety benchmarks across 192 language models, the largest psychometric analysis of LLM safety evaluations to date, and contribute three results. First, we find that three interpretable factors of refusal strictness, truthfulness, and contextual harm explain most of the variance between models across benchmarks. Second, psychometrically selected items recover full benchmark scores with lower error than random subsets of the same size, and roughly ten adaptively chosen items suffice for several individual benchmarks, cutting evaluation cost by 97-99%. Third, IRT supports audits of individual models, showing that it can be used to detect naive sandbagging and changes of model behind APIs. Overall, we show IRT is a ready-made toolkit for reading, reducing, and auditing safety benchmarks, which we recommend frontier labs and evaluators adopt. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-06 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2608.05086 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T11:02:13.439Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-06?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).