Metraipolis › archive › dc_9cfbd2d94c7a
How Reliable Are Psychological Measurements? The Distribution of Marginal Reliability Across 889 Item-Response Datasets — arXiv:2608.06806
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.06806
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T11:05:29.369Z
- SHA-256 of the archived text
ddace61436f52382852f1a7e116705df39dafc794305db23f03dd56b3dbbbbff- Stable URL of this page
https://metraipolis.com/documento/dc_9cfbd2d94c7a
How to cite
How Reliable Are Psychological Measurements? The Distribution of Marginal Reliability Across 889 Item-Response Datasets — arXiv:2608.06806 — arXiv (en), https://arxiv.org/abs/2608.06806 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9cfbd2d94c7a (Metraipolis archive dc_9cfbd2d94c7a, sha256 ddace61436f52382852f1a7e116705df39dafc794305db23f03dd56b3dbbbbff)
BibTeX
@misc{dc9cfbd2d94c7a,
title = {How Reliable Are Psychological Measurements? The Distribution of Marginal Reliability Across 889 Item-Response Datasets — arXiv:2608.06806},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.06806},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9cfbd2d94c7a; sha256 ddace61436f52382852f1a7e116705df39dafc794305db23f03dd56b3dbbbbff},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.06806. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Nearly every quantitative study in psychology reports a reliability coefficient, so the field knows a great deal about the reliability that authors choose to publish. It knows much less about the reliability of the data psychology actually produces, because published coefficients pass through decisions about what to compute and what to report. We therefore measure reliability directly, applying the same estimators under the same rules to 889 datasets from the Item Response Warehouse, a public collection of item-response data that spans cognitive tests, clinical screeners, personality inventories, and attitude scales. Three findings emerge. Low reliability is common: even under a lenient definition of reliability, 30% of datasets fall below the conventional .80 threshold. The variation across datasets is real rather than statistical, since estimation noise accounts for only about one percent of it. Finally, the answer depends on the definition itself: under a strict definition the share below .80 rises to 52%, a difference large enough to change what one concludes about the field. We conclude that a reliability report should say which definition it uses, attach a measure of uncertainty, and give a strict coefficient alongside a lenient one. Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-08-10 in stat.AP). Categorie arXiv: Applications (stat.AP). Pagina del lavoro: https://arxiv.org/abs/2608.06806 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T11:03:24.949Z dalla pagina https://arxiv.org/catchup/stat.AP/2026-08-10?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).