Metraipolis › archive › dc_4333664e2e04
Why Ranking Anomaly Detection Algorithms Isn't as Reliable as You May Think — arXiv:2608.04613
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.04613
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T11:34:51.736Z
- SHA-256 of the archived text
27f8300442e56f89f676bb109fa68870ddb81b9ded609028b8aa9628fd9b9239- Stable URL of this page
https://metraipolis.com/documento/dc_4333664e2e04
How to cite
Why Ranking Anomaly Detection Algorithms Isn't as Reliable as You May Think — arXiv:2608.04613 — arXiv (en), https://arxiv.org/abs/2608.04613 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4333664e2e04 (Metraipolis archive dc_4333664e2e04, sha256 27f8300442e56f89f676bb109fa68870ddb81b9ded609028b8aa9628fd9b9239)
BibTeX
@misc{dc4333664e2e04,
title = {Why Ranking Anomaly Detection Algorithms Isn't as Reliable as You May Think — arXiv:2608.04613},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.04613},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4333664e2e04; sha256 27f8300442e56f89f676bb109fa68870ddb81b9ded609028b8aa9628fd9b9239},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.04613. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Anomaly detection is a safety-critical machine learning problem with applications ranging from fraud detection to network intrusion prevention and industrial monitoring. Despite the large number of proposed anomaly detection algorithms, many novel methods claim state-of-the-art performance. However, many authors do so under benchmark settings that are not aligned with one another. This lack of comparability raises concerns regarding the reproducibility and reliability of anomaly detection benchmarks. In this work, we study the impact of common benchmarking choices on the stability of algorithm rankings. Using seven representative anomaly detection algorithms and 690 datasets from the OddBench benchmark suite, we analyze how rankings change under varying dataset selections, evaluation metrics, hyperparameter configurations, and random seeds. To quantify this effect, we introduce a rank instability metric measuring the variability of algorithm rankings across benchmark settings. Our results show that algorithm rankings in anomaly detection are highly unstable. In many cases, almost every competitive algorithm can appear as the best-performing method under some benchmark configuration. Among the studied factors, dataset selection and hyperparameter choice contribute most strongly to ranking uncertainty, while random seeds and evaluation metrics have a comparatively limited impact. We also observe that reliable benchmarking requires substantially larger and more diverse dataset collections than the ones commonly used in prior work. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-06 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.04613 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T11:24:48.787Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-06?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).