Metraipolis › archive › dc_e09330775608

On Hyperparameter Tuning on the Test Set — arXiv:2610.05902

Source
arXiv
Original
https://arxiv.org/abs/2610.05902
License (as archived)
CC0-1.0
Archived
2026-10-06T05:27:15.752Z
SHA-256 of the archived text
9aea66ce6a85166814d25f61350a85273e31424ac942f50b5640fe89a4984a11
Stable URL of this page
https://metraipolis.com/documento/dc_e09330775608

How to cite

On Hyperparameter Tuning on the Test Set — arXiv:2610.05902 — arXiv, https://arxiv.org/abs/2610.05902 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e09330775608 (Metraipolis archive dc_e09330775608, sha256 9aea66ce6a85166814d25f61350a85273e31424ac942f50b5640fe89a4984a11)
BibTeX
@misc{dce09330775608,
  title = {On Hyperparameter Tuning on the Test Set — arXiv:2610.05902},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.05902},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e09330775608; sha256 9aea66ce6a85166814d25f61350a85273e31424ac942f50b5640fe89a4984a11},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05902. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

"Don't tune hyperparameters on the test set" is often stated in machine learning textbooks. Violating it is considered a cardinal sin that produces misleadingly optimistic results, corrupts benchmark integrity, and thus can even be interpreted as scientific fraud. Yet evidence suggests that test set hyperparameter tuning does occur in practice, making it all the more important to understand its actual consequences. So how bad is it, really? In this work we question this dogma and put it to an empirical test. We systematically study the magnitude of the performance inflation caused by tuning the hyperparameters on the test set for MNIST-1D, CIFAR-10, and three tasks from the GLUE benchmark. Our experiments show that while the effect is real and significant, it is frequently small relative to other sources of noise. In many cases, we find that tuning on the test set recovers exactly the same model as when tuning on the validation set. Most importantly, we find that the rankings of models remain essentially preserved after tuning on the test set and therefore that consistent test-set tuning may not invalidate benchmarks or model selection. Our results call for a more nuanced view of tuning hyperparameters on the test set, stimulating researchers to openly report test tuning.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2610.05902 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:06.087Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).