Metraipolis › archive › dc_83a68653ea78

Equivalence testing and run-count design for stochastic machine-learning experiments — arXiv:2610.23700

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.23700
License (as archived)
CC0-1.0
Archived
2026-10-19T09:10:25.091Z
SHA-256 of the archived text
6e97f04a61dd94ca42cc08654a8c52c0957ad5703050d3a4c509476d5a9f061a
Stable URL of this page
https://metraipolis.com/documento/dc_83a68653ea78

How to cite

Equivalence testing and run-count design for stochastic machine-learning experiments — arXiv:2610.23700 — arXiv (en), https://arxiv.org/abs/2610.23700 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_83a68653ea78 (Metraipolis archive dc_83a68653ea78, sha256 6e97f04a61dd94ca42cc08654a8c52c0957ad5703050d3a4c509476d5a9f061a)
BibTeX
@misc{dc83a68653ea78,
  title = {Equivalence testing and run-count design for stochastic machine-learning experiments — arXiv:2610.23700},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.23700},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_83a68653ea78; sha256 6e97f04a61dd94ca42cc08654a8c52c0957ad5703050d3a4c509476d5a9f061a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.23700. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Claims that two training methods perform equally well often rest on a non-significant difference between two or three runs per method. We develop a protocol, with its power theory, for certifying practical equivalence in per-run metrics such as the AUROC (area under the receiver operating characteristic curve). It rests on a two-level (seed x item) variance model separating between-run dispersion from item-level noise. Equivalence within a pre-specified margin is tested by two one-sided tests (TOST), equivalently by a confidence interval inside the margin; a two-level bootstrap covers composite statistics. A closed-form formula converts a margin and a target power into runs per method; its shifted-t refinement is calibrated against the realised two-level process. Simulations on a 270-cell grid and under non-Gaussian seed laws confirm the formula within its design-relevant range, from five runs per method. For four published runs of a joint energy-based model, the failure to certify equivalence was largely decided by design: with the variance calibrated to those runs, two per method certify with probability near three per cent, whereas certifying 0.01 AUROC at 80 per cent power needs twelve even under exact equality. A second case study on a thousand-seed CIFAR-10 zoo and the MultiBERTs checks the formula against real seed laws (within 0.05 in that range) and resolves comparisons that two or three runs leave undecided. In a two-stage design (pilot, then formula), our experiments suggest that an upper confidence bound on the pilot variance substantially reduces the undersizing seen with its point estimate.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2610.23700 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T09:06:29.566Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).