Metraipolis › archive › dc_d798ab497cdc
Can Training Logs Make Model Comparisons More Precise? — arXiv:2608.02705
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.02705
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T12:17:42.738Z
- SHA-256 of the archived text
8792ded8980cb702ef439388b3a612d2ec1710eefdaa3d3011a7386602ae0a8a- Stable URL of this page
https://metraipolis.com/documento/dc_d798ab497cdc
How to cite
Can Training Logs Make Model Comparisons More Precise? — arXiv:2608.02705 — arXiv (en), https://arxiv.org/abs/2608.02705 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d798ab497cdc (Metraipolis archive dc_d798ab497cdc, sha256 8792ded8980cb702ef439388b3a612d2ec1710eefdaa3d3011a7386602ae0a8a)
BibTeX
@misc{dcd798ab497cdc,
title = {Can Training Logs Make Model Comparisons More Precise? — arXiv:2608.02705},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.02705},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d798ab497cdc; sha256 8792ded8980cb702ef439388b3a612d2ec1710eefdaa3d3011a7386602ae0a8a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.02705. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Comparing stochastically trained models requires estimating both a performance difference and its uncertainty from repeated runs. We study whether training logs from those same runs can make such comparisons more precise. Because training-log covariates are produced during training rather than measured before it, we use arm-specific covariate adjustment: each model is adjusted only with statistics from its own runs, and the raw mean difference remains the reported effect. In a vision study spanning three architectures and three datasets, simple adjustments based on early training logs often reduce uncertainty in model comparisons. The main limitation is covariate selection. Broadly searching the log pool for the most correlated statistic often adds more noise than it removes, even when useful statistics exist in hindsight. Training logs therefore appear useful for more precise model comparisons, but only when the adjustment avoids large selection noise. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Machine Learning (stat.ML). Pagina del lavoro: https://arxiv.org/abs/2608.02705 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T12:05:52.942Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).