Metraipolis › archive › dc_d798ab497cdc

Can Training Logs Make Model Comparisons More Precise? — arXiv:2608.02705

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.02705
License (as archived)
CC0-1.0
Archived
2026-10-31T12:17:42.738Z
SHA-256 of the archived text
8792ded8980cb702ef439388b3a612d2ec1710eefdaa3d3011a7386602ae0a8a
Stable URL of this page
https://metraipolis.com/documento/dc_d798ab497cdc

How to cite

Can Training Logs Make Model Comparisons More Precise? — arXiv:2608.02705 — arXiv (en), https://arxiv.org/abs/2608.02705 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d798ab497cdc (Metraipolis archive dc_d798ab497cdc, sha256 8792ded8980cb702ef439388b3a612d2ec1710eefdaa3d3011a7386602ae0a8a)
BibTeX
@misc{dcd798ab497cdc,
  title = {Can Training Logs Make Model Comparisons More Precise? — arXiv:2608.02705},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.02705},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d798ab497cdc; sha256 8792ded8980cb702ef439388b3a612d2ec1710eefdaa3d3011a7386602ae0a8a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.02705. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Comparing stochastically trained models requires estimating both a performance difference and its uncertainty from repeated runs. We study whether training logs from those same runs can make such comparisons more precise. Because training-log covariates are produced during training rather than measured before it, we use arm-specific covariate adjustment: each model is adjusted only with statistics from its own runs, and the raw mean difference remains the reported effect. In a vision study spanning three architectures and three datasets, simple adjustments based on early training logs often reduce uncertainty in model comparisons. The main limitation is covariate selection. Broadly searching the log pool for the most correlated statistic often adds more noise than it removes, even when useful statistics exist in hindsight. Training logs therefore appear useful for more precise model comparisons, but only when the adjustment avoids large selection noise.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2608.02705 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T12:05:52.942Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).