Metraipolis › archive › dc_cecabfa66590
Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search — arXiv:2609.01431
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.01431
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T06:34:21.373Z
- SHA-256 of the archived text
673d5c74e8dbb02e8b77a8b93b17e26406cddf48d1cbc90db06322ffdb0df694- Stable URL of this page
https://metraipolis.com/documento/dc_cecabfa66590
How to cite
Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search — arXiv:2609.01431 — arXiv (en), https://arxiv.org/abs/2609.01431 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cecabfa66590 (Metraipolis archive dc_cecabfa66590, sha256 673d5c74e8dbb02e8b77a8b93b17e26406cddf48d1cbc90db06322ffdb0df694)
BibTeX
@misc{dccecabfa66590,
title = {Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search — arXiv:2609.01431},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.01431},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cecabfa66590; sha256 673d5c74e8dbb02e8b77a8b93b17e26406cddf48d1cbc90db06322ffdb0df694},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.01431. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Optimal hyperparameter scaling laws describe how the best hyperparameters for large language model (LLM) training change with model and data scale, enabling practitioners to predict optimal configurations at production scales without expensive large-scale tuning. However, estimating these scaling laws conventionally requires exhaustive grid searches over thousands of training runs, consuming enormous computational resources. We introduce Power-Law Entropy Search (PLES), a computational cost-aware acquisition function built on multi-fidelity Bayesian optimization that efficiently estimates optimal hyperparameter scaling laws through adaptive experimentation. A key innovation in PLES is that it searches for candidates that reduce the overall uncertainty of a scaling law estimate, instead of optimizing a single objective function. At each iteration, PLES selects the candidate configuration that maximally reduces the uncertainty of the scaling law estimates per unit computational cost, naturally favoring informative small-scale experiments. We evaluate PLES on synthetic benchmarks, surrogate models fitted to real LLM training data, and actual LLM pre-training runs. Across all settings, PLES converges to accurate optimal hyperparameter scaling laws using less than one-tenth of the computational budget required by conventional grid search and other baselines. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.01431 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T06:15:12.899Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).