Metraipolis › archive › dc_08d37704ab3d
When do data mixtures improve scaling laws? Insights from high-dimensional regression — arXiv:2609.38011
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.38011
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T17:31:59.033Z
- SHA-256 of the archived text
bbbf09f1daa01193acbe388a9a9dc54b8821c881469cd144f6ff58e55d1c492a- Stable URL of this page
https://metraipolis.com/documento/dc_08d37704ab3d
How to cite
When do data mixtures improve scaling laws? Insights from high-dimensional regression — arXiv:2609.38011 — arXiv (en), https://arxiv.org/abs/2609.38011 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_08d37704ab3d (Metraipolis archive dc_08d37704ab3d, sha256 bbbf09f1daa01193acbe388a9a9dc54b8821c881469cd144f6ff58e55d1c492a)
BibTeX
@misc{dc08d37704ab3d,
title = {When do data mixtures improve scaling laws? Insights from high-dimensional regression — arXiv:2609.38011},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.38011},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_08d37704ab3d; sha256 bbbf09f1daa01193acbe388a9a9dc54b8821c881469cd144f6ff58e55d1c492a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38011. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Modern machine learning systems are trained on mixtures of data from different domains, and choosing the right mixture can substantially improve downstream performance. Despite an extensive literature on data mixing and reweighting, existing work is largely empirical and it remains unclear when auxiliary data genuinely improves scaling laws rather than merely providing more samples. To gain insight into this question, we study a high-dimensional mixed-data regression model with a shared regression function, heterogeneous covariances and noise levels, and dataset sizes that may grow at different rates. We establish the minimax risk under an ellipsoidal parameter constraint for the general covariance structure and derive deterministic equivalents for the test error of ridge regression under commutative covariances. We then specialize to a target domain and an auxiliary domain with aligned power-law covariance spectra, where the theory yields explicit scaling laws in terms of spectral decay, target regularity, and the relative growth of the two datasets. These laws identify regimes in which combining data mixtures provably yields a faster scaling rate than using either dataset alone. In particular, improving the scaling law requires a specific interplay between spectra and relative sample sizes of the domains. Our numerical experiments on language models exhibit the same qualitative phenomenon: appropriate data mixtures yield a faster decrease in target-domain test loss than training on either domain alone. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.LG). Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML). Pagina del lavoro: https://arxiv.org/abs/2609.38011 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T16:59:48.408Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).