Metraipolis › archive › dc_be97ef2ec730

Data Mixing as Mixture Experiment: Response Surface Methodology and Optimal Design for Large Language Model Pretraining — arXiv:2608.23922

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.23922
License (as archived): CC0-1.0
Archived: 2026-10-18T06:37:46.010Z
SHA-256 of the archived text: f8c3d6e6025cbd12e3691b1f85b742dba49ef7d272a024a96d7f72646ef8e561
Stable URL of this page: https://metraipolis.com/documento/dc_be97ef2ec730

How to cite

Data Mixing as Mixture Experiment: Response Surface Methodology and Optimal Design for Large Language Model Pretraining — arXiv:2608.23922 — arXiv (en), https://arxiv.org/abs/2608.23922 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_be97ef2ec730 (Metraipolis archive dc_be97ef2ec730, sha256 f8c3d6e6025cbd12e3691b1f85b742dba49ef7d272a024a96d7f72646ef8e561)

BibTeX

@misc{dcbe97ef2ec730,
  title = {Data Mixing as Mixture Experiment: Response Surface Methodology and Optimal Design for Large Language Model Pretraining — arXiv:2608.23922},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.23922},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_be97ef2ec730; sha256 f8c3d6e6025cbd12e3691b1f85b742dba49ef7d272a024a96d7f72646ef8e561},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.23922. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Data mixing is a central design problem in large language model pretraining: given a fixed token budget, practitioners must decide how much data to allocate to each domain. Recent proxy-based methods address this problem by training small models on candidate mixtures, fitting a response model, and using the response to select mixtures for larger-scale training. We show that this workflow has the structure of a classical mixture experiment. Under this view, data domains are mixture components, token shares are component proportions, proxy-training runs are experimental design points, and validation loss defines a response surface over the probability simplex. We develop this formulation using sparse second-order Scheffé response-surface models and construct model-robust $\mathcal{I}$-optimal designs for proxy data-mixing experiments. Using RegMix as an empirical case study, we demonstrate how the framework can both interpret observed mixture responses and design more efficient proxy experiments. The Scheffé analysis shows that domain value is strongly relational: several domains that are weak under additive effects become favourable through pairwise interactions, especially through combinations with web-derived text. The sparse Scheffé model preserves mixture rankings across model scales and remains competitive with a flexible machine-learning predictor while providing an explicit decomposition of additive and interaction effects. In a simulation study calibrated to observed proxy-training responses, model-robust $\mathcal{I}$-optimal designs recover the relevant mixture ordering after removing about 25\% of the original proxy runs. These results suggest that LLM data mixing should be treated not only as a prediction problem, but also as an experimental-design problem in which the proxy mixtures themselves can be chosen to improve statistical efficiency.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-08-26 in stat.ML).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2608.23922 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T06:37:24.427Z dalla pagina https://arxiv.org/catchup/stat.ML/2026-08-26?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).