Metraipolis › archive › dc_acbcae2041b7

Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling — arXiv:2609.32484

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.32484
License (as archived)
CC0-1.0
Archived
2026-10-07T20:55:32.808Z
SHA-256 of the archived text
3743aa1bbfd64558ebe3ca4415775757163422b6908ba65a48cbf52571e6c738
Stable URL of this page
https://metraipolis.com/documento/dc_acbcae2041b7

How to cite

Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling — arXiv:2609.32484 — arXiv (en), https://arxiv.org/abs/2609.32484 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_acbcae2041b7 (Metraipolis archive dc_acbcae2041b7, sha256 3743aa1bbfd64558ebe3ca4415775757163422b6908ba65a48cbf52571e6c738)
BibTeX
@misc{dcacbcae2041b7,
  title = {Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling — arXiv:2609.32484},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.32484},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_acbcae2041b7; sha256 3743aa1bbfd64558ebe3ca4415775757163422b6908ba65a48cbf52571e6c738},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32484. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Data selection for language model pretraining faces a fundamental tension between quality and diversity. While quality filtering is empirically effective, it often induces diversity collapse: by favoring texts similar to high-quality reference corpora (e.g., educational or QA-style data), it systematically excludes valuable data from underrepresented domains. In contrast, diversified selection preserves domain balance and encourages robust downstream performance, yet existing methods either focus on coverage-oriented objectives that indirectly enhance diversity, or directly optimize for diversity via costly covariance matrix recomputation that limits scalability. To address these issues, we introduce \textbf{Leverage Score Sampling (Lev)}, which iteratively selects samples that maximally expand the determinantal volume of the embedded data via leverage scores, a computationally efficient criterion that eliminates matrix recomputation and enables scalable selection. Empirically, Lev delivers up to $72\times$ speedup and improves dataset diversity, measured by the Vendi score, by $9.2\%$ over the strong diversification baseline \textbf{DiSF}. On CommonCrawl (CC) web data selection, Lev improves accuracy across seven downstream tasks by up to $1.31\%$ over existing baselines. For domains where robust quality criteria are inherently difficult to define (e.g., code), Lev serves as an effective unsupervised curation alternative: on StarCoderData, the selected subset reduces bits-per-byte by $3.08\%$ over DiSF. Notably, we uncover a cross-domain collapse of quality filtering: CC data filtered by DCLM-fastText fail to retain sufficient code-related content, yielding inferior code performance relative to Lev-selected data. These findings advocate for integrating diversity-aware practices into quality filtering for more effective data curation in language model pretraining.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.32484 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:00.821Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).