Metraipolis › archive › dc_8baaf8b01ac5

OptiSelect: How does the Optimizer Shape Data Curriculum? — arXiv:2610.03432

Source: arXiv
Original: https://arxiv.org/abs/2610.03432
License (as archived): CC0-1.0
Archived: 2026-10-05T22:50:39.878Z
SHA-256 of the archived text: 6786f72f0d62c24bab1a85921226b065bc02999f9e8119f4f01afb0d1ea26faa
Stable URL of this page: https://metraipolis.com/documento/dc_8baaf8b01ac5

How to cite

OptiSelect: How does the Optimizer Shape Data Curriculum? — arXiv:2610.03432 — arXiv, https://arxiv.org/abs/2610.03432 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8baaf8b01ac5 (Metraipolis archive dc_8baaf8b01ac5, sha256 6786f72f0d62c24bab1a85921226b065bc02999f9e8119f4f01afb0d1ea26faa)

BibTeX

@misc{dc8baaf8b01ac5,
  title = {OptiSelect: How does the Optimizer Shape Data Curriculum? — arXiv:2610.03432},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.03432},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8baaf8b01ac5; sha256 6786f72f0d62c24bab1a85921226b065bc02999f9e8119f4f01afb0d1ea26faa},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.03432. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Online data selection has demonstrated substantial efficiency gains for LLM pretraining by training on the most valuable candidates within each batch. Since a candidate's value is realized through its effective model update, principled selection should account for the optimizer step, which reshapes the raw gradient before it updates model parameters. We formalize this optimizer-aware selection paradigm as OptiSelect and present the first systematic study of how the optimizer shapes data selection. Our theory establishes a selection gain principle in which the advantage of online selection is governed by the discriminability of the optimizer-induced utility scores. We prove that sign-based and polar-tangential preconditioners of Lion and Muon would suffer from a discriminability collapse which caps attainable gains from OptiSelect, whereas diagonal-adaptive optimizers such as AdamW and Sophia admit strictly better upper bounds. The proposed principle also yields a quantitative derivation of the optimal candidate oversampling ratio. Pretraining experiments on 124M and 720M models are consistent with our theoretical analysis and show that AdamW's diagonal-adaptive scoring geometry remains the strongest scoring geometry even with Muon as optimizer. We further demonstrate that OptiSelect retains its benefits under data rephrasing, a technique used in modern data processing pipelines. Our findings provide theoretical foundations and practical guidance for co-designing optimizers and data selection in LLM pretraining.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.03432 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-05T22:45:55.860Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).