Metraipolis › archive › dc_b058dbc446c9

Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models — arXiv:2610.02593

Source
arXiv
Original
https://arxiv.org/abs/2610.02593
License (as archived)
CC0-1.0
Archived
2026-10-05T23:09:40.232Z
SHA-256 of the archived text
88bbdc281e86cf99e2ebdf0578b949ccf2afab9b9b12c46e96d78381095db290
Stable URL of this page
https://metraipolis.com/documento/dc_b058dbc446c9

How to cite

Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models — arXiv:2610.02593 — arXiv, https://arxiv.org/abs/2610.02593 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b058dbc446c9 (Metraipolis archive dc_b058dbc446c9, sha256 88bbdc281e86cf99e2ebdf0578b949ccf2afab9b9b12c46e96d78381095db290)
BibTeX
@misc{dcb058dbc446c9,
  title = {Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models — arXiv:2610.02593},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.02593},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b058dbc446c9; sha256 88bbdc281e86cf99e2ebdf0578b949ccf2afab9b9b12c46e96d78381095db290},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.02593. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Data selection is already a central bottleneck in large-language-model training, where web-scale corpora are noisy and token budgets are finite. In continual pre-training (CPT), it becomes a forgetting-control problem: a poorly chosen target-domain corpus can overwrite capabilities encoded in the pretrained checkpoint. Existing CPT practice either scores candidates with parameter-agnostic scalars such as perplexity, or mitigates forgetting by spending many extra general-domain replay tokens. Neither strategy directly asks how training on a candidate will move the model parameters. We show that loss-based selection causes the post-CPT Fisher diagonal to drift downward on exactly the high-Fisher coordinates the pretrained model had committed to, while leaving low-Fisher coordinates largely untouched. This asymmetry exposes a parameter-space mechanism for catastrophic forgetting. Motivated by this observation, we propose a Fisher-aware CPT selector that decomposes each candidate's gradient into an anchor component, which measures perturbation along committed parameter directions, and a frontier component, which measures update capacity in unconstrained low-Fisher subspaces. We aggregate these signals with a log-determinant submodular objective and optimize it in a single pass using a scalable streaming data selection pipeline. On TinyLlama-1.1B and Llama-3.1-8B CPT over medical data, our selector improves target-domain quality while bounding forgetting on held-out pretraining benchmarks. Most importantly, it is substantially more token-efficient than forgetting-aware replay. 1B selected tokens already outperform the replay strategy trained with 10B tokens on both adaptation and forgetting, giving a 10x token-efficiency advantage.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.02593 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-05T23:01:38.605Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).