Metraipolis › archive › dc_b058dbc446c9
Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models — arXiv:2610.02593
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.02593
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-05T23:09:40.232Z
- SHA-256 of the archived text
88bbdc281e86cf99e2ebdf0578b949ccf2afab9b9b12c46e96d78381095db290- Stable URL of this page
https://metraipolis.com/documento/dc_b058dbc446c9
How to cite
Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models — arXiv:2610.02593 — arXiv, https://arxiv.org/abs/2610.02593 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b058dbc446c9 (Metraipolis archive dc_b058dbc446c9, sha256 88bbdc281e86cf99e2ebdf0578b949ccf2afab9b9b12c46e96d78381095db290)
BibTeX
@misc{dcb058dbc446c9,
title = {Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models — arXiv:2610.02593},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.02593},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b058dbc446c9; sha256 88bbdc281e86cf99e2ebdf0578b949ccf2afab9b9b12c46e96d78381095db290},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.02593. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Data selection is already a central bottleneck in large-language-model training, where web-scale corpora are noisy and token budgets are finite. In continual pre-training (CPT), it becomes a forgetting-control problem: a poorly chosen target-domain corpus can overwrite capabilities encoded in the pretrained checkpoint. Existing CPT practice either scores candidates with parameter-agnostic scalars such as perplexity, or mitigates forgetting by spending many extra general-domain replay tokens. Neither strategy directly asks how training on a candidate will move the model parameters. We show that loss-based selection causes the post-CPT Fisher diagonal to drift downward on exactly the high-Fisher coordinates the pretrained model had committed to, while leaving low-Fisher coordinates largely untouched. This asymmetry exposes a parameter-space mechanism for catastrophic forgetting. Motivated by this observation, we propose a Fisher-aware CPT selector that decomposes each candidate's gradient into an anchor component, which measures perturbation along committed parameter directions, and a frontier component, which measures update capacity in unconstrained low-Fisher subspaces. We aggregate these signals with a log-determinant submodular objective and optimize it in a single pass using a scalable streaming data selection pipeline. On TinyLlama-1.1B and Llama-3.1-8B CPT over medical data, our selector improves target-domain quality while bounding forgetting on held-out pretraining benchmarks. Most importantly, it is substantially more token-efficient than forgetting-aware replay. 1B selected tokens already outperform the replay strategy trained with 10B tokens on both adaptation and forgetting, giving a 10x token-efficiency advantage. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.02593 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-05T23:01:38.605Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).