Metraipolis › archive › dc_54be77d61866

Which and When to Admit: Gradient Admission for Data-Centric Small Language Model Finetuning — arXiv:2610.07553

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.07553
License (as archived)
CC0-1.0
Archived
2026-10-07T11:20:47.926Z
SHA-256 of the archived text
97facafee9f6213c82ddcd5ec7c08715b3fb444e838cd399b169ba66a841e4a0
Stable URL of this page
https://metraipolis.com/documento/dc_54be77d61866

How to cite

Which and When to Admit: Gradient Admission for Data-Centric Small Language Model Finetuning — arXiv:2610.07553 — arXiv (en), https://arxiv.org/abs/2610.07553 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_54be77d61866 (Metraipolis archive dc_54be77d61866, sha256 97facafee9f6213c82ddcd5ec7c08715b3fb444e838cd399b169ba66a841e4a0)
BibTeX
@misc{dc54be77d61866,
  title = {Which and When to Admit: Gradient Admission for Data-Centric Small Language Model Finetuning — arXiv:2610.07553},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.07553},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_54be77d61866; sha256 97facafee9f6213c82ddcd5ec7c08715b3fb444e838cd399b169ba66a841e4a0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.07553. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LoRA fine-tuning adapts small language models (SLMs) to heterogeneous instruction data within a low-rank update subspace, making it vulnerable to three structural problems: conflicting gradients that cancel, static data selection that cannot track evolving learning dynamics, and subspace saturation that causes later updates to overwrite useful directions. We argue that effective adaptation therefore requires controlling which data-induced gradients enter the LoRA subspace and when. We propose GRADE (GRadient-Aligned Data-centric rEcipe), a data-centric framework combining two mechanisms: a state-aware selector that continually admits samples aligned with the evolving multi-task gradient field, and a self-calibrating step-level gate that rejects updates likely to cause destructive overwrite near saturation. Across three current-generation backbones and a heterogeneous seven-dataset instruction pool, GRADE outperforms strong data-selection and PEFT-stabilization baselines in accuracy and robustness. It is the only method to improve consistently over standard LoRA on every architecture, while producing more coherent gradient trajectories and less destructive overwrite. These results show that successful SLM adaptation depends not only on which data are selected, but also on which gradients are allowed to enter and persist in the constrained update subspace.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.07553 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T11:15:24.728Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).