Metraipolis › archive › dc_a3e4412cf3fc

ICLHerd: Herding-Based Data Selection for LLMs with In-Context Learning as a Proxy — arXiv:2610.27142

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.27142
License (as archived)
CC0-1.0
Archived
2026-10-21T07:07:28.748Z
SHA-256 of the archived text
1d433e772330ee2128f32415cf876696a9ed527fc3ab6b2f39cc05a78e8992f8
Stable URL of this page
https://metraipolis.com/documento/dc_a3e4412cf3fc

How to cite

ICLHerd: Herding-Based Data Selection for LLMs with In-Context Learning as a Proxy — arXiv:2610.27142 — arXiv (en), https://arxiv.org/abs/2610.27142 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a3e4412cf3fc (Metraipolis archive dc_a3e4412cf3fc, sha256 1d433e772330ee2128f32415cf876696a9ed527fc3ab6b2f39cc05a78e8992f8)
BibTeX
@misc{dca3e4412cf3fc,
  title = {ICLHerd: Herding-Based Data Selection for LLMs with In-Context Learning as a Proxy — arXiv:2610.27142},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.27142},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a3e4412cf3fc; sha256 1d433e772330ee2128f32415cf876696a9ed527fc3ab6b2f39cc05a78e8992f8},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.27142. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Data selection plays a central role in modern large language model training, as the quality of training data strongly influences downstream performance. A common approach is to curate a development set that encodes the desired skills (e.g., reasoning or multilinguality) and select data that are similar to them. Existing methods compute similarity using cosine distance over either the model's hidden states or gradients. However, we observe that this max-similarity strategy induces a mode-seeking bias: the selected data clusters tightly around the development samples. To address this, we propose ICLHerd, a data selection framework that leverages the von Mises-Fisher kernel to balance mode-seeking with mode-averaging. We further introduce in-context learning as a validation proxy, enabling efficient hyperparameter search over the balance trade-off. Results show that ICLHerd outperforms existing LLM data selection algorithms, highlighting the importance of balancing mode-seeking and mode-averaging.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.27142 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-21T07:05:43.528Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).