Metraipolis › archive › dc_2fbb4f16b48a
Model-Aware Data Selection from In-and-Out Information Interplay — arXiv:2609.33010
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.33010
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T20:56:20.891Z
- SHA-256 of the archived text
5fb5261e8a1806bf6a84ed06c386b684e76e86e95f35ebe5ea967cd891fa1268- Stable URL of this page
https://metraipolis.com/documento/dc_2fbb4f16b48a
How to cite
Model-Aware Data Selection from In-and-Out Information Interplay — arXiv:2609.33010 — arXiv (en), https://arxiv.org/abs/2609.33010 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2fbb4f16b48a (Metraipolis archive dc_2fbb4f16b48a, sha256 5fb5261e8a1806bf6a84ed06c386b684e76e86e95f35ebe5ea967cd891fa1268)
BibTeX
@misc{dc2fbb4f16b48a,
title = {Model-Aware Data Selection from In-and-Out Information Interplay — arXiv:2609.33010},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.33010},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2fbb4f16b48a; sha256 5fb5261e8a1806bf6a84ed06c386b684e76e86e95f35ebe5ea967cd891fa1268},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33010. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
LLMs are effective representations that assimilate vast amounts of knowledge during pretraining, but post-training is necessary for models to reliably access this knowledge and "know what they know." We observe an interesting rank equilibrium between knowledge stored in the weights and the data stream passing through the model. Across all model layers, we find that the hidden states (data stream) follow a U-shaped pattern, showing substantial compression in early layers and a steep rise during the late-layer decoding phase. In contrast, the weight rank follows an inverted U-shaped pattern, with very low rank in the early and late layers and high rank in the middle. We interpret this as an in-and-out information interplay: intermediate activations do not need to carry content that the weights can supply later, so they primarily preserve what the weights cannot provide. Motivated by this observation, we propose a model-aware data selection method, CAP (Counterfactual Assimilation Profile), which can determine whether a data candidate contains information accessible to the current model by utilizing the divergence gap in early- and late-layer representations between model-generated and reference responses. Across math, code, and science domains, CAP delivers 35.4% greater average improvement over the base model than the strongest baseline under different selection budgets. With only 10% of the data pool, CAP surpasses or matches full-pool training on math and science. We further show that CAP transfers to multimodal data selection and is robust to response horizon and noise. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.33010 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:01.128Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).