Metraipolis › archive › dc_b26328b33ce0
Data-Efficient Language Modeling: From Frontier Advancement to Principle-Guided Model Improvement — arXiv:2609.10702
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.10702
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T05:24:36.274Z
- SHA-256 of the archived text
f76c6612ce7cb42301cf53699dd4e20e1c0b60525b95e12ec8433c5f09ad7df3- Stable URL of this page
https://metraipolis.com/documento/dc_b26328b33ce0
How to cite
Data-Efficient Language Modeling: From Frontier Advancement to Principle-Guided Model Improvement — arXiv:2609.10702 — arXiv (en), https://arxiv.org/abs/2609.10702 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b26328b33ce0 (Metraipolis archive dc_b26328b33ce0, sha256 f76c6612ce7cb42301cf53699dd4e20e1c0b60525b95e12ec8433c5f09ad7df3)
BibTeX
@misc{dcb26328b33ce0,
title = {Data-Efficient Language Modeling: From Frontier Advancement to Principle-Guided Model Improvement — arXiv:2609.10702},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.10702},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b26328b33ce0; sha256 f76c6612ce7cb42301cf53699dd4e20e1c0b60525b95e12ec8433c5f09ad7df3},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.10702. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Learning from limited text requires models to use context, generalize to new inputs, and retain useful capabilities. Qiushi Engine conducted a long-horizon, end-to-end autonomous research program on BabyLM 2026 Strict-Small, within 10 million corpus words and 100 million cumulative word presentations. Three stages connected frontier advancement, principle discovery, and principle-guided model improvement. Stage I combined compact restatements, budget reinvestment, and residual incremental learning to build a frontier model. Stage II found that exact repetition and aligned restatement produce different patterns of context use, depending on target relations and prediction windows. In controlled tasks, recovering familiar performance did not ensure that unseen inputs could still use learned computations. These findings support a testable data-efficient learning principle: organize experience around the contextual dependencies needed for prediction; separately design visible information, supervision, and preservation; test learning, generalization, and retention. Stage III retained source text, masked more local clues, supervised selected targets, and preserved predictions on ordinarily masked inputs. Two continuation seeds from the same parent outperformed ordinary continuation on the complete nine-metric aggregate. Overall rose from 42.02 to 42.25 across two generations; the second achieved the highest Overall in the public Strict-Small snapshot of 8 September 2026. Further studies addressed compression, relational anchors, shared representations, and measurement. Models are available on Hugging Face; code and research records accompany the GitHub repository. Together, these stages illustrate Research RSI: recursive self-improvement of the research process. Scientific understanding and method innovations change subsequent questions and designs; new experiments test and refine them. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-11 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.10702 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T05:17:52.068Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).