Metraipolis › archive › dc_42b33ef7dee3
Different Corpora, Different Skills? A Paired Study of Small Language Models — arXiv:2611.07812
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.07812
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-06T14:43:10.565Z
- SHA-256 of the archived text
90bdc81a760e43b10327477fe912359314025290bd4c8e244da0db47932a85a5- Stable URL of this page
https://metraipolis.com/documento/dc_42b33ef7dee3
How to cite
Different Corpora, Different Skills? A Paired Study of Small Language Models — arXiv:2611.07812 — arXiv (en), https://arxiv.org/abs/2611.07812 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_42b33ef7dee3 (Metraipolis archive dc_42b33ef7dee3, sha256 90bdc81a760e43b10327477fe912359314025290bd4c8e244da0db47932a85a5)
BibTeX
@misc{dc42b33ef7dee3,
title = {Different Corpora, Different Skills? A Paired Study of Small Language Models — arXiv:2611.07812},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.07812},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_42b33ef7dee3; sha256 90bdc81a760e43b10327477fe912359314025290bd4c8e244da0db47932a85a5},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.07812. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Small language models are often trained under tight data budgets, where the choice of pretraining corpus may matter as much as model size or training recipe. We compare two contrasting corpora at matched 10-million- and 37-million-word budgets: the 1911 Encyclopaedia Britannica and the BabyLM 2026 strict pretraining corpus. For each corpus, we train teacher models, distilled 58-million-parameter Baby-Llama students, and no-distillation student ablations, using the same model families, training recipes, and evaluation protocol across corpora. Across BLiMP, BLiMP Supplement, EWoK, Entity Tracking, and COMPS, the corpora produce different skill profiles rather than a single quality ranking. BabyLM 2026 is consistently stronger on linguistic acceptability benchmarks, while Britannica is often stronger on Entity Tracking, a synthetic state-tracking task. The contrast appears across teacher and student models and at both budgets, with the cleanest cross-seed replication in the distilled student comparison. Targeted controls show that the central contrast persists after matching training exposure and at a longer diagnostic context; direct teacher ensembling also does not reproduce the 10M Britannica student's Entity Tracking behavior. Entity Tracking magnitudes nevertheless remain sensitive to training dynamics. For this corpus pair, corpus choice can therefore produce different relative skill profiles at small scale, motivating multi-benchmark evaluation when comparing pretraining data. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2611.07812 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-06T14:18:57.588Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).