Metraipolis › archive › dc_4fbcc30ab91d
A Systematic Study of Small Language Models on Abstract Reasoning Tasks — arXiv:2610.08680
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.08680
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T11:23:50.319Z
- SHA-256 of the archived text
dc1e6ad707fc09b1faaaf106b700a52b35fb27e170bd7f95d43bcb886b2a96e8- Stable URL of this page
https://metraipolis.com/documento/dc_4fbcc30ab91d
How to cite
A Systematic Study of Small Language Models on Abstract Reasoning Tasks — arXiv:2610.08680 — arXiv (en), https://arxiv.org/abs/2610.08680 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4fbcc30ab91d (Metraipolis archive dc_4fbcc30ab91d, sha256 dc1e6ad707fc09b1faaaf106b700a52b35fb27e170bd7f95d43bcb886b2a96e8)
BibTeX
@misc{dc4fbcc30ab91d,
title = {A Systematic Study of Small Language Models on Abstract Reasoning Tasks — arXiv:2610.08680},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.08680},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4fbcc30ab91d; sha256 dc1e6ad707fc09b1faaaf106b700a52b35fb27e170bd7f95d43bcb886b2a96e8},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.08680. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Endpoint accuracy on abstract-reasoning benchmarks does not reveal whether a language model has acquired a transferable rule or fit distribution-specific regularities. We study this distinction in small language models on the ARC-TGI benchmark, which organizes abstract grid transformations into controllable task families and supports resampling, spatial shifts, and cross-benchmark transfer. Across more than 1,000 runs, we profile decoder-only, encoder--decoder, and mixture-of-experts model families under supervised fine-tuning. We examine the efficiency and stability of skill acquisition, robustness beyond the training distribution, interactions with model family and task formulation, and layer-wise attention signatures that accompany behavioral differences. Substantial in-distribution accuracy is attainable, but acquisition is sensitive to optimization and unevenly distributed across task families. Performance deteriorates sharply outside the training distribution, including when the rule is retained but grid scale changes. Greater training-set depth and breadth yield uneven gains, while the effect of additional in-context examples depends on model family. Executable-rule induction also yields correct solutions not observed under direct grid generation. On selected tasks, attention diagnostics show distinct concentration and context-dependence profiles, but do not establish general causal mechanisms. Overall, abstract-reasoning scores are conditional on the model, adaptation regime, evaluation distribution, and response format. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.08680 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T11:15:39.502Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).