Metraipolis › archive › dc_d3d4bbf7fa5e
PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks — arXiv:2609.27288
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.27288
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T09:11:15.387Z
- SHA-256 of the archived text
981803d34f453d47127980913ac216c40aed5cdda233f49d58bc1ed05939464c- Stable URL of this page
https://metraipolis.com/documento/dc_d3d4bbf7fa5e
How to cite
PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks — arXiv:2609.27288 — arXiv (en), https://arxiv.org/abs/2609.27288 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d3d4bbf7fa5e (Metraipolis archive dc_d3d4bbf7fa5e, sha256 981803d34f453d47127980913ac216c40aed5cdda233f49d58bc1ed05939464c)
BibTeX
@misc{dcd3d4bbf7fa5e,
title = {PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks — arXiv:2609.27288},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.27288},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d3d4bbf7fa5e; sha256 981803d34f453d47127980913ac216c40aed5cdda233f49d58bc1ed05939464c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.27288. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The Abstraction and Reasoning Corpus (ARC) has become a prominent benchmark for evaluating general abstract reasoning and fluid intelligence in AI models. Yet standard ARC evaluation considers only a single capability: producing the correct output grid for a test input. We argue that this narrow format fails to evaluate the diversity of abilities that genuine abstract skill acquisition should enable. We introduce PotARCin, a benchmark that extends ARC by assessing understanding of a task's underlying abstract rule across five dimensions: Definition, Classification, Constrained Generation, Editing, and Inversion. PotARCin employs programmatic methods to generate new task instances and transform given inputs for a given ARC task, enabling dynamic generative sampling beyond fixed input-output pairs. Across five state-of-the-art models evaluated on the ARC-AGI-1 training set, we observe a 25-52 percentage-point performance gap between standard ARC evaluation and evaluation on PotARCin, and find that multi-dimensional evaluation reorders models that standard accuracy ranks alike. We further investigate effects of generative sampling, difficulty of corruption types, and questions of self-consistency, showing that models frequently contradict their own formalized rule even where they have stated it correctly. We also introduce P-ARC, a held-out hand-crafted test set, on which models achieve 1-8% accuracy across all five dimensions, underscoring the importance of more holistic evaluations of abstract reasoning capabilities. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-24 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.27288 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T08:13:26.489Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).