Metraipolis › archive › dc_eef2b994ac5e

Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3 — arXiv:2610.30499

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.30499
License (as archived): CC0-1.0
Archived: 2026-10-25T21:33:25.448Z
SHA-256 of the archived text: 3ee7aabc6adc0ad1298612c0f1073ebdcf880e3ffc2c8f2d18103178903f3398
Stable URL of this page: https://metraipolis.com/documento/dc_eef2b994ac5e

How to cite

Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3 — arXiv:2610.30499 — arXiv (en), https://arxiv.org/abs/2610.30499 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_eef2b994ac5e (Metraipolis archive dc_eef2b994ac5e, sha256 3ee7aabc6adc0ad1298612c0f1073ebdcf880e3ffc2c8f2d18103178903f3398)

BibTeX

@misc{dceef2b994ac5e,
  title = {Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3 — arXiv:2610.30499},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.30499},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_eef2b994ac5e; sha256 3ee7aabc6adc0ad1298612c0f1073ebdcf880e3ffc2c8f2d18103178903f3398},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.30499. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A generally intelligent system acquires new skills efficiently without requiring humans to redesign its software scaffold for every new task. Yet language models rarely act alone. A harness is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model's behalf. ARC-AGI-3 is an interactive benchmark in which agents must explore unfamiliar environments, figure out their goals on the fly, and keep learning as they go. On it, human-steered systems already report ceiling scores across the public games, so we know effective designs exist. We ask whether frontier models can build and improve such a harness on their own. Through a fixed coding bootstrap, each model constructs a complete task harness, acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation. We measure task performance with Relative Human Action Efficiency (RHAE), which combines level completion with action efficiency relative to human baselines, and record dollar-denominated inference cost as a practical resource measure. Across 27 lineages, harness performance ranges from zero to near-ceiling scores. Two revisions raise mean held-out RHAE by only 2.80 points (95% interval [-1.44,7.35]). Development gains overstate held-out improvement, and reductions in inference cost can come at the expense of task performance. Human-steered systems demonstrate further headroom. These findings reveal a gap between autonomous harness construction and reliable adaptation under this two-revision protocol: even strong task performance does not ensure that subsequent revisions generalize or preserve existing capabilities.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2610.30499 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-25T21:25:18.087Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).