Metraipolis › archive › dc_90d540a5288e
EngramBench: A Capability-Grounded Benchmark for Skill-Evolution Harnesses — arXiv:2609.39284
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.39284
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T12:45:09.336Z
- SHA-256 of the archived text
00aef870098d699b3d39dd8679e86489376fe57ec12e4a2b198ffae2a3e6a5dc- Stable URL of this page
https://metraipolis.com/documento/dc_90d540a5288e
How to cite
EngramBench: A Capability-Grounded Benchmark for Skill-Evolution Harnesses — arXiv:2609.39284 — arXiv (en), https://arxiv.org/abs/2609.39284 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_90d540a5288e (Metraipolis archive dc_90d540a5288e, sha256 00aef870098d699b3d39dd8679e86489376fe57ec12e4a2b198ffae2a3e6a5dc)
BibTeX
@misc{dc90d540a5288e,
title = {EngramBench: A Capability-Grounded Benchmark for Skill-Evolution Harnesses — arXiv:2609.39284},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.39284},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_90d540a5288e; sha256 00aef870098d699b3d39dd8679e86489376fe57ec12e4a2b198ffae2a3e6a5dc},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.39284. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
While large language models have achieved remarkable success in isolated code generation, authentic software engineering requires sustained reasoning, complex state management, and continuous cross-domain abstraction. However, current evaluations of skill evolution in autonomous agents suffer from a critical identifiability problem: they structurally confound genuine capability abstraction with rote solution leakage (i.e., copying highly similar code from historical training data). To resolve this, we introduce EngramBench, a rigorous, capability-grounded benchmark governed by the strict axiom of capability overlap without solution overlap. Comprising 30 diverse learning tasks and 13 unseen transfer tasks, EngramBench challenges agents to navigate interactive, multi-hour development cycles driven by LLM-simulated users. Our extensive evaluation across 48 multi-hour execution trajectories -- corroborated by human-expert validation -- reveals a profound insight into procedural memory. We demonstrate that static skill banks do not magically bypass the "last mile" of exact code implementation, which remains bottlenecked by the base model's inherent reasoning limits. However, they serve as an indispensable execution compass. By navigating agents away from catastrophic, token-heavy trial-and-error, genuine capability abstraction slashes redundant context bloat and reduces overall coding time by over 55%. Ultimately, EngramBench shifts the evaluation paradigm from trivial pattern matching to the verifiable measurement of deep, cross-domain capability transfer. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI). Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.39284 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T12:14:14.045Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).