Metraipolis › archive › dc_eab9baa62dcb
Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation — arXiv:2609.04298
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.04298
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T22:18:23.241Z
- SHA-256 of the archived text
ec70954290efe5b1c483b9bb060453bd22c37cf0fbdea3f0869c8fd69e83145f- Stable URL of this page
https://metraipolis.com/documento/dc_eab9baa62dcb
How to cite
Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation — arXiv:2609.04298 — arXiv (en), https://arxiv.org/abs/2609.04298 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_eab9baa62dcb (Metraipolis archive dc_eab9baa62dcb, sha256 ec70954290efe5b1c483b9bb060453bd22c37cf0fbdea3f0869c8fd69e83145f)
BibTeX
@misc{dceab9baa62dcb,
title = {Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation — arXiv:2609.04298},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.04298},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_eab9baa62dcb; sha256 ec70954290efe5b1c483b9bb060453bd22c37cf0fbdea3f0869c8fd69e83145f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.04298. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Evaluating agents on the growing number of agentic benchmarks is challenging because they often require complex environments and agent integrations. We introduce Harbor Adapters, a unified evaluation infrastructure for agentic benchmarks. Our work makes three contributions. First, we develop benchmark adapters that port more than 80 benchmarks to evaluate arbitrary agents, and validate them through rigorous code review and parity experiments. Second, we conduct a large-scale evaluation of 8 models spanning capability tiers across 54 benchmarks; every model is run with Terminus-2 and with one of 3 native harnesses. This enables a broader analysis of agent capabilities and failure modes than was previously possible. Third, we introduce Harbor-Index, a curated set of 82 difficult, diverse, and high-quality tasks spanning 29 benchmarks, refined from the adapted suite through difficulty filtering, AI and human audit, and an audit-and-fix loop. Harbor-Index preserves the challenge and breadth of large-scale agentic evaluations while being affordable to run; no evaluated model-harness configuration exceeds 30% pass rate, and the strongest (GPT-5.5 with Codex) reaches 28.0%. We release the adapters, evaluation results, in-depth analysis, and Harbor-Index as open-source artifacts to support more reliable and comprehensive evaluation of language-model agents. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-07 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.04298 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T22:18:11.338Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).