Metraipolis › archive › dc_4cce5d5cd2df
Grounded Checklist Partial Credit for Agent Skill Trajectories — arXiv:2608.27487
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.27487
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T11:14:01.820Z
- SHA-256 of the archived text
39e8c2b540beeb124778bbb96a866f8690a8223c2f9ed57896ac5879cdd25517- Stable URL of this page
https://metraipolis.com/documento/dc_4cce5d5cd2df
How to cite
Grounded Checklist Partial Credit for Agent Skill Trajectories — arXiv:2608.27487 — arXiv (en), https://arxiv.org/abs/2608.27487 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4cce5d5cd2df (Metraipolis archive dc_4cce5d5cd2df, sha256 39e8c2b540beeb124778bbb96a866f8690a8223c2f9ed57896ac5879cdd25517)
BibTeX
@misc{dc4cce5d5cd2df,
title = {Grounded Checklist Partial Credit for Agent Skill Trajectories — arXiv:2608.27487},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.27487},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4cce5d5cd2df; sha256 39e8c2b540beeb124778bbb96a866f8690a8223c2f9ed57896ac5879cdd25517},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27487. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Language-model agents increasingly tackle long-horizon tasks in interactive environments, yet their evaluation commonly relies on task-level success rates by reducing an entire execution trajectory to whether the task passes an official verifier. This binary score hides partial progress and is particularly limited for procedural agent skill evaluations, since a skill can alter execution without changing the final outcome. While checklists provide finer-grained evaluation by scoring individual task requirements, costly manual authoring and unreliable automatic generation make trustworthy evaluation difficult to scale. To address these challenges, we introduce Grounded Checklist Partial Credit (GCPC), a human-governed and LLM-instantiated partial-credit evaluation of agent trajectories. Humans define reusable rules once, from which an LLM instantiates a task-specific checklist grounded in the task instruction and official verifier. To keep judgment tied to evidence, a judge scores each item from execution log evidence alone and abstains when evidence is missing. A separate scripted step then applies the official verifier outcome to the score. Across a 4,455-trajectory, deduplicated SkillsBench evaluation population, GCPC better discriminates official PASS and FAIL outcomes than holistic judging on the shared subset (AUC 0.689 vs. 0.619). Human evaluation on 96 trajectories from 12 tasks shows that GCPC aligns more closely with human assessments of progress. Applied to 1,946 matched with/without-skill pairs, GCPC exposes the effects hidden by pass@1: among 879 pairs whose binary outcome does not change, 20.9% improve by more than 0.10 while 18.7% regress by the same margin. The GCPC pipeline also transfers to Terminal-Bench and SWE-bench, demonstrating applicability beyond skill-conditioned evaluation. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-31 in cs.SE). Categorie arXiv: Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2608.27487 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T10:56:43.366Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-08-31?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).