Metraipolis › archive › dc_8bed740eeedd

TRACE Bench: Task-driven Roleplay Agentic Checklist Evaluation — arXiv:2608.11236

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.11236
License (as archived)
CC0-1.0
Archived
2026-10-19T05:15:38.476Z
SHA-256 of the archived text
c2584f34a861e89f5e9babadebf0e2e1668b36fe4351e5f0532a9e39a44216bf
Stable URL of this page
https://metraipolis.com/documento/dc_8bed740eeedd

How to cite

TRACE Bench: Task-driven Roleplay Agentic Checklist Evaluation — arXiv:2608.11236 — arXiv (en), https://arxiv.org/abs/2608.11236 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8bed740eeedd (Metraipolis archive dc_8bed740eeedd, sha256 c2584f34a861e89f5e9babadebf0e2e1668b36fe4351e5f0532a9e39a44216bf)
BibTeX
@misc{dc8bed740eeedd,
  title = {TRACE Bench: Task-driven Roleplay Agentic Checklist Evaluation — arXiv:2608.11236},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.11236},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8bed740eeedd; sha256 c2584f34a861e89f5e9babadebf0e2e1668b36fe4351e5f0532a9e39a44216bf},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.11236. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Roleplay evaluation should do more than assign a single score: it should reveal which role requirements were tested, which failed, and which dialogue evidence supports the judgment. We propose TRACE Bench, a task-driven agentic checklist evaluation framework. It decomposes each role profile offline into a fixed checklist, then uses a User Agent to converse naturally with the target roleplay model while privately updating checklist states from model responses. Scores therefore trace back to checklist items and supporting dialogue turns rather than a black-box holistic impression. For coverage cross-validation, we audit released M2 free-dialogue transcripts from the MiniMax Role-play Benchmark against the same role-derived checklist. The released free-chat transcripts cover only 73.74% of key role-profile points, whereas TRACE Bench reaches 99.91% coverage in fewer turns. Robustness experiments show stable rankings under repeated runs and User Agent replacement. Across 26 models, TRACE Bench reports overall rankings together with capability breakdowns and checklist traces. It also supports Closed-Loop Benchmark Evolution, distilling verification methods proven effective in failed traces so later evaluations can more reliably elicit and examine observed failure modes.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-13 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.11236 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T05:08:46.663Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-13?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).