Metraipolis › archive › dc_6dbae897489a
Designing Reliable LLM-as-a-Judge Measurement Systems for Multi-Turn Business Agents — arXiv:2609.33955
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.33955
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T20:57:16.602Z
- SHA-256 of the archived text
97b50c4dcb58a6d4f13338e0a410cb8bd9aef6bd202a7a01b33b9e77d4cb6e3e- Stable URL of this page
https://metraipolis.com/documento/dc_6dbae897489a
How to cite
Designing Reliable LLM-as-a-Judge Measurement Systems for Multi-Turn Business Agents — arXiv:2609.33955 — arXiv (en), https://arxiv.org/abs/2609.33955 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6dbae897489a (Metraipolis archive dc_6dbae897489a, sha256 97b50c4dcb58a6d4f13338e0a410cb8bd9aef6bd202a7a01b33b9e77d4cb6e3e)
BibTeX
@misc{dc6dbae897489a,
title = {Designing Reliable LLM-as-a-Judge Measurement Systems for Multi-Turn Business Agents — arXiv:2609.33955},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.33955},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6dbae897489a; sha256 97b50c4dcb58a6d4f13338e0a410cb8bd9aef6bd202a7a01b33b9e77d4cb6e3e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33955. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Many LLM-as-a-judge evaluations score fixed outputs under a fixed task definition. Production multi-turn business agents instead require a maintained measurement system: correctness depends on business-specific facts and procedures, outcomes emerge across turns, and failures must be attributed to either agent capability or missing business knowledge before they are actionable. We present an integrated methodology spanning evaluation specification, modular LLM judges, intent-preserving user simulation, and human-in-the-loop governance. The specification defines conversation-level end states and actionable failure ownership. Atomic judges share versioned evidence and feed an explicit aggregation graph. The simulator is released only after task-preservation and stability checks. Independent human audits estimate measurement fidelity, renew tiered reference sets, and route disagreements to label correction, guideline revision, or judge improvement. Production studies show that system-level fidelity improved across repeated audits, that human reviewers and automated judges improved together under the shared feedback loop, and that their combined workflow had the strongest descriptive performance in both reported task-completion settings. Because the studies are observational and the human reference itself required revision, these findings demonstrate operational usefulness rather than causal or universal superiority. The contribution is a practical framework for making multi-turn agent measurement reliable, actionable, and maintainable as the evaluated system and its evidence evolve. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.33955 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:01.396Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).