Metraipolis › archive › dc_8a04757cadbc

GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents — arXiv:2609.12191

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.12191
License (as archived)
CC0-1.0
Archived
2026-10-16T03:08:19.201Z
SHA-256 of the archived text
a6111b13b034983aef59a1f460af4ff46a5af998cb0f54e288050f04175e78aa
Stable URL of this page
https://metraipolis.com/documento/dc_8a04757cadbc

How to cite

GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents — arXiv:2609.12191 — arXiv (en), https://arxiv.org/abs/2609.12191 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8a04757cadbc (Metraipolis archive dc_8a04757cadbc, sha256 a6111b13b034983aef59a1f460af4ff46a5af998cb0f54e288050f04175e78aa)
BibTeX
@misc{dc8a04757cadbc,
  title = {GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents — arXiv:2609.12191},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.12191},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8a04757cadbc; sha256 a6111b13b034983aef59a1f460af4ff46a5af998cb0f54e288050f04175e78aa},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.12191. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Comparing and selecting task-oriented LLM agents increasingly relies on a low-cost offline evaluation gate: persona-driven LLM user-simulators converse with each candidate, an LLM-as-a-judge scores the transcripts, and the higher-scoring agent is promoted. We introduce GAUGE, a reusable offline protocol that measures whether this gate's ranking matches a grounded verifiable reward across 25 agents from six providers on the $\tau^2$-bench and SimulatorArena benchmarks, separating two kinds of evaluation validity that release practices conflate: ranking validity and construct validity. First, a satisfaction-success gap: satisfaction carries essentially no information about task success, as conversations rated satisfied by our blind panel are decorrelated from actual success, with 57.5% of them failing the customer's task, a pattern consistent across five rater populations, both benchmarks, and every subjective dimension we rated. Second, while the gate's ranking is robust across the broad capability span, it loses resolution among the near-equal strong agents: this decision-disagreement rate jumps from $<$1% on wide-reward pairs to 31% on close pairs. The gate is thus human-validated yet mis-anchored. As a remedy, we propose a calibrate-then-trust cadence in which a judge-free completion bit is a zero-cost tripwire for truncation regressions.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-14 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.12191 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T03:02:47.531Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).