Metraipolis › archive › dc_609e0dfd99d2

Calibrating LLM Judges for Human and AI Conversations — arXiv:2609.29431

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.29431
License (as archived): CC0-1.0
Archived: 2026-10-08T05:25:26.999Z
SHA-256 of the archived text: 00b5acb620362f697b134e7b2091f30d10aeae86f7730a875b5d31a2a6751e40
Stable URL of this page: https://metraipolis.com/documento/dc_609e0dfd99d2

How to cite

Calibrating LLM Judges for Human and AI Conversations — arXiv:2609.29431 — arXiv (en), https://arxiv.org/abs/2609.29431 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_609e0dfd99d2 (Metraipolis archive dc_609e0dfd99d2, sha256 00b5acb620362f697b134e7b2091f30d10aeae86f7730a875b5d31a2a6751e40)

BibTeX

@misc{dc609e0dfd99d2,
  title = {Calibrating LLM Judges for Human and AI Conversations — arXiv:2609.29431},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.29431},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_609e0dfd99d2; sha256 00b5acb620362f697b134e7b2091f30d10aeae86f7730a875b5d31a2a6751e40},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.29431. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Measuring how successful a conversation is remains difficult, even for humans judging spoken dialogue. We evaluate state-of-the-art LLMs as pointwise and pairwise judges of conversational success on CANDOR, finding pointwise scoring correlates moderately with human ratings, while pairwise comparison suffers from long transcripts and positional bias. Since this leaves judge scores incomparable across models, we propose a small anchor set and a calibration function that calibrates any judge onto a shared, interpretable scale. We further release the Voice Arena Goal Dataset (VA), 200 task-oriented human-AI and human-agent conversations with pairwise annotations, revealing a substantial gap between current judges and human-level discrimination. Using VA, we test whether CANDOR-fitted calibration transfers to human-AI conversations, finding it brings judges onto a shared scale despite never observing VA during fitting.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-25 in cs.HC).
Categorie arXiv: Human-Computer Interaction (cs.HC).
Pagina del lavoro: https://arxiv.org/abs/2609.29431 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T04:50:17.670Z dalla pagina https://arxiv.org/catchup/cs.HC/2026-09-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).