Metraipolis › archive › dc_2f0f91eac0c8
Validating the Comparison, Not Just the Scorer: Why Overall Accuracy Can Mislead in Interactive LLM Evaluation — arXiv:2610.31678
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.31678
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-26T19:50:50.725Z
- SHA-256 of the archived text
0aa1bfb5fa0e289c2f726cac5fe4ac3a783ef19570963bd05d40af6f6d8ab9af- Stable URL of this page
https://metraipolis.com/documento/dc_2f0f91eac0c8
How to cite
Validating the Comparison, Not Just the Scorer: Why Overall Accuracy Can Mislead in Interactive LLM Evaluation — arXiv:2610.31678 — arXiv (en), https://arxiv.org/abs/2610.31678 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2f0f91eac0c8 (Metraipolis archive dc_2f0f91eac0c8, sha256 0aa1bfb5fa0e289c2f726cac5fe4ac3a783ef19570963bd05d40af6f6d8ab9af)
BibTeX
@misc{dc2f0f91eac0c8,
title = {Validating the Comparison, Not Just the Scorer: Why Overall Accuracy Can Mislead in Interactive LLM Evaluation — arXiv:2610.31678},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.31678},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2f0f91eac0c8; sha256 0aa1bfb5fa0e289c2f726cac5fe4ac3a783ef19570963bd05d40af6f6d8ab9af},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.31678. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Many LLM studies compare how models respond to different follow-up prompts. For example, one prompt may neutrally ask a model to reconsider, while another insists that an invented film really exists. The difference is used to measure the effect of conversational pressure. An automatic scorer then labels whether each response makes an unsupported factual claim. Researchers commonly validate the scorer with one overall accuracy or $F_1$ score. However, high overall performance can hide prompt-specific mistakes: if the scorer makes more errors after one prompt, the measured difference can be too large, too small, or point in the wrong direction. An audit containing mostly difficult responses can also misrepresent performance in the full study. We derive how such errors alter a prompt comparison and examine the problem using six models, 180 invented-entity questions, and six prompt conditions. In a balanced audit of 288 responses, overall $F_1$ was $.916$. Yet, depending on the prompt, the scorer flagged 2.6%-33.3% of responses that the primary LLM rater judged not to contain an unsupported claim; these prompt-specific estimates were uncertain. In a second audit, the scorer and two separate LLM raters evaluated the same 300 model-question cases under each prompt. An apparent 9.3-point increase from asserting certainty rather than neutrally asking the model to reconsider fell to 2.3 points under both raters. The primary-rater interval included zero, so it did not establish an increase. Two other prompt comparisons kept their direction under both raters, while an attempted prompt-specific correction remained too uncertain for a firm conclusion. Researchers should therefore validate the exact difference they plan to report, not just the scorer as a whole: audit both prompts on matching cases and include scorer and rater uncertainty in the final result. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC). Pagina del lavoro: https://arxiv.org/abs/2610.31678 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-26T19:45:33.342Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).