Metraipolis › archive › dc_eccbfa7ff8e3

Same Rule, Different Verdicts: Rubric Language and Clause Structure in LLM Judges for Korean Support Dialogues — arXiv:2611.04091

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.04091
License (as archived)
CC0-1.0
Archived
2026-11-04T21:42:20.777Z
SHA-256 of the archived text
d4e56860f4308ddd85114667f9a77fc44a0dd8b6585c2b42a08dd22f5882e97d
Stable URL of this page
https://metraipolis.com/documento/dc_eccbfa7ff8e3

How to cite

Same Rule, Different Verdicts: Rubric Language and Clause Structure in LLM Judges for Korean Support Dialogues — arXiv:2611.04091 — arXiv (en), https://arxiv.org/abs/2611.04091 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_eccbfa7ff8e3 (Metraipolis archive dc_eccbfa7ff8e3, sha256 d4e56860f4308ddd85114667f9a77fc44a0dd8b6585c2b42a08dd22f5882e97d)
BibTeX
@misc{dceccbfa7ff8e3,
  title = {Same Rule, Different Verdicts: Rubric Language and Clause Structure in LLM Judges for Korean Support Dialogues — arXiv:2611.04091},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04091},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_eccbfa7ff8e3; sha256 d4e56860f4308ddd85114667f9a77fc44a0dd8b6585c2b42a08dd22f5882e97d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04091. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Services that operate in Korean often write their evaluation rubrics in Korean and hand them to an LLM judge. We test whether a judge's verdict depends on the language of the rubric and on how a rubric item is phrased. We write 12 criteria for support dialogues in four clause structures (affirmative, negated, conditional, exception) such that two pairs of structures have the same meaning, and we give each item in Korean and in English. Six low-cost models from three providers, plus one reference model, judge 60 translated ESConv dialogues and 144 constructed boundary cases under every combination (51,744 requests across the seven models). The analysis plan was fixed and committed before data collection. Cross-provider agreement is lower with Korean rubrics than with English rubrics (Cohen's kappa 0.687 vs. 0.745; difference -0.058, 95% CI [-0.072,-0.045]). Disagreement concentrates in negated, conditional, and exception items (odds ratio 5.6 relative to affirmative items). A single model changes its verdict between two items with the same meaning in 11-15% of cases. Two hypotheses are not supported: models from the same provider do not agree more than models from different providers, and adding paired examples does not reduce the language gap. We measure consistency only; no human labels are used, and all materials were drafted by an LLM.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.04091 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T21:36:04.404Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).