Metraipolis › archive › dc_f91aebbfcb68

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness — arXiv:2610.31172

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.31172
License (as archived): CC0-1.0
Archived: 2026-10-23T15:15:21.966Z
SHA-256 of the archived text: 0b06eb21c22ad01cc044ce7f072ca6ebc216576d6de6c80d34d88688f25db74b
Stable URL of this page: https://metraipolis.com/documento/dc_f91aebbfcb68

How to cite

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness — arXiv:2610.31172 — arXiv (en), https://arxiv.org/abs/2610.31172 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f91aebbfcb68 (Metraipolis archive dc_f91aebbfcb68, sha256 0b06eb21c22ad01cc044ce7f072ca6ebc216576d6de6c80d34d88688f25db74b)

BibTeX

@misc{dcf91aebbfcb68,
  title = {Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness — arXiv:2610.31172},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.31172},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f91aebbfcb68; sha256 0b06eb21c22ad01cc044ce7f072ca6ebc216576d6de6c80d34d88688f25db74b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.31172. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM-as-a-Judge evaluation is increasingly applied to multilingual tasks, yet prompting strategies such as Chain-of-Thought and Role Conditioning are recommended based solely on English benchmarks. Whether their effectiveness transfers across judge models and languages remains unknown: prompt-sensitivity studies are largely English-centric, while existing multilingual judge studies fix the prompt template. In this work, we address this gap by measuring this interaction through a benchmark of 9 prompting strategies, 4 open-weight judges, 27 languages, and 4 tasks across 13 datasets, yielding over 1M pointwise scores on gold-reference outputs, validated against human judgments. We find that prompting strategies are not transferable: the choice of the judge model dominates the choice of prompting strategy, strategy effects reverse direction across judges, and stability degrades in low-resource languages, where these effects are concealed by aggregate reporting. These findings show that prompting effects in multilingual evaluation are conditioned on the judge, task, and language. Consequently, prompting strategies should be validated for the intended evaluation setting, and cross-study comparisons should account for both judge and prompt configuration.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.31172 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-23T15:09:08.594Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).