Metraipolis › archive › dc_22d5c4069bda
Post-hoc Alignment of LLM-judges to Human Judgment Distribution — arXiv:2609.01073
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.01073
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T06:41:28.006Z
- SHA-256 of the archived text
59e8b09782859ebd202b269a3364b96eedaea1071804ff2171c74ca27373ad8b- Stable URL of this page
https://metraipolis.com/documento/dc_22d5c4069bda
How to cite
Post-hoc Alignment of LLM-judges to Human Judgment Distribution — arXiv:2609.01073 — arXiv (en), https://arxiv.org/abs/2609.01073 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_22d5c4069bda (Metraipolis archive dc_22d5c4069bda, sha256 59e8b09782859ebd202b269a3364b96eedaea1071804ff2171c74ca27373ad8b)
BibTeX
@misc{dc22d5c4069bda,
title = {Post-hoc Alignment of LLM-judges to Human Judgment Distribution — arXiv:2609.01073},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.01073},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_22d5c4069bda; sha256 59e8b09782859ebd202b269a3364b96eedaea1071804ff2171c74ca27373ad8b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.01073. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The LLM-as-a-judge (LLMaJ) framework offers a cost-effective and reproducible solution for automatic evaluation. However, current evaluation practices typically compare LLMaJ judgments against aggregated ground-truth labels, overlooking the valuable information contained in Human Label Variation (HLV). Inspired by an increasing line of work that proposes to leverage HLV, we systematically study LLMaJ performance on predicting both a single, aggregated ground truth hard-label and unaggregated soft-labels that represent Human Judgment Distributions (HJD). Our results across five diverse datasets reveal that while LLMs achieve near human-level performance at hard-label prediction on most tasks, they exhibit poor performance when predicting soft-labels. To address this limitation, we propose NAPHA (eNtropy-Aware Post-Hoc Alignment), a simple yet effective lightweight post-hoc alignment method that matches the LLM distribution to the HJD by first assigning an instance to a discrete entropy class and then routing it to specialized, trained alignment models. We find that NAPHA consistently improves soft-labels prediction across base LLM models and datasets, with particularly strong gains on high-entropy instances where capturing diverse human perspectives is most critical. We also show via oracle experiments that improving entropy class prediction can substantially enhance NAPHA's practical effectiveness. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.01073 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T06:33:38.841Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).