Metraipolis › archive › dc_039e7bbae31e
How Many Humans Are 32 LLM Judges Worth? — arXiv:2609.21277
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.21277
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-10T01:27:21.034Z
- SHA-256 of the archived text
32a96062e17ab6b02d2fc7e90f12859db1b2b331f4eaefdfdfeb470117560feb- Stable URL of this page
https://metraipolis.com/documento/dc_039e7bbae31e
How to cite
How Many Humans Are 32 LLM Judges Worth? — arXiv:2609.21277 — arXiv (en), https://arxiv.org/abs/2609.21277 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_039e7bbae31e (Metraipolis archive dc_039e7bbae31e, sha256 32a96062e17ab6b02d2fc7e90f12859db1b2b331f4eaefdfdfeb470117560feb)
BibTeX
@misc{dc039e7bbae31e,
title = {How Many Humans Are 32 LLM Judges Worth? — arXiv:2609.21277},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.21277},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_039e7bbae31e; sha256 32a96062e17ab6b02d2fc7e90f12859db1b2b331f4eaefdfdfeb470117560feb},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.21277. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
A panel's human-equivalent size is target-specific. Matching a fixed 32-judge panel to empirical human label distributions on three ChaosNLI tasks yields two distinct effective sizes: distributional-error matching gives $\nu_{\mathrm{MSE}}=2.304$, $3.750$, and $3.445$, whereas spectral matching gives $\nu_H=4.242$, $6.459$, and $6.499$, a gap of $1.72$--$1.89\times$; a binary-error diagnostic credits the same panels with only $1.971$--$2.227$ effective votes. Extrapolating the distributional-error curve at fixed squared mean residual, mean member variance, and normalized mean covariance gives asymptotes of $2.392$, $3.990$, and $3.655$, with 32 judges already reaching $94.0$--$96.3\%$. An exact spectral identity explains the gap: error depends on member energy and on the orientation of residual variation relative to averaging, information that the participation ratio (PR) discards. A realizable hard-label construction confirms that higher spectral diversity can coexist with worse distribution recovery even under equal member energies and nonnegative correlations, and the consensus direction retains $\gamma_{\mathrm{co}}=43.8\%$, $33.7\%$, and $35.9\%$ of centered residual variance. An external check on CC-1000, a 1,000-item Civil Comments subset with a different panel, gives $\nu_H=2.84$. For panel choice, we establish an existence result and one feasible path: exhaustive enumeration at $k\in\{5,7\}$ shows that panels beating the accuracy-top-$k$ baseline on both accuracy and $\nu_H$ always exist, and greedily swapping at most two members reaches $24.8$--$56.0\%$ higher $\nu_H$ at $0.10$--$1.10$ percentage points higher accuracy. Our dataset and code are available at this https URL .
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-21 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.21277 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T01:00:30.275Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).