Metraipolis › archive › dc_7ba36b48686f

Optimal Design for Active Preference Learning with Biased LLM Judges — arXiv:2609.38860

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.38860
License (as archived)
CC0-1.0
Archived
2026-10-07T12:44:25.438Z
SHA-256 of the archived text
3a589d1634ab98de4f534a536f397023aee25bd4ad2d385781957280d0828c49
Stable URL of this page
https://metraipolis.com/documento/dc_7ba36b48686f

How to cite

Optimal Design for Active Preference Learning with Biased LLM Judges — arXiv:2609.38860 — arXiv (en), https://arxiv.org/abs/2609.38860 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7ba36b48686f (Metraipolis archive dc_7ba36b48686f, sha256 3a589d1634ab98de4f534a536f397023aee25bd4ad2d385781957280d0828c49)
BibTeX
@misc{dc7ba36b48686f,
  title = {Optimal Design for Active Preference Learning with Biased LLM Judges — arXiv:2609.38860},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.38860},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7ba36b48686f; sha256 3a589d1634ab98de4f534a536f397023aee25bd4ad2d385781957280d0828c49},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38860. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Learning from human preferences is central to large language model (LLM) alignment, but human preference annotation is costly. Active preference learning reduces this cost by selecting informative comparisons, and LLM judges can provide additional scalable feedback. However, the preferences of the judges may deviate from those of the target human population. Even after calibration on trusted reference data, active acquisition can shift the comparison distribution and expose residual judge bias. We therefore incorporate judge deviations into the acquisition design rather than relying on a separate calibration stage. Under joint estimation, comparisons that appear highly informative about the reward may also reflect judge bias and therefore provide less information about human preferences. To address this issue, we propose Nuisance-Adjusted Optimal Design (NAOD), a comparison-selection strategy that prioritizes policy-relevant target information after nuisance adjustment and uses the Frank-Wolfe algorithm for optimization. Theoretically, we establish a sharp conditional local asymptotic minimax lower bound on policy risk and construct an estimator that attains it. We further characterize the finite-sample cost of learning the nuisance representation and show that representation error can reverse an oracle design advantage. Finally, we validate these predictions experimentally and evaluate NAOD on Chatbot Arena data across 17 judges, 15 budget configurations, and 15 random cluster-level splits. NAOD reduces the mean regret of proxy policy by 29.1% relative to a matched target-information design, outperforms existing methods, and improves human-preference prediction on held-out data.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.38860 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:14:13.959Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).