Metraipolis › archive › dc_baee844514e2

PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information? — arXiv:2610.19314

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.19314
License (as archived)
CC0-1.0
Archived
2026-10-15T20:23:47.327Z
SHA-256 of the archived text
d57a15d0ac8af6a8cb68deae1018b7f1c640624c978ada936c164d33aeb3b0eb
Stable URL of this page
https://metraipolis.com/documento/dc_baee844514e2

How to cite

PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information? — arXiv:2610.19314 — arXiv (en), https://arxiv.org/abs/2610.19314 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_baee844514e2 (Metraipolis archive dc_baee844514e2, sha256 d57a15d0ac8af6a8cb68deae1018b7f1c640624c978ada936c164d33aeb3b0eb)
BibTeX
@misc{dcbaee844514e2,
  title = {PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information? — arXiv:2610.19314},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.19314},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_baee844514e2; sha256 d57a15d0ac8af6a8cb68deae1018b7f1c640624c978ada936c164d33aeb3b0eb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.19314. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language model (LLM) evaluators are increasingly used to assess personalized assistants, yet whether they can reliably judge personalization has rarely been tested. We introduce PersEvalBench, a benchmark testing whether an evaluator can recognize appropriate and inappropriate uses of user history. It contains 403 multi-episode scenarios where user information should persist, transfer, or cease to apply. The benchmark covers nine controlled error subtypes across three families: under-personalization, over-personalization, and misapplied-state errors. Human audits confirm that generated errors match their intended types and that the prescribed uses of user information are reasonable. Across five LLM evaluators, recognition of controlled personalization errors remains limited even with the full dialogue history. On the human-evaluated subset, under-personalization shows the largest gap: humans recognize 94.4% of these errors, compared with 63.9% for the strongest reported evaluator. Evaluators often recognize that a response is problematic but reverse the error direction, mistaking failure to follow an applicable preference (UNDER) for use of an inapplicable preference (OVER). Easier evidence access offers little help: highlighting the relevant history turns raises error recognition only slightly, suggesting that the remaining difficulty lies in judging what role that information should play in the response.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.19314 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T19:36:18.181Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).