Metraipolis › archive › dc_8fd0168011bb

Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluation — arXiv:2608.10385

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.10385
License (as archived): CC0-1.0
Archived: 2026-10-19T06:09:06.314Z
SHA-256 of the archived text: 0491dd26c82d7759dbe107f8c411cd43ed4689217e541c52b210cc494cf11614
Stable URL of this page: https://metraipolis.com/documento/dc_8fd0168011bb

How to cite

Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluation — arXiv:2608.10385 — arXiv (en), https://arxiv.org/abs/2608.10385 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8fd0168011bb (Metraipolis archive dc_8fd0168011bb, sha256 0491dd26c82d7759dbe107f8c411cd43ed4689217e541c52b210cc494cf11614)

BibTeX

@misc{dc8fd0168011bb,
  title = {Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluation — arXiv:2608.10385},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.10385},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8fd0168011bb; sha256 0491dd26c82d7759dbe107f8c411cd43ed4689217e541c52b210cc494cf11614},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.10385. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) are increasingly used as relevance assessors in information retrieval (IR) evaluation, raising questions about how assessor framing affects judgment reliability and downstream system comparison. We study persona conditioning as a diagnostic mechanism for exposing LLM assessor sensitivity. Using task-oriented personas drawn from two complementary sources (PersonaHub and NVIDIA Nemotron-Personas-USA), we instantiate five assessor roles emphasizing intent interpretation, domain expertise, contrastive judgment, evidence verification, and global search-quality assessment, compared with a standard UMBRELA baseline. Across six LLM backbones on TREC DL20 and RAG24, our analyses reveal structured rather than uniform assessor sensitivity. Judgments usually remain close to the baseline while shifting assessment strictness, evidential threshold, or interpretation emphasis rather than producing widespread relevance reversals. At the system level, high-capacity models preserve system-ranking agreement, while smaller models amplify persona-induced instability. Local rank-displacement analysis shows sensitivity concentrates on particular retrieval systems and system types, especially neural ranking/reranking systems on DL20 and RAG-oriented pipelines on RAG24. Persona source matters less than assessor role and model capacity. These findings position persona-conditioned judging as a controlled sensitivity probe for stress-testing LLM-based IR evaluation pipelines and identifying systems whose evaluation outcomes are sensitive to assessor framing.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-12 in cs.AI).
Categorie arXiv: Information Retrieval (cs.IR) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.10385 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T05:57:28.358Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-12?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).