Metraipolis › archive › dc_d12eaafa6367

PersonaChatBench: Evaluating Chat Assistants with Diverse Simulated Users — arXiv:2610.42060

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.42060
License (as archived): CC0-1.0
Archived: 2026-11-02T07:03:54.279Z
SHA-256 of the archived text: c1116c6159bf6a7aa4c94ddb0451111c42fc460a1dd7f3f61cfee763c5c46cfa
Stable URL of this page: https://metraipolis.com/documento/dc_d12eaafa6367

How to cite

PersonaChatBench: Evaluating Chat Assistants with Diverse Simulated Users — arXiv:2610.42060 — arXiv (en), https://arxiv.org/abs/2610.42060 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d12eaafa6367 (Metraipolis archive dc_d12eaafa6367, sha256 c1116c6159bf6a7aa4c94ddb0451111c42fc460a1dd7f3f61cfee763c5c46cfa)

BibTeX

@misc{dcd12eaafa6367,
  title = {PersonaChatBench: Evaluating Chat Assistants with Diverse Simulated Users — arXiv:2610.42060},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.42060},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d12eaafa6367; sha256 c1116c6159bf6a7aa4c94ddb0451111c42fc460a1dd7f3f61cfee763c5c46cfa},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.42060. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Conversational assistants are expected to help people with diverse backgrounds, preferences, and communication styles. Yet evaluations centered on a single cooperative user provide limited insight into how reliably an assistant serves users with different personas. We introduce PersonaChatBench, a framework for evaluating task competence and responsiveness to diverse user needs. PersonaChatBench pairs the same underlying task with multiple simulated users, each guided by a private persona card that specifies distinct needs, preferences, and interaction behaviors. By holding the task fixed while varying the user, the framework examines how assistant performance changes across user conditions. We evaluate 20 language models across six domains, separately assessing native task outcomes, fulfillment of expressed persona needs, and simulator fidelity. Our results show that task performance and need fulfillment vary across personas and domains. Strong task performance does not necessarily translate into satisfying individual needs, and models' relative strengths in meeting those needs are not consistent across domains. PersonaChatBench makes these differences measurable, revealing gaps obscured by aggregate scores and helping identify which users an assistant serves well and where adaptation is needed. It provides a basis for evaluating consistency across users and guiding the development of assistants that better accommodate diverse needs in everyday applications.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.42060 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-02T06:57:13.234Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).