Metraipolis › archive › dc_1c2caae85551

A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models — arXiv:2609.02054

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.02054
License (as archived)
CC0-1.0
Archived
2026-10-17T04:16:34.512Z
SHA-256 of the archived text
63caaa44fc713f4360a02f9efe3c07373f1b3452718369ce84f2b633e530ee3b
Stable URL of this page
https://metraipolis.com/documento/dc_1c2caae85551

How to cite

A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models — arXiv:2609.02054 — arXiv (en), https://arxiv.org/abs/2609.02054 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1c2caae85551 (Metraipolis archive dc_1c2caae85551, sha256 63caaa44fc713f4360a02f9efe3c07373f1b3452718369ce84f2b633e530ee3b)
BibTeX
@misc{dc1c2caae85551,
  title = {A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models — arXiv:2609.02054},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.02054},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1c2caae85551; sha256 63caaa44fc713f4360a02f9efe3c07373f1b3452718369ce84f2b633e530ee3b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.02054. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large Language Models (LLMs) are increasingly deployed in interactive systems where understanding user intent precisely is paramount. A key capability for such systems is effective question clarification, especially when user queries are ambiguous or underspecified. This paper introduces a novel tri-agent framework for the robust evaluation of an LLM's ability to engage in clarifying dialogue. Our framework comprises three distinct LLM-based agents: (1) a Question Clarifying Agent (QCA), the system under evaluation, tasked with identifying ambiguities and posing clarifying questions; (2) a Respondent Agent (RA), designed to simulate human user responses, potentially including irrelevant or challenging replies; and (3) an Evaluator Agent (EA), an LLM-as-a-judge, which assesses the quality of the dialogue based on a comprehensive set of metrics. We detail a methodology for synthetic data generation in the supply chain domain as an example. We propose metrics evaluating ambiguity handling, question quality, dialogue efficiency, language appropriateness, and final intent alignment. We also briefly discuss the validation of the EA against human judgments. This work provides a structured approach to benchmark, validate, and improve the clarification capabilities of conversational LLM applications.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-03 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.02054 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T03:26:49.775Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-03?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).