Metraipolis › archive › dc_bf6e57faf397
Do LLMs Know What to Ask and When? Evaluating Multi-Turn Information Seeking — arXiv:2608.14808
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.14808
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T17:08:07.894Z
- SHA-256 of the archived text
1853437660c1d568d9ee246997c3340490edd7b9b05738ed330f9bb13f206776- Stable URL of this page
https://metraipolis.com/documento/dc_bf6e57faf397
How to cite
Do LLMs Know What to Ask and When? Evaluating Multi-Turn Information Seeking — arXiv:2608.14808 — arXiv (en), https://arxiv.org/abs/2608.14808 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bf6e57faf397 (Metraipolis archive dc_bf6e57faf397, sha256 1853437660c1d568d9ee246997c3340490edd7b9b05738ed330f9bb13f206776)
BibTeX
@misc{dcbf6e57faf397,
title = {Do LLMs Know What to Ask and When? Evaluating Multi-Turn Information Seeking — arXiv:2608.14808},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.14808},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bf6e57faf397; sha256 1853437660c1d568d9ee246997c3340490edd7b9b05738ed330f9bb13f206776},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.14808. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
When a user question is underspecified, a capable model should recognize that its context is insufficient, identify the missing information, ask for it, and respond only once that information determines a unique answer. We formalize multi-turn information seeking as solving a k-underspecified constraint satisfaction problem, where k is the number of variables jointly required to determine the target and therefore measures the degree of missing information. We instantiate the formulation in MT-InfoSeek, a controlled evaluation suite of 5,251 problems and 9,006 task instances spanning mathematics, logic, biology, medicine, and general knowledge. We evaluate models along three axes: what they ask, when they ask it, and how the acquired information affects the final answer. Performance degrades across models and domains as underspecification increases. Models recognize that additional information is needed but underestimate how much, and in logical problems at k = 2 they under-predict the degree of missing information about four times as often as they over-predict it. They also fail to identify a minimal sufficient set of queries, improve only marginally when given the true k, and often stop before acquiring sufficient information. In tasks with ordered dependencies, an incorrect query order reduces final accuracy even when the model eventually acquires all necessary information. We measure information seeking directly through final sufficiency, which records whether the acquired information determines the target independent of answer generation. This separation shows differences between models that final accuracy alone does not capture, and indicates that the ability to seek information over multiple turns is distinct from the ability to generate answers and is not measured by current LLM evaluations. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.14808 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T16:29:13.738Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).