Metraipolis › archive › dc_96643eee69e8

COMPREHEND: Ultra-Fine-Grained Benchmark Understanding by Entailment Hierarchy — arXiv:2611.04249

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.04249
License (as archived)
CC0-1.0
Archived
2026-11-04T18:38:39.915Z
SHA-256 of the archived text
13bcd6fc149bfd3c762237615986f1fa9a8dca5337d6cb305d0e97d3299b472e
Stable URL of this page
https://metraipolis.com/documento/dc_96643eee69e8

How to cite

COMPREHEND: Ultra-Fine-Grained Benchmark Understanding by Entailment Hierarchy — arXiv:2611.04249 — arXiv (en), https://arxiv.org/abs/2611.04249 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_96643eee69e8 (Metraipolis archive dc_96643eee69e8, sha256 13bcd6fc149bfd3c762237615986f1fa9a8dca5337d6cb305d0e97d3299b472e)
BibTeX
@misc{dc96643eee69e8,
  title = {COMPREHEND: Ultra-Fine-Grained Benchmark Understanding by Entailment Hierarchy — arXiv:2611.04249},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04249},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_96643eee69e8; sha256 13bcd6fc149bfd3c762237615986f1fa9a8dca5337d6cb305d0e97d3299b472e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04249. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Understanding what kind of prompts users typically ask and how well an LLM performs on these prompt categories is crucial for LLM practitioners. The existing methods satisfy the demand by hierarchically clustering the user prompts and synthesizing the cluster names using an LLM to build a taxonomy. However, the cluster names from these methods are usually too general for humans to understand the detailed prompt distributions and identify the exact strengths and weaknesses of LLMs. In this study, we propose COMPREHEND, which automatically constructs an ultra-fine-grained entailment hierarchy for the prompts. The entailment hierarchy replaces the cluster names in the taxonomy with sentences to provide more precise cluster descriptions for more fine-grained categorization. Compared with EvalTree, a state-of-the-art taxonomy construction method, COMPREHEND improves an entailment/faithfulness metric from 22.6 to 41.7 and an informative metric from 16.3 to 39.1 in WildBench. Our experiments also show that the weakness report generated based on our entailment hierarchy is much more informative than EvalTree in terms of predicting the response scores in the benchmark. All the code is released in this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.04249 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T18:32:52.145Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).