Metraipolis › archive › dc_bfd588b9abfb

Information Satisfaction: A Reader-Centered Axis for Summarization Evaluation — arXiv:2608.14457

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.14457
License (as archived)
CC0-1.0
Archived
2026-10-19T02:27:58.369Z
SHA-256 of the archived text
450b240eb6078eb9847493d4e2bac4142c8abca7b561ed031776479524585268
Stable URL of this page
https://metraipolis.com/documento/dc_bfd588b9abfb

How to cite

Information Satisfaction: A Reader-Centered Axis for Summarization Evaluation — arXiv:2608.14457 — arXiv (en), https://arxiv.org/abs/2608.14457 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bfd588b9abfb (Metraipolis archive dc_bfd588b9abfb, sha256 450b240eb6078eb9847493d4e2bac4142c8abca7b561ed031776479524585268)
BibTeX
@misc{dcbfd588b9abfb,
  title = {Information Satisfaction: A Reader-Centered Axis for Summarization Evaluation — arXiv:2608.14457},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.14457},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bfd588b9abfb; sha256 450b240eb6078eb9847493d4e2bac4142c8abca7b561ed031776479524585268},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.14457. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The majority of work on summarization evaluation focuses on general summary quality (e.g., ROUGE, BERTScore) or specific desired properties (e.g., readability, factuality). However, these metrics fail to measure the utility of a summary to an individual user. For example, a biomedical researcher learning about the latest vaccine research will have different informational needs from a family doctor. Query-focused summarization captures part of this need, but in practice, users rarely state everything relevant in a query: a single short query is likely inadequate to distinguish the needs of a researcher from those of a physician. By contrast, a reader's background or persona (their role and expertise) is comparatively stable across queries and recovers much of this missing context, which makes it a practical signal for assessing whether a summary satisfies that reader's needs. In this work, we assess how sensitive popular summarization metrics are to both informational and persona differences, and find that many popular metrics, including strong LLM-as-judge metrics, fail basic perturbation tests of informational content. We additionally conduct an expert human evaluation, measuring summary preferences based on information satisfaction given a specific person's background and use case. We find that both traditional and LLM-based metrics are insufficient measures of information satisfaction and agree poorly with human judgment.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-17 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.14457 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T02:20:31.912Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).