Metraipolis › archive › dc_45441b2f89eb

Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation — arXiv:2609.22603

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.22603
License (as archived)
CC0-1.0
Archived
2026-10-09T06:49:33.718Z
SHA-256 of the archived text
646102f4e0ecd7ee5d67147cf35622821ac3d3f1ce3ce80736326f76f22ade2d
Stable URL of this page
https://metraipolis.com/documento/dc_45441b2f89eb

How to cite

Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation — arXiv:2609.22603 — arXiv (en), https://arxiv.org/abs/2609.22603 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_45441b2f89eb (Metraipolis archive dc_45441b2f89eb, sha256 646102f4e0ecd7ee5d67147cf35622821ac3d3f1ce3ce80736326f76f22ade2d)
BibTeX
@misc{dc45441b2f89eb,
  title = {Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation — arXiv:2609.22603},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.22603},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_45441b2f89eb; sha256 646102f4e0ecd7ee5d67147cf35622821ac3d3f1ce3ce80736326f76f22ade2d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.22603. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Summarization ships in countless production systems, making model selection a routine decision that depends on measuring summary quality. Existing metrics struggle to support this: ROUGE captures only surface overlap, while LLM-as-judge scores saturate to near-identical values that fail to rank models effectively. We observe this saturation across three public datasets, two proprietary datasets, and multilingual settings. Motivated by this, we introduce Semantic Scaffold, an evaluation framework that extracts a hierarchical representation of facts, questions, and entity attributes from a source text, labeling each as a main point or supporting detail, and reusing this structure as a fixed reference for scoring summaries. From this representation, we derive three diagnostic metrics: Fact Preservation Score (FPS), Question Preservation Score (QPS), and Entity Preservation Score (EPS), designed to reward the preservation of essential information while penalizing detail overload, and position them as interpretable diagnostics that remain informative where holistic axes collapse. Finally, we analyze four recurring failure modes of ROUGE and LLM-as-judge scores, demonstrating that scaffold-based evaluation remains informative where conventional metrics collapse.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.22603 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T06:48:44.943Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).