Metraipolis › archive › dc_5592cf0040e2

MMLongBench-Doc-V2: A Corrected-Annotation, Semantics-Aware Revision of MMLongBench-Doc — arXiv:2608.03397

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.03397
License (as archived)
CC0-1.0
Archived
2026-10-31T12:16:14.791Z
SHA-256 of the archived text
d57e2c92e794cee84b9f679fa5ddb8225e279ab8db8b29e09b31ff135932d992
Stable URL of this page
https://metraipolis.com/documento/dc_5592cf0040e2

How to cite

MMLongBench-Doc-V2: A Corrected-Annotation, Semantics-Aware Revision of MMLongBench-Doc — arXiv:2608.03397 — arXiv (en), https://arxiv.org/abs/2608.03397 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5592cf0040e2 (Metraipolis archive dc_5592cf0040e2, sha256 d57e2c92e794cee84b9f679fa5ddb8225e279ab8db8b29e09b31ff135932d992)
BibTeX
@misc{dc5592cf0040e2,
  title = {MMLongBench-Doc-V2: A Corrected-Annotation, Semantics-Aware Revision of MMLongBench-Doc — arXiv:2608.03397},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.03397},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5592cf0040e2; sha256 d57e2c92e794cee84b9f679fa5ddb8225e279ab8db8b29e09b31ff135932d992},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03397. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

MMLongBench-Doc is a long-document QA benchmark of 1,082 questions over 135 PDFs. Two properties of it push measured scores away from the quantity they are meant to capture: the reference metric compares extracted answers, so 1,358,000 loses to 1358000; and a non-trivial share of ground-truth annotations are wrong, ambiguous, or incomplete --- concentrated, because of how they were found, in exactly the questions capable systems answer correctly. MMLongBench-Doc-V2 corrects 106 annotations, each published with the page and arithmetic that settle it, and replaces the string metric with a pinned LLM judge asked whether a response means the reference. Ten questions whose document ships under the wrong filename are removed rather than counted wrong, along with one duplicated question, leaving 1,071 questions over 134 documents. The most reusable contribution is a decision procedure for when an empty set key may be widened and when widening would destroy a deliberate negative sample; applied to all 208 rows, it widened 14. V2 scores are not comparable with published V1 numbers. The corrected corpus, the per-entry correction record and the evaluation harness are available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.03397 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T12:05:52.783Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).