Metraipolis › archive › dc_5b2faf2ee842

Evidence First, Arithmetic Second: A System Report and Failure Analysis for DocSem — arXiv:2609.39013

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.39013
License (as archived)
CC0-1.0
Archived
2026-10-07T13:13:38.657Z
SHA-256 of the archived text
59dd408ff83efc40f62b693347d7ade4eaa596711e3d486d6fe7e97f20c584f0
Stable URL of this page
https://metraipolis.com/documento/dc_5b2faf2ee842

How to cite

Evidence First, Arithmetic Second: A System Report and Failure Analysis for DocSem — arXiv:2609.39013 — arXiv (en), https://arxiv.org/abs/2609.39013 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5b2faf2ee842 (Metraipolis archive dc_5b2faf2ee842, sha256 59dd408ff83efc40f62b693347d7ade4eaa596711e3d486d6fe7e97f20c584f0)
BibTeX
@misc{dc5b2faf2ee842,
  title = {Evidence First, Arithmetic Second: A System Report and Failure Analysis for DocSem — arXiv:2609.39013},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.39013},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5b2faf2ee842; sha256 59dd408ff83efc40f62b693347d7ade4eaa596711e3d486d6fe7e97f20c584f0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.39013. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

EVICALC, our system for the DocSem shared task, achieved 8.61% joint accuracy on 1,730 tasks in the official final test evaluation. It reads a PDF, selects a passage, asks a language model to write an arithmetic expression, and evaluates that expression in local code. Saved intermediate results support inspection of failures. A separate public-validation run achieved 92.17% answer accuracy and 1.00 evidence F1. The configurations and metrics differ, so these scores are not a controlled comparison. Our manual, post-hoc analysis is descriptive: in one inspected case, optical character recognition (OCR) and block grouping merged the relevant passage into another block, and the system answered from unrelated text. An exploratory study of reading page images on 100 documents returned evidence identifiers for only 22 documents. These descriptive findings motivate further evaluation; they do not establish the causes of the overall score.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Information Retrieval (cs.IR).
Pagina del lavoro: https://arxiv.org/abs/2609.39013 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:44:51.785Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).