Metraipolis › archive › dc_5bbe976277ba

Evaluating Structured Information Extraction with Open Models in a High Risk Public Sector Application — arXiv:2608.18289

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.18289
License (as archived)
CC0-1.0
Archived
2026-10-18T13:44:25.679Z
SHA-256 of the archived text
206bc02e9da29ca4745d5be4eec3d41c04c3de735e0d29fd40669e3e613961e3
Stable URL of this page
https://metraipolis.com/documento/dc_5bbe976277ba

How to cite

Evaluating Structured Information Extraction with Open Models in a High Risk Public Sector Application — arXiv:2608.18289 — arXiv (en), https://arxiv.org/abs/2608.18289 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5bbe976277ba (Metraipolis archive dc_5bbe976277ba, sha256 206bc02e9da29ca4745d5be4eec3d41c04c3de735e0d29fd40669e3e613961e3)
BibTeX
@misc{dc5bbe976277ba,
  title = {Evaluating Structured Information Extraction with Open Models in a High Risk Public Sector Application — arXiv:2608.18289},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.18289},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5bbe976277ba; sha256 206bc02e9da29ca4745d5be4eec3d41c04c3de735e0d29fd40669e3e613961e3},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.18289. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The extraction of structured information from unstructured documents represents a critical component of digital transformations in all sectors. While proprietary solutions dominate commercial applications, a rapidly growing ecosystem of open-source Optical Character Recognition (OCR) engines, Large Language Models (LLMs), and Vision-Language Models (VLMs) offers accessible alternatives. However, systematic evaluations on realistic, multi-step extraction pipelines remain scarce. Responsible usage of such extraction tools require comprehensive evaluations on realistic tasks, especially as these solutions will be key components of applications in the public sector that the EU AI act categorizes as high risk. To address this gap we present a comprehensive benchmark assessing the end-to-end performance of open-source systems on a complex real-world document processing task classified as high risk: Student applications for an international study program. We conduct a comprehensive empirical evaluation with state-of-the-art OCR engines, LLMs and VLMs. Our results reveal that while VLMs generally outperform OCR+LLM pipelines, even state-of-the-art open-source models struggle to handle such tasks reliably in zero-shot settings. Only 4 of 35 configurations achieved F1 scores above 0.5, with the best OCR+LLM pipeline matching top VLM performance, though most OCR+LLM combinations performed substantially worse. Roughly 75\% of all configurations scored below 0.25. Model scale influences performance, yet the relationship is non-linear: substantially larger models do not guarantee proportionally better results. Input quality, particularly the structural preservation of OCR output, emerges as a critical factor independent of downstream model capability.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-20 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Cryptography and Security (cs.CR); Information Retrieval (cs.IR); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.18289 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T13:41:17.854Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-20?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).