Metraipolis › archive › dc_b079372bd17d

Schema-Guided Hierarchical Information Extraction and Semantic Evaluation Using Generative AI — arXiv:2608.06167

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.06167
License (as archived)
CC0-1.0
Archived
2026-10-25T21:31:26.050Z
SHA-256 of the archived text
5cb5f2f278933345399b9b34da6ddab66e8607628201140b131d2e0e6912e301
Stable URL of this page
https://metraipolis.com/documento/dc_b079372bd17d

How to cite

Schema-Guided Hierarchical Information Extraction and Semantic Evaluation Using Generative AI — arXiv:2608.06167 — arXiv (en), https://arxiv.org/abs/2608.06167 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b079372bd17d (Metraipolis archive dc_b079372bd17d, sha256 5cb5f2f278933345399b9b34da6ddab66e8607628201140b131d2e0e6912e301)
BibTeX
@misc{dcb079372bd17d,
  title = {Schema-Guided Hierarchical Information Extraction and Semantic Evaluation Using Generative AI — arXiv:2608.06167},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.06167},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b079372bd17d; sha256 5cb5f2f278933345399b9b34da6ddab66e8607628201140b131d2e0e6912e301},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.06167. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We present a schema-based framework for extracting complex, structured information from unstructured text documents using generative AI, followed by automated semantic evaluation of the extracted information against a gold standard. The schema, serving as an information model encoding domain knowledge, provides a unified, systematic, and consistent framework for extraction of hierarchical, nested information, with attributes of variable cardinality, and subsequent evaluation of the results. Information extraction from a document is performed in a single call to the model, in zero-shot mode. In the evaluation step, we introduce a path-based semantic matching algorithm to align the nested, variable-cardinality attributes in the extracted results with those in the gold standard. We use generative AI for semantic comparison of the extracted and gold standard values of an attribute, and introduce a rubric to classify the result of the comparison, according to domain-specific considerations, as an exact, semantic, useful, or non-match. We were able to extract 12 out of 14 attributes with an F1 score of $>$90\% from documents published by the health technology assessment organisation NICE, using the generative AI model Claude Opus 3. The time needed to extract the attributes from a document was $\sim$30 times lower than the time taken by a human domain expert. We further demonstrate generalisability of this framework across different generative AI models and transferability across different HTA organisations and languages.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-07 in cs.CL).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.06167 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-25T21:24:38.279Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).