Metraipolis › archive › dc_95439087aedd
Institutional Books - Enriched Text: A customizable multilingual open-source pipeline for denoising, deduplicating, and annotating OCR text at scale — arXiv:2608.19026
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.19026
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T13:50:09.569Z
- SHA-256 of the archived text
a0f65982874fc3aa7d3c674ab0312a8ab1cc7f4b3e6ed88823605fce34ff84d9- Stable URL of this page
https://metraipolis.com/documento/dc_95439087aedd
How to cite
Institutional Books - Enriched Text: A customizable multilingual open-source pipeline for denoising, deduplicating, and annotating OCR text at scale — arXiv:2608.19026 — arXiv (en), https://arxiv.org/abs/2608.19026 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_95439087aedd (Metraipolis archive dc_95439087aedd, sha256 a0f65982874fc3aa7d3c674ab0312a8ab1cc7f4b3e6ed88823605fce34ff84d9)
BibTeX
@misc{dc95439087aedd,
title = {Institutional Books - Enriched Text: A customizable multilingual open-source pipeline for denoising, deduplicating, and annotating OCR text at scale — arXiv:2608.19026},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.19026},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_95439087aedd; sha256 a0f65982874fc3aa7d3c674ab0312a8ab1cc7f4b3e6ed88823605fce34ff84d9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.19026. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Released in 2025, Institutional Books: Harvard Library (IB-HL) is a collection of 983,004 volumes (242B o200k_base tokens), originally digitized through Harvard Library's participation in the Google Books Library project. As researchers and developers have begun to use IB-HL, a tension has emerged between standard large-scale preprocessing practices and the goals of careful information stewardship. Many existing pipelines optimize for web text: as a result, they tend to aggressively filter, deduplicate, restrict by language, and sometimes discard meaningful metadata. Meanwhile, researchers seeking to use IB-HL duplicate effort while performing similar processing and analysis. We describe an approach that we call Enriched Text. Instead of producing a single 'complete' stream of tokens, we normalize the text while preserving metadata through annotations. We separate endmatter, detect per-paragraph language, identify clusters of duplicate paragraphs, and compute per-paragraph bits-per-byte scores. We provide this information through HTML-like annotations layered on top of the text. By parsing these annotations, users can tailor the output to their own needs instead of accepting a global editorial decision on content. The pipeline applies to all $\approx$250 languages in the collection. This report describes this project's goals, implementation, and design rationale. The release includes IB-HL-ET (an enriched-text version of IB-HL containing 217B o200k_base tokens across 983,003 volumes, organized into 1.39B annotated subtopic paragraphs) and the pipeline that produced it. These serve to make the collection easier for machines to parse and for humans to study. Tema degli esploratori: ricerca e scienza aperta (retro-catalogo arXiv: annunci del 2026-08-20 in cs.DL). Categorie arXiv: Computation and Language (cs.CL) ; Digital Libraries (cs.DL). Pagina del lavoro: https://arxiv.org/abs/2608.19026 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T13:48:53.148Z dalla pagina https://arxiv.org/catchup/cs.DL/2026-08-20?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).