Metraipolis › archive › dc_ecf11be33442

Counting Documents Is Not Counting Text: Unit Bias in Web-PDF Corpus Statistics — arXiv:2608.16390

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.16390
License (as archived)
CC0-1.0
Archived
2026-10-18T18:07:14.616Z
SHA-256 of the archived text
ab7ca50268ca9981eb25634375be3680914ef4f228c10ed168cc35e7bb003a99
Stable URL of this page
https://metraipolis.com/documento/dc_ecf11be33442

How to cite

Counting Documents Is Not Counting Text: Unit Bias in Web-PDF Corpus Statistics — arXiv:2608.16390 — arXiv (en), https://arxiv.org/abs/2608.16390 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ecf11be33442 (Metraipolis archive dc_ecf11be33442, sha256 ab7ca50268ca9981eb25634375be3680914ef4f228c10ed168cc35e7bb003a99)
BibTeX
@misc{dcecf11be33442,
  title = {Counting Documents Is Not Counting Text: Unit Bias in Web-PDF Corpus Statistics — arXiv:2608.16390},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.16390},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ecf11be33442; sha256 ab7ca50268ca9981eb25634375be3680914ef4f228c10ed168cc35e7bb003a99},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.16390. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

PDF corpora advertise their size in tokens, but every rate they publish (coverage, OCR routing, re-fetch recovery, language mix) is computed per document, and none decomposes its token total. Because PDF length is extremely skewed, the two units can describe the same corpus very differently. We ask how the headline statistics of a web-PDF corpus change when each document is weighted by the text it contributes rather than counted once. We used CC-MAIN-2021-31-PDF-UNTRUNCATED (7.9M Common Crawl PDFs, 32.6B tokens), the one public corpus that pairs the fragments Common Crawl stored with the re-fetched originals. Text mass is highly concentrated: 3.02% of text-bearing documents hold half the tokens (Gini 0.807). The clearest consequence is Common Crawl's payload cap, which truncated 23.06% of these documents but 63.08% of their text. Reconstructing the truncated fragments and extracting both versions, two widely used text-layer parsers recover only 1.4% and 11.4% of that exposed text, so roughly 55-62% of the corpus's text is unrecoverable from the crawl by such pipelines; under the 5MiB cap adopted in March 2025, 30.19% of tokens would still be exposed. We recommend that corpus statistics be reported in both units, documents and tokens.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.16390 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T16:29:14.114Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).