Metraipolis › archive › dc_47050addbb49
RefinedGAW: A Refined German Academic Web Dataset — arXiv:2610.34572
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.34572
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-27T08:11:35.724Z
- SHA-256 of the archived text
3de364536531dd41f7bd077a4eab5786c948a027d5d143f78c4f5a0b7e7c8119- Stable URL of this page
https://metraipolis.com/documento/dc_47050addbb49
How to cite
RefinedGAW: A Refined German Academic Web Dataset — arXiv:2610.34572 — arXiv (en), https://arxiv.org/abs/2610.34572 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_47050addbb49 (Metraipolis archive dc_47050addbb49, sha256 3de364536531dd41f7bd077a4eab5786c948a027d5d143f78c4f5a0b7e7c8119)
BibTeX
@misc{dc47050addbb49,
title = {RefinedGAW: A Refined German Academic Web Dataset — arXiv:2610.34572},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.34572},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_47050addbb49; sha256 3de364536531dd41f7bd077a4eab5786c948a027d5d143f78c4f5a0b7e7c8119},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.34572. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The largest open German text datasets are built from broad web crawls, which provide scale but limited coverage of specific domains. We introduce RefinedGAW, a processed and annotated resource derived from 26 semi-annual crawls of the German Academic Web, totaling 122 million documents and 314 billion tokens, with full crawl and capture history of every text. Documents labeled as academic research account for 84% of German RefinedGAW tokens across HTML and PDF. At 87.6 billion tokens, this exceeds the 64.9 billion tokens in the German subsets of FineWeb2 and FinePDFs, despite their almost five-times-larger combined size. RefinedGAW also contains substantial amounts of educational, math-heavy, and expert-oriented content with minimal commercial bias. Across HTML and PDF, 88% of German RefinedGAW documents have no near-duplicate candidate match in the three evaluated reference datasets, supporting complementary coverage. We release a queryable metadata index for fine-grained data selection and a rehydration tool for retrieving and processing source documents. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Information Retrieval (cs.IR). Pagina del lavoro: https://arxiv.org/abs/2610.34572 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-27T07:56:00.231Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).