Metraipolis › archive › dc_e0269c19ae34

ReScraper: Unified Scraping and Cleaning of Web Data for Effective LLM Pretraining — arXiv:2609.34287

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.34287
License (as archived): CC0-1.0
Archived: 2026-10-07T21:02:21.130Z
SHA-256 of the archived text: 033ccf504c3412fe6c1aa6e6b3584ab772691dd5af0f9a5cf035116a22baaa05
Stable URL of this page: https://metraipolis.com/documento/dc_e0269c19ae34

How to cite

ReScraper: Unified Scraping and Cleaning of Web Data for Effective LLM Pretraining — arXiv:2609.34287 — arXiv (en), https://arxiv.org/abs/2609.34287 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e0269c19ae34 (Metraipolis archive dc_e0269c19ae34, sha256 033ccf504c3412fe6c1aa6e6b3584ab772691dd5af0f9a5cf035116a22baaa05)

BibTeX

@misc{dce0269c19ae34,
  title = {ReScraper: Unified Scraping and Cleaning of Web Data for Effective LLM Pretraining — arXiv:2609.34287},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.34287},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e0269c19ae34; sha256 033ccf504c3412fe6c1aa6e6b3584ab772691dd5af0f9a5cf035116a22baaa05},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34287. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM pretraining corpora are normally cleaned by a stack of hand-written heuristics. A heuristic scraper extracts the main content from HTML, and dozens of rule-based filters then clean it, so corpus quality is capped by the coarseness and accuracy of the rules. In this work, we propose ReScraper, a unified language model of only 0.6B parameters that replaces this entire stack. To train ReScraper, we carefully curate supervised data from the outputs of three teacher models, so it learns to first extract the main content from raw data and then choose among four operations: keeping the page as extracted, editing out noisy lines and spans, deleting it entirely, or rewriting it when it is poorly written but informative. Based on the same crawled data pool, pretraining 400M, 1.4B, and 2.8B models on our curated data improves the DCLM Core score by a relative 3.8--4.7% over the strongest baseline at each scale, including the costly multi-agent curation. Our analyses show that each operation plays a distinct and complementary role, and that extracting and cleaning in one model outperforms a cascade of separate models. ReScraper also concentrates its operations on the pages that need them, raising the quality of poor pages the most while keeping the corpus diverse. These results demonstrate the feasibility and effectiveness of AI4AI for pretraining data curation, where a small learned model takes over an entire stage of the pipeline from hand-written heuristics. We open-source our code at this https URL

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.34287 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:02.926Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).