Metraipolis › archive › dc_853bb7381dcd

ProdCrawl: A Replayable Test Collection for Focused Crawling — arXiv:2610.40855

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.40855
License (as archived): CC0-1.0
Archived: 2026-11-03T08:20:03.699Z
SHA-256 of the archived text: 3da18cdcb8dc295ff1be3d88a1893894c8c3c7eff74b20f19ad9f840e65f5dbf
Stable URL of this page: https://metraipolis.com/documento/dc_853bb7381dcd

How to cite

ProdCrawl: A Replayable Test Collection for Focused Crawling — arXiv:2610.40855 — arXiv (en), https://arxiv.org/abs/2610.40855 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_853bb7381dcd (Metraipolis archive dc_853bb7381dcd, sha256 3da18cdcb8dc295ff1be3d88a1893894c8c3c7eff74b20f19ad9f840e65f5dbf)

BibTeX

@misc{dc853bb7381dcd,
  title = {ProdCrawl: A Replayable Test Collection for Focused Crawling — arXiv:2610.40855},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.40855},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_853bb7381dcd; sha256 3da18cdcb8dc295ff1be3d88a1893894c8c3c7eff74b20f19ad9f840e65f5dbf},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40855. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A focused crawler searches for web pages on a specific topic under a constrained download budget. Since it can only fetch a limited number of pages, its success depends on choosing which links to follow. Focused crawlers are usually evaluated on live websites by harvest rate, the fraction of downloaded pages that are relevant. This has two problems: on the one hand, websites change over time, so a crawl cannot be repeated under the same conditions; on the other hand, there is no ground truth for which pages are relevant. As a result, comparisons between crawlers are unreliable. We present ProdCrawl, a replayable test collection for focused crawling. It comprises static mirrors of 621 company and organization websites, about 1.1 million pages, with each page labeled as a product page or non-product page based on a majority vote among three large language models. Any crawler can be run on exactly the same sites and scored against the same labels. We use ProdCrawl to compare simple baselines (breadth-first, depth-first and text classifiers) with an approach that scores each link by searching for similar link texts whose target pages have already been labeled. Our results show that text classifiers and keyword matching do not significantly outperform breadth-first crawling, whereas neural embeddings, which compare link texts by semantic meaning, outperform all simple strategies. Because every run evaluates over identical pages and labels, these results are exactly reproducible. The collection, codebase, and models are publicly released to support experimental replication and fair crawler comparison.

Tema degli esploratori: statistica e scienza dei dati (elenco arXiv cs.IR, nuovi lavori del giorno).
Categorie arXiv: Information Retrieval (cs.IR).
Pagina del lavoro: https://arxiv.org/abs/2610.40855 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T08:06:23.746Z dalla pagina https://arxiv.org/list/cs.IR/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).