Metraipolis › archive › dc_ec9be51c90cd

ProdCrawl: A Replayable Test Collection for Focused Crawling — arXiv:2610.40855

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.40855
License (as archived): CC0-1.0
Archived: 2026-11-03T05:45:34.177Z
SHA-256 of the archived text: 3cfa4202a977790ece0bab4336d8f42f66e3325132aa5c409f35f48d53bcbe8b
Stable URL of this page: https://metraipolis.com/documento/dc_ec9be51c90cd

How to cite

ProdCrawl: A Replayable Test Collection for Focused Crawling — arXiv:2610.40855 — arXiv (en), https://arxiv.org/abs/2610.40855 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ec9be51c90cd (Metraipolis archive dc_ec9be51c90cd, sha256 3cfa4202a977790ece0bab4336d8f42f66e3325132aa5c409f35f48d53bcbe8b)

BibTeX

@misc{dcec9be51c90cd,
  title = {ProdCrawl: A Replayable Test Collection for Focused Crawling — arXiv:2610.40855},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.40855},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ec9be51c90cd; sha256 3cfa4202a977790ece0bab4336d8f42f66e3325132aa5c409f35f48d53bcbe8b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40855. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A focused crawler searches for web pages on a specific topic under a constrained download budget. Since it can only fetch a limited number of pages, its success depends on choosing which links to follow. Focused crawlers are usually evaluated on live websites by harvest rate, the fraction of downloaded pages that are relevant. This has two problems: on the one hand, websites change over time, so a crawl cannot be repeated under the same conditions; on the other hand, there is no ground truth for which pages are relevant. As a result, comparisons between crawlers are unreliable. We present ProdCrawl, a replayable test collection for focused crawling. It comprises static mirrors of 621 company and organization websites, about 1.1 million pages, with each page labeled as a product page or non-product page based on a majority vote among three large language models. Any crawler can be run on exactly the same sites and scored against the same labels. We use ProdCrawl to compare simple baselines (breadth-first, depth-first and text classifiers) with an approach that scores each link by searching for similar link texts whose target pages have already been labeled. Our results show that text classifiers and keyword matching do not significantly outperform breadth-first crawling, whereas neural embeddings, which compare link texts by semantic meaning, outperform all simple strategies. Because every run evaluates over identical pages and labels, these results are exactly reproducible. The collection, codebase, and models are publicly released to support experimental replication and fair crawler comparison.

Tema degli esploratori: statistica e scienza dei dati (elenco arXiv cs.IR, nuovi lavori del giorno).
Categorie arXiv: Information Retrieval (cs.IR).
Pagina del lavoro: https://arxiv.org/abs/2610.40855 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T05:33:19.060Z dalla pagina https://arxiv.org/list/cs.IR/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).