Metraipolis › archive › dc_06506f80bd30
SieveIVF: Threshold-Aware IVF Execution for Large-Scale Training Data Deduplication — arXiv:2608.03199
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.03199
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T13:08:35.748Z
- SHA-256 of the archived text
819605e3666319fc49a395bb6d993be5387cb14afde8d5c6b234e3e9dfe061b9- Stable URL of this page
https://metraipolis.com/documento/dc_06506f80bd30
How to cite
SieveIVF: Threshold-Aware IVF Execution for Large-Scale Training Data Deduplication — arXiv:2608.03199 — arXiv (en), https://arxiv.org/abs/2608.03199 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_06506f80bd30 (Metraipolis archive dc_06506f80bd30, sha256 819605e3666319fc49a395bb6d993be5387cb14afde8d5c6b234e3e9dfe061b9)
BibTeX
@misc{dc06506f80bd30,
title = {SieveIVF: Threshold-Aware IVF Execution for Large-Scale Training Data Deduplication — arXiv:2608.03199},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.03199},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_06506f80bd30; sha256 819605e3666319fc49a395bb6d993be5387cb14afde8d5c6b234e3e9dfe061b9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03199. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Embedding-based training data deduplication retrieves candidate duplicate edges above an application similarity threshold, but fixed-probe inverted-file (IVF) search ignores this predicate when giving every query the same partition budget. Across four Hunyuan workloads, qualifying neighbors appear early despite sharply varying search depths. We present SieveIVF, a threshold-aware IVF executor that stops after $W$ consecutive searches find no qualifying candidate. The systems challenge is to preserve partition-major batching when each query's remaining work depends on prior results. Continuous batching groups ready queries by partition. A lookahead scheduler layers on top, exposing only work committed by the stopping rule to increase concurrency without changing stopping decisions or returned results. We implement SieveIVF in Lance. At $W=8$, SieveIVF is $4.1$--$7.6\times$ faster than fixed-probe IVF on four 10M Hunyuan workloads and $6.1$--$8.4\times$ faster on two public 100M workloads under the same index and search parameters, with pooled filtered top-10 recall losses of $0.03$--$1.13$ percentage points on Hunyuan and $1.43$--$2.29$ percentage points on the public workloads. These results show how an application predicate can guide IVF work allocation without changing the index or bounded top-$k$ interface. Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-08-05 in cs.DB). Categorie arXiv: Databases (cs.DB). Pagina del lavoro: https://arxiv.org/abs/2608.03199 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T12:56:10.509Z dalla pagina https://arxiv.org/catchup/cs.DB/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).