Metraipolis › archive › dc_06506f80bd30

SieveIVF: Threshold-Aware IVF Execution for Large-Scale Training Data Deduplication — arXiv:2608.03199

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.03199
License (as archived)
CC0-1.0
Archived
2026-10-31T13:08:35.748Z
SHA-256 of the archived text
819605e3666319fc49a395bb6d993be5387cb14afde8d5c6b234e3e9dfe061b9
Stable URL of this page
https://metraipolis.com/documento/dc_06506f80bd30

How to cite

SieveIVF: Threshold-Aware IVF Execution for Large-Scale Training Data Deduplication — arXiv:2608.03199 — arXiv (en), https://arxiv.org/abs/2608.03199 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_06506f80bd30 (Metraipolis archive dc_06506f80bd30, sha256 819605e3666319fc49a395bb6d993be5387cb14afde8d5c6b234e3e9dfe061b9)
BibTeX
@misc{dc06506f80bd30,
  title = {SieveIVF: Threshold-Aware IVF Execution for Large-Scale Training Data Deduplication — arXiv:2608.03199},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.03199},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_06506f80bd30; sha256 819605e3666319fc49a395bb6d993be5387cb14afde8d5c6b234e3e9dfe061b9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03199. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Embedding-based training data deduplication retrieves candidate duplicate edges above an application similarity threshold, but fixed-probe inverted-file (IVF) search ignores this predicate when giving every query the same partition budget. Across four Hunyuan workloads, qualifying neighbors appear early despite sharply varying search depths. We present SieveIVF, a threshold-aware IVF executor that stops after $W$ consecutive searches find no qualifying candidate. The systems challenge is to preserve partition-major batching when each query's remaining work depends on prior results. Continuous batching groups ready queries by partition. A lookahead scheduler layers on top, exposing only work committed by the stopping rule to increase concurrency without changing stopping decisions or returned results. We implement SieveIVF in Lance. At $W=8$, SieveIVF is $4.1$--$7.6\times$ faster than fixed-probe IVF on four 10M Hunyuan workloads and $6.1$--$8.4\times$ faster on two public 100M workloads under the same index and search parameters, with pooled filtered top-10 recall losses of $0.03$--$1.13$ percentage points on Hunyuan and $1.43$--$2.29$ percentage points on the public workloads. These results show how an application predicate can guide IVF work allocation without changing the index or bounded top-$k$ interface.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-08-05 in cs.DB).
Categorie arXiv: Databases (cs.DB).
Pagina del lavoro: https://arxiv.org/abs/2608.03199 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T12:56:10.509Z dalla pagina https://arxiv.org/catchup/cs.DB/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).