Metraipolis › archive › dc_1386da068bf5

Filtered Vector Search in a Disaggregated Lakehouse: Composing Table-Format Pruning with Per-File ANN — arXiv:2608.05441

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.05441
License (as archived)
CC0-1.0
Archived
2026-10-25T21:33:09.829Z
SHA-256 of the archived text
4bc4307124ea905e5e67dea61976ac063543de149e476b7d2cb2e8b82848964a
Stable URL of this page
https://metraipolis.com/documento/dc_1386da068bf5

How to cite

Filtered Vector Search in a Disaggregated Lakehouse: Composing Table-Format Pruning with Per-File ANN — arXiv:2608.05441 — arXiv (en), https://arxiv.org/abs/2608.05441 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1386da068bf5 (Metraipolis archive dc_1386da068bf5, sha256 4bc4307124ea905e5e67dea61976ac063543de149e476b7d2cb2e8b82848964a)
BibTeX
@misc{dc1386da068bf5,
  title = {Filtered Vector Search in a Disaggregated Lakehouse: Composing Table-Format Pruning with Per-File ANN — arXiv:2608.05441},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.05441},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1386da068bf5; sha256 4bc4307124ea905e5e67dea61976ac063543de149e476b7d2cb2e8b82848964a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.05441. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Approximate nearest-neighbor (ANN) search increasingly runs alongside structured data - "find the 10 nearest documents where tenant='acme' AND lang='en'" - yet similarity and filtering are usually bolted together: a specialized vector index for one, a separate filter step for the other. We ask what happens when both live inside an open lakehouse table (Apache Iceberg over Parquet on object storage), where the engine already owns a mature file-pruning stack (partition pruning, zone-maps, a bitmap index). We embed an IVF index in place in each Parquet file's footer and make filtered vector queries fast not with a new filtering algorithm but by composing the table's existing file pruning with per-file ANN: the planner prunes data files by the predicate first, then runs IVF only over the survivors. The index is built distributed and non-destructively - a metadata-only Iceberg replace that every other engine still reads - and a rendezvous-hashed per-file cache keeps object-store read latency from swamping the algorithmic win. The payoff comes entirely from file pruning. On an 11.5M x 768 table, warm IVF search is ~32x faster than brute force at recall@10 >= 0.90, a selective predicate having pruned 355 of 444 data files before ANN runs; on 5M real IBM Granite embeddings, a filter arriving across a join prunes four of five region partitions and runs nearly two orders of magnitude (~94x: 14.7 s -> 157 ms) faster than the query-time join at identical top-k, once the reduction is materialized into a region-partitioned layout. We characterize when the composition pays off - it requires file-level locality on the filter column, and the residual predicate is only safe to push into the search over a provably pure (partitioned) column, not a merely sorted one - and report the failure modes we hit bolting ANN onto a lakehouse engine.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-07 in cs.DC).
Categorie arXiv: Databases (cs.DB) ; Distributed, Parallel, and Cluster Computing (cs.DC); Emerging Technologies (cs.ET); Information Retrieval (cs.IR).
Pagina del lavoro: https://arxiv.org/abs/2608.05441 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-25T21:25:05.078Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-08-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).