Metraipolis › archive › dc_e5b98fa69657

Extending DBLog-Based Incremental Snapshots to Handle Orphan Rows in CDC Pipelines — arXiv:2611.10810

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.10810
License (as archived)
CC0-1.0
Archived
2026-11-10T18:45:56.024Z
SHA-256 of the archived text
875acf6dc1fe80104fc43c5a11d9e40051315a27d868f5b67961e3ff51b5938f
Stable URL of this page
https://metraipolis.com/documento/dc_e5b98fa69657

How to cite

Extending DBLog-Based Incremental Snapshots to Handle Orphan Rows in CDC Pipelines — arXiv:2611.10810 — arXiv (en), https://arxiv.org/abs/2611.10810 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e5b98fa69657 (Metraipolis archive dc_e5b98fa69657, sha256 875acf6dc1fe80104fc43c5a11d9e40051315a27d868f5b67961e3ff51b5938f)
BibTeX
@misc{dce5b98fa69657,
  title = {Extending DBLog-Based Incremental Snapshots to Handle Orphan Rows in CDC Pipelines — arXiv:2611.10810},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.10810},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e5b98fa69657; sha256 875acf6dc1fe80104fc43c5a11d9e40051315a27d868f5b67961e3ff51b5938f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.10810. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Applications use heterogeneous datastores where a source feeds sinks such as search indexes or data warehouses via change data capture (CDC). As transaction logs have limited retention, CDC alone cannot bootstrap sinks, and traditional approaches either stall source writes or hold back log events. DBLog-based incremental snapshots, adopted by Debezium and Flink CDC, read tables in chunks interleaved with the log without blocking writes or long log stalls and resume where they stopped. After the initial bootstrap, a sink can drift when losing change events or receiving unintended writes. Even a repeated bootstrap can leave orphan rows that exist in the sink and not at the source. A common remedy is to populate a new empty sink, but repairing an existing sink avoids parallel pipelines, a switch and serving traffic from a drifted sink until the new sink is ready. In DBLog's original design, each chunk only emits the latest state of the rows the table still contains, so it does not handle orphan rows. We extend DBLog to delete them using what a chunk read already knows, namely the keys that the range select did not return, emitting a new kind of event that sinks apply as deletes. We also add a repair request that an external process calls with orphan candidates, which the connector checks with its own read, emitting these events for keys confirmed absent. This enables three modes. In the fully integrated mode, every chunk pass removes the orphans it finds during full or periodic repairs. In the second mode, an external drift process hands suspected orphans to the connector. The third mode combines both and is what we recommend. Together, they turn DBLog into a complete data synchronization engine whether a sink catches up on live changes or missing rows, holds wrong column values, or contains orphan rows. Their safety and convergence are formally verified in Isabelle/HOL and Lean 4.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno).
Categorie arXiv: Databases (cs.DB) ; Distributed, Parallel, and Cluster Computing (cs.DC); Logic in Computer Science (cs.LO).
Pagina del lavoro: https://arxiv.org/abs/2611.10810 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-10T18:38:07.450Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).