Metraipolis › archive › dc_7aae65460629
Deduplication-while-Training: A Resilient Paradigm for Privacy-Preserving Cross-Client Deduplication in Federated Learning — arXiv:2609.31262
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.31262
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T02:04:57.076Z
- SHA-256 of the archived text
5b483f32c2fe3a4a946766539721a1f63f7c18f2819f66a721e9b8247da9bf99- Stable URL of this page
https://metraipolis.com/documento/dc_7aae65460629
How to cite
Deduplication-while-Training: A Resilient Paradigm for Privacy-Preserving Cross-Client Deduplication in Federated Learning — arXiv:2609.31262 — arXiv (en), https://arxiv.org/abs/2609.31262 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7aae65460629 (Metraipolis archive dc_7aae65460629, sha256 5b483f32c2fe3a4a946766539721a1f63f7c18f2819f66a721e9b8247da9bf99)
BibTeX
@misc{dc7aae65460629,
title = {Deduplication-while-Training: A Resilient Paradigm for Privacy-Preserving Cross-Client Deduplication in Federated Learning — arXiv:2609.31262},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.31262},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7aae65460629; sha256 5b483f32c2fe3a4a946766539721a1f63f7c18f2819f66a721e9b8247da9bf99},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.31262. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Cross-client duplicate data in large language model training corpora degrades the efficiency of federated learning (FL) while exacerbating model memorization and privacy risks. Privacy-preserving cross-client deduplication effectively mitigates this issue by eliminating duplicate training data. However, existing schemes all follow a "Deduplication-before-Training" paradigm. This serially coupled paradigm incurs high fault-tolerance costs and lacks support for dynamic client joining. To this end, we propose an unexplored paradigm called "Deduplication-while-Training (DwT)", which enables concurrent deduplication and training. DwT transforms cross-client deduplication from a one-time, globally synchronous preprocessing operation into a continuous online service with state management, concurrent claiming, and failure recovery. By enabling state synchronization and task takeover, it minimizes the impact of client disconnections on the overall training progress while supporting the dynamic joining of clients. We design DwT-FL, a privacy-preserving deduplication system, to support DwT. By designing a concurrent state-claim mechanism and a hot-cold dual-queue scheduling strategy, DwT-FL enables the parallel execution of secure deduplication and model training, while effectively handling client disconnections and dynamic joins. Experimental evaluations demonstrate that, compared to the state-of-the-art scheme, DwT-FL significantly reduces the time overhead of failure recovery and dynamic joining by up to 93.04% and 94.18%, respectively. This provides an efficient and elastic concurrent deduplication scheme for dynamic and unstable FL environments. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-28 in cs.DC). Categorie arXiv: Cryptography and Security (cs.CR) ; Distributed, Parallel, and Cluster Computing (cs.DC). Pagina del lavoro: https://arxiv.org/abs/2609.31262 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T01:40:22.307Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-09-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).