Metraipolis › archive › dc_aadb647bcb23

Beyond Marginal Monitoring: Distributed Joint-Distribution Testing for Data Concept Drift in Large Scale E-Commerce Operations — arXiv:2610.08132

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.08132
License (as archived)
CC0-1.0
Archived
2026-10-07T11:24:41.187Z
SHA-256 of the archived text
43c7bc3cedbc43137fc0a9a17c4412d6b1c05ef16a689eb0c5c3c458d7c84a31
Stable URL of this page
https://metraipolis.com/documento/dc_aadb647bcb23

How to cite

Beyond Marginal Monitoring: Distributed Joint-Distribution Testing for Data Concept Drift in Large Scale E-Commerce Operations — arXiv:2610.08132 — arXiv (en), https://arxiv.org/abs/2610.08132 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_aadb647bcb23 (Metraipolis archive dc_aadb647bcb23, sha256 43c7bc3cedbc43137fc0a9a17c4412d6b1c05ef16a689eb0c5c3c458d7c84a31)
BibTeX
@misc{dcaadb647bcb23,
  title = {Beyond Marginal Monitoring: Distributed Joint-Distribution Testing for Data Concept Drift in Large Scale E-Commerce Operations — arXiv:2610.08132},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.08132},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_aadb647bcb23; sha256 43c7bc3cedbc43137fc0a9a17c4412d6b1c05ef16a689eb0c5c3c458d7c84a31},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.08132. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Concept drift threatens production machine learning, yet the empirical behavior of multivariate two-sample drift detectors at scale remains under-characterized. Existing benchmarks rarely address the hundreds of millions of rows and high-cardinality features typical of industrial-operational datasets. We evaluate five multi-column two-sample tests (marginal, projection-based, and kernel embedding methods) across three complementary environments: the Harvard Dataverse, a validated Failing Loudly reproduction (mean absolute error between 0.030 and 0.053), and a novel synthetic-injection benchmark on the 137.5-million-row Trendyol collection-ranking feature table. Testing four drift types across two severity-scope regimes, we demonstrate that distributed Maximum Mean Discrepancy with Random Fourier Features on Apache Spark scales robustly. Averaged over the four drift types in the strong regime and under a calibrated threshold, it achieves a Pearson correlation of r = 0.940 with expected drift magnitude, an 80.4% true positive rate, and a 3.2% false positive rate. Conversely, the per-dimension Kolmogorov-Smirnov test failed due to statistic saturation from ID-like columns under asymmetric sampling, establishing a critical constraint for large-scale sampling design. At weak configurations (realized-flip fractions of at most 0.57%), detectors struggled to reliably discriminate, highlighting the need for future intensity-grid power analyses to distinguish fundamental sensitivity bounds from scalable threshold shifts.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Machine Learning (cs.LG); Methodology (stat.ME); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2610.08132 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T11:15:54.477Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).