Metraipolis › archive › dc_aa2185264b27

Bayesian Inference Procedures for A/B Testing: An Overview — arXiv:2608.12949

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.12949
License (as archived)
CC0-1.0
Archived
2026-10-19T04:45:19.672Z
SHA-256 of the archived text
347bfdc3eeddeb37b234560329e9828c66d4eb76149a6c7ad142750d964c9185
Stable URL of this page
https://metraipolis.com/documento/dc_aa2185264b27

How to cite

Bayesian Inference Procedures for A/B Testing: An Overview — arXiv:2608.12949 — arXiv (en), https://arxiv.org/abs/2608.12949 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_aa2185264b27 (Metraipolis archive dc_aa2185264b27, sha256 347bfdc3eeddeb37b234560329e9828c66d4eb76149a6c7ad142750d964c9185)
BibTeX
@misc{dcaa2185264b27,
  title = {Bayesian Inference Procedures for A/B Testing: An Overview — arXiv:2608.12949},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.12949},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_aa2185264b27; sha256 347bfdc3eeddeb37b234560329e9828c66d4eb76149a6c7ad142750d964c9185},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.12949. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Bayesian inference for A/B testing is a family of prior and stopping-rule configurations with fundamentally different statistical properties, but it is often discussed as a single method, and no systematic overview exists. This paper organizes common configurations into a three-tier hierarchy: 1) posterior coherence with no error control, 2) false positive rates bounded under continuous monitoring via Bayes factor stopping, and 3) false discovery rate control and calibrated shrinkage via empirical Bayes. Many commercial platforms operate at the lowest tier by default. We show that Bayes factor stopping is near-optimal for a broad class of cost functions, including most proposed in the A/B testing literature; because the same rule also controls the false positive rate, the choice between a decision-theoretic and a frequentist formulation is largely one of parameterization. Furthermore, the empirical Bayes prior is the only path to the third tier, but winner-selected corpora, pooled programs, and heterogeneous metrics can each prevent calibration regardless of corpus size. Simulations against group-sequential and always-valid frequentist baselines show that flat-prior posterior stopping exactly reproduces naive peeking, that a well-calibrated empirical Bayes prior achieves the lowest estimation error, and that expected-loss stopping minimizes regret only when shipping a null-effect variant is nearly free. Error rates, estimation accuracy, and regret are all different risks, and the appropriate method follows from the risks an experimentation program needs to control, not the other way around.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-08-14 in stat.ME).
Categorie arXiv: Methodology (stat.ME) ; Statistics Theory (math.ST).
Pagina del lavoro: https://arxiv.org/abs/2608.12949 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T04:26:22.344Z dalla pagina https://arxiv.org/catchup/stat.ME/2026-08-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).