Metraipolis › archive › dc_1b03c51138df

Robust A/B Decisions — arXiv:2609.07633

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.07633
License (as archived)
CC0-1.0
Archived
2026-10-16T14:09:59.975Z
SHA-256 of the archived text
0e6f3525c0c1bbc2a5d17efc5e95c13fd120f15db8c86bd4364c9f1134587b72
Stable URL of this page
https://metraipolis.com/documento/dc_1b03c51138df

How to cite

Robust A/B Decisions — arXiv:2609.07633 — arXiv (en), https://arxiv.org/abs/2609.07633 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1b03c51138df (Metraipolis archive dc_1b03c51138df, sha256 0e6f3525c0c1bbc2a5d17efc5e95c13fd120f15db8c86bd4364c9f1134587b72)
BibTeX
@misc{dc1b03c51138df,
  title = {Robust A/B Decisions — arXiv:2609.07633},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.07633},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1b03c51138df; sha256 0e6f3525c0c1bbc2a5d17efc5e95c13fd120f15db8c86bd4364c9f1134587b72},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.07633. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A/B tests are standard in firm decision making. In the standard pipeline, experimental data is converted to a deployment decision by applying a t-test of the difference in means (the lift) and deploying the treatment if lift is positive and statistically significant. This common workflow answers the wrong question. We argue that firms need a decision rule for economic payoffs in the future deployment environment, not a test of equality in the experimental sample. We develop an ambiguity-averse decision framework in which each arm is evaluated by its ambiguity-penalized value over distributions close to the experimental outcome distribution. The resulting rule has a simple closed form thanks to the Donsker-Varadhan representation and it requires only the outcome data from a standard A/B test plus one interpretable parameter governing trust in the experiment. Our rule is thus no more difficult to implement than a t-test. A mean-variance approximation shows how the rule penalizes variability, while a connection to utility maximization shows it to be a certainty equivalent. We are able to perform a real-world evaluation of our proposed rule in the context of digital marketing using an archive of 552 advertising experiments from an anonymous US-based online platform. The proposed rule substantially reduces regret relative to conventional hypothesis testing. The results show that economically conservative, distribution-aware deployment rules can outperform statistical-significance rules in digital experimentation.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-09 in stat.ME).
Categorie arXiv: Econometrics (econ.EM) ; Methodology (stat.ME).
Pagina del lavoro: https://arxiv.org/abs/2609.07633 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T13:59:27.929Z dalla pagina https://arxiv.org/catchup/stat.ME/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).