Metraipolis › archive › dc_c8a997a9f7ba

Rewarding Noise: Incentives in Decentralized A/B Testing — arXiv:2610.19512

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.19512
License (as archived)
CC0-1.0
Archived
2026-10-15T15:02:47.896Z
SHA-256 of the archived text
ec3a0c3176938222d9e71b98b98cbdcb0c6a31a679f6cd56714931a183108b32
Stable URL of this page
https://metraipolis.com/documento/dc_c8a997a9f7ba

How to cite

Rewarding Noise: Incentives in Decentralized A/B Testing — arXiv:2610.19512 — arXiv (en), https://arxiv.org/abs/2610.19512 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c8a997a9f7ba (Metraipolis archive dc_c8a997a9f7ba, sha256 ec3a0c3176938222d9e71b98b98cbdcb0c6a31a679f6cd56714931a183108b32)
BibTeX
@misc{dcc8a997a9f7ba,
  title = {Rewarding Noise: Incentives in Decentralized A/B Testing — arXiv:2610.19512},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.19512},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c8a997a9f7ba; sha256 ec3a0c3176938222d9e71b98b98cbdcb0c6a31a679f6cd56714931a183108b32},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.19512. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Many firms that run A/B tests struggle with low statistical power, increasing the risk that launch decisions are driven by noise rather than true impact. We argue that in decentralized organizations, low power can reflect misaligned incentives: product teams choose what and how to test on a customer-facing experience to maximize their private reward, while the experience owner sets evaluation rules and launch criteria to optimize the experience. We formalize this interaction via a principal-agent model and show that evaluating tests using raw impact estimates creates a reward to sampling noise. Correctly calibrated posterior-mean evaluation removes this reward and creates a local incentive to reallocate toward fewer, more precise tests of more ambitious ideas. Using data on A/B tests run at Amazon, we document patterns consistent with these model predictions. After an Amazon experience shifted to posterior-mean-based evaluation in 2024, test volume declined while both precision and the dispersion of true impacts increased, more than doubling signal-to-noise ratios. Our results highlight that A/B testing methodologies can shape not only measurement and launch decisions, but which tests are pursued in the first place.

Tema degli esploratori: economia dei servizi IA (elenco arXiv econ.TH, nuovi lavori del giorno).
Categorie arXiv: Theoretical Economics (econ.TH).
Pagina del lavoro: https://arxiv.org/abs/2610.19512 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T14:06:53.561Z dalla pagina https://arxiv.org/list/econ.TH/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).