Metraipolis › archive › dc_59633ef5bb3e

Learnings and Pitfalls from AI-Assisted A/B Experimentation at Scale — arXiv:2611.04515

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.04515
License (as archived): CC0-1.0
Archived: 2026-11-04T12:28:27.775Z
SHA-256 of the archived text: ecc041783d63259f0e9aaf85afe6e7fca86ce01a24af206165a8ae6e2f09ee5b
Stable URL of this page: https://metraipolis.com/documento/dc_59633ef5bb3e

How to cite

Learnings and Pitfalls from AI-Assisted A/B Experimentation at Scale — arXiv:2611.04515 — arXiv (en), https://arxiv.org/abs/2611.04515 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_59633ef5bb3e (Metraipolis archive dc_59633ef5bb3e, sha256 ecc041783d63259f0e9aaf85afe6e7fca86ce01a24af206165a8ae6e2f09ee5b)

BibTeX

@misc{dc59633ef5bb3e,
  title = {Learnings and Pitfalls from AI-Assisted A/B Experimentation at Scale — arXiv:2611.04515},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04515},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_59633ef5bb3e; sha256 ecc041783d63259f0e9aaf85afe6e7fca86ce01a24af206165a8ae6e2f09ee5b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04515. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A/B testing underpins product decisions by providing trustworthy, reproducible quantification of proposed changes' effects. Growing large language model (LLM) capabilities have motivated efforts to automate experimentation workflows. Yet LLMs are nondeterministic and prone to hallucination, properties that conflict with the trust experimentation systems must provide. Experiments are also operationally messy: diagnosing anomalies, debugging configurations, and conducting bespoke follow-up analyses require flexible reasoning and manual effort. These tasks let agents accelerate experimentation without replacing its deterministic statistical foundations. We report lessons from building and operating such an agent. Our central finding is that this tension is resolved not by making the model deterministic, but by bounding where its nondeterminism can reach; the model selects and sequences procedures, while deterministic code computes every quantity it reports. Three consequences surprised us: withholding information from the model improved its answers, demand concentrated on mundane retrieval rather than the autonomous decisions we anticipated, and the agent's eagerness to help threatened statistical validity.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.04515 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T12:26:47.164Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).