Metraipolis › archive › dc_33e66f06691c
The Judge Knows When It Knows: Calibrated Abstention for LLM-Based A/B-Test Prediction — arXiv:2608.07517
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.07517
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T08:50:36.370Z
- SHA-256 of the archived text
07cde2324536c83292f706b23e122132505866a8e90786301d9be8a0ead6325f- Stable URL of this page
https://metraipolis.com/documento/dc_33e66f06691c
How to cite
The Judge Knows When It Knows: Calibrated Abstention for LLM-Based A/B-Test Prediction — arXiv:2608.07517 — arXiv (en), https://arxiv.org/abs/2608.07517 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_33e66f06691c (Metraipolis archive dc_33e66f06691c, sha256 07cde2324536c83292f706b23e122132505866a8e90786301d9be8a0ead6325f)
BibTeX
@misc{dc33e66f06691c,
title = {The Judge Knows When It Knows: Calibrated Abstention for LLM-Based A/B-Test Prediction — arXiv:2608.07517},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.07517},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_33e66f06691c; sha256 07cde2324536c83292f706b23e122132505866a8e90786301d9be8a0ead6325f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.07517. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Can a multimodal LLM predict which version of a web page will win a real A/B test from screenshots alone? We report the most complete answer we are aware of, from six weeks of pre-registered experiments on real conversion tests: mostly no -- and the exceptions are identifiable in advance. On 330 real A/B tests a Gemini 3 Flash judge reaches Cohen's kappa = 0.14, but on the trustworthy (statistically significant) half of the labels the evidence is inconclusive (kappa = 0.11, CI includes zero). We show that 44% of the "ground-truth" labels in a leading CRO agency's catalog come from non-significant tests, and that the judge agrees more with the unreliable labels than the reliable ones -- a shared prior between labeler and model, not prediction. Every standard improvement lever (a 2.8x more expensive frontier model, prompt redesign, stimulus fidelity, change-type priors) fails its pre-registered gate. The judge's confident calls are different: a vote-margin gate isolates a subset (49% coverage) reaching kappa = 0.31 on significant labels. We measure the mechanism directly -- judges differing in model or prompt agree with each other at kappa = 0.74-0.88 while agreeing with real outcomes at only ~0.2, so a 16-vote panel carries about 2 effective independent votes -- and we reproduce it in humans: 15 CRO experts agree with each other (inter-rater kappa = 0.53) but score at chance against real outcomes (kappa ~ 0). Consensus, human or model, is reproducible, persuasive, and not evidence. We release our pre-registrations, locked gates, negative results, statistical harness, human responses, and a claims ledger in which every number carries an evidence tier. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-11 in cs.CL). Categorie arXiv: Human-Computer Interaction (cs.HC) ; Computation and Language (cs.CL); Applications (stat.AP). Pagina del lavoro: https://arxiv.org/abs/2608.07517 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T08:49:41.957Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).