Metraipolis › archive › dc_d76fa7462750
When Can You Trust Offline Evaluation of Equal-Cost Top-k Allocation? A Controlled, Reproducible Benchmark and Practitioner's Guide — arXiv:2608.12489
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.12489
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T04:27:18.856Z
- SHA-256 of the archived text
8cd2d3e84e381116f263a7980600c4e8842188461e0df9242327344b9995c2ee- Stable URL of this page
https://metraipolis.com/documento/dc_d76fa7462750
How to cite
When Can You Trust Offline Evaluation of Equal-Cost Top-k Allocation? A Controlled, Reproducible Benchmark and Practitioner's Guide — arXiv:2608.12489 — arXiv (en), https://arxiv.org/abs/2608.12489 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d76fa7462750 (Metraipolis archive dc_d76fa7462750, sha256 8cd2d3e84e381116f263a7980600c4e8842188461e0df9242327344b9995c2ee)
BibTeX
@misc{dcd76fa7462750,
title = {When Can You Trust Offline Evaluation of Equal-Cost Top-k Allocation? A Controlled, Reproducible Benchmark and Practitioner's Guide — arXiv:2608.12489},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.12489},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d76fa7462750; sha256 8cd2d3e84e381116f263a7980600c4e8842188461e0df9242327344b9995c2ee},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.12489. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Organizations decide whom to treat under a budget and want to know what a targeting rule would have earned before deploying it. Off-policy evaluation promises this from logged data, but the deployable rule is a deterministic top-k policy: it removes all averaging over actions, so weak overlap hits the estimate directly. We benchmark six estimators across five datasets and two known-effect sweeps, and validate the mechanisms against a non-simulated paired reference. First, weak overlap is governed by logger-target action alignment, not by logging sharpness alone: what governs support is the logger's probability of the target's actions. Sharpening a logger built from the target's own score barely moves overlap over the tested range; action-level disagreement collapses it. Effective sample size ranks this risk across logging environments, but is weak at ranking candidates within the single log a practitioner holds, and its cut point does not transfer. Second, the optimizer's curse is not fixed by cross-fitting the outcome nuisance. When the rule is fit on the data used to evaluate it, cross-fitting the nuisance alone leaves the reuse bias in place and makes it worse. Honest policy-level splitting avoids the reuse by targeting the learning procedure's value -- a change of estimand, not a de-biasing of the full-sample policy. Third, propensity-estimation error is the largest degradation we measure: an out-of-fold estimate hurts IPS more than any other stress we apply, leaves doubly-robust estimation almost unchanged, and can invert the overlap diagnostic itself. Logging is synthesized and propensities floored at 0.02, so every failure occurs with bounded weights; the floor also reduces the two tuned hybrids to their untuned parents, leaving four practically distinct estimators, and all exact-value surfaces are synthetic or semi-synthetic. We release the benchmark; public data only. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-14 in cs.LG). Categorie arXiv: Machine Learning (cs.LG) ; Methodology (stat.ME); Machine Learning (stat.ML). Pagina del lavoro: https://arxiv.org/abs/2608.12489 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T04:20:35.810Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).