Metraipolis › archive › dc_dd2c39d27cc2
No Task Fails Every Time: Why One-Shot Audits Are Structurally Blind to Agent Damage — arXiv:2608.15286
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.15286
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T18:02:40.193Z
- SHA-256 of the archived text
228ef1360ef9a595d6ee6b3353fee74d546890bb821fafb86fff875aca153b13- Stable URL of this page
https://metraipolis.com/documento/dc_dd2c39d27cc2
How to cite
No Task Fails Every Time: Why One-Shot Audits Are Structurally Blind to Agent Damage — arXiv:2608.15286 — arXiv (en), https://arxiv.org/abs/2608.15286 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dd2c39d27cc2 (Metraipolis archive dc_dd2c39d27cc2, sha256 228ef1360ef9a595d6ee6b3353fee74d546890bb821fafb86fff875aca153b13)
BibTeX
@misc{dcdd2c39d27cc2,
title = {No Task Fails Every Time: Why One-Shot Audits Are Structurally Blind to Agent Damage — arXiv:2608.15286},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.15286},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dd2c39d27cc2; sha256 228ef1360ef9a595d6ee6b3353fee74d546890bb821fafb86fff875aca153b13},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.15286. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We introduce AgentRelBench, an environment-agnostic reliability instrument that computes ground-truth, severity-priced damage from database state diffs across repeated runs, with no LLM in the measurement path, demonstrated on EnterpriseOps-Gym. Across 2,128 evaluation runs spanning nine models in six families (four development, three pre-registered held-out, plus a frontier pass on two frontier-tier models that the pre-registration designates exploratory), we find: (1) damage on irreversible actions is universal across the families we measured and stochastic within them on pinned, single-provider stacks. (2) No task damaged on every run: zero always-fail cells across 42 confirmatory held-out damage events. A single clean run misses a damage-producing (model, task) pair 0.80 of the time on the development pool (13 pairs); the held-out pool is descriptively consistent (0.575 over 5 pairs, pair-weighted) but sits below our pre-registered power floor and is reported as underpowered, not as confirmation. (3) Damage-producing task count falls with model capability, from 7 of 20 tasks for an 8B model to 1 of 20 for the most capable; capability is confounded with family and training, so this is an observed gradient, not a causal claim. The residual damage does not change in character: in the exploratory frontier pass, the most capable model's one damaging task damages at $\hat{p} = 0.16$ per run, inside the same demonstrably-stochastic band, and a single audit misses it 84% of the time. (4) One model family committed the gated irreversible change while declaring it had refused: transcript- and judge-based grading scores those runs as safe refusals, only state diffs as damage. All confirmatory findings were pre-registered with per-claim demote criteria; one demoted our own initially favored finding, which we report.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.15286 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T16:29:14.004Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).