Metraipolis › archive › dc_18e2af95bfce

Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments — arXiv:2608.02670

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.02670
License (as archived): CC0-1.0
Archived: 2026-10-31T12:17:35.816Z
SHA-256 of the archived text: c6c253fcd5f84a3b9a3d90b072ec7914d2f7c0e54f19edd53f6bff87c226498d
Stable URL of this page: https://metraipolis.com/documento/dc_18e2af95bfce

How to cite

Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments — arXiv:2608.02670 — arXiv (en), https://arxiv.org/abs/2608.02670 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_18e2af95bfce (Metraipolis archive dc_18e2af95bfce, sha256 c6c253fcd5f84a3b9a3d90b072ec7914d2f7c0e54f19edd53f6bff87c226498d)

BibTeX

@misc{dc18e2af95bfce,
  title = {Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments — arXiv:2608.02670},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.02670},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_18e2af95bfce; sha256 c6c253fcd5f84a3b9a3d90b072ec7914d2f7c0e54f19edd53f6bff87c226498d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.02670. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Coding agents increasingly run inside organizations whose security controls (scoped credentials, restricted egress, read-only filesystems, non-root execution) constrain them like any other software. Existing benchmarks, however, evaluate agents almost exclusively in permissive sandboxes, so it is unknown how performance changes when policy is enforced. In this work, we evaluate 12 coding agents on Terminal-Bench 2.1 across nested security policy levels derived from common real-world enterprise restrictions. Hardening is never free but far from uniform: under the strictest policy, success losses reach 18.3 points and cost inflation 167.3\%, and the two axes disagree; the model that best preserves success is also the one that loses the most efficiency, so model choice is policy-dependent. Beyond aggregate scores, we characterize how agents behave when policy blocks their actions and decompose the failures hardening induces: runs grind into timeouts or wrong solutions rather than stopping early, in a mix that differs by model. To ground comparisons, we verify task solvability under the strictest policy, separating model failures from tasks the policy forecloses. We release Boundary-Bench, an open-source hardening plugin enabling policy-constrained evaluation of coding agents on Terminal-Bench and compatible benchmarks.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.AI).
Categorie arXiv: Cryptography and Security (cs.CR) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.02670 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T12:05:52.922Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).