Metraipolis › archive › dc_284fe74a96ff

NAQD Env: A benchmark for selective withdrawal in language agents — arXiv:2609.38460

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.38460
License (as archived)
CC0-1.0
Archived
2026-10-07T12:41:53.480Z
SHA-256 of the archived text
13a8f4d100d5b097a5219778e22d033e699b9bc68f0ef27671e3aa52c6f87b79
Stable URL of this page
https://metraipolis.com/documento/dc_284fe74a96ff

How to cite

NAQD Env: A benchmark for selective withdrawal in language agents — arXiv:2609.38460 — arXiv (en), https://arxiv.org/abs/2609.38460 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_284fe74a96ff (Metraipolis archive dc_284fe74a96ff, sha256 13a8f4d100d5b097a5219778e22d033e699b9bc68f0ef27671e3aa52c6f87b79)
BibTeX
@misc{dc284fe74a96ff,
  title = {NAQD Env: A benchmark for selective withdrawal in language agents — arXiv:2609.38460},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.38460},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_284fe74a96ff; sha256 13a8f4d100d5b097a5219778e22d033e699b9bc68f0ef27671e3aa52c6f87b79},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38460. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Language agents must revise planned actions when evidence changes, permission is revoked, or a stop instruction arrives. A useful response is selective: suspend affected actions, preserve unaffected work, and resume only after sufficient repair. We introduce NAQD-Env, a synthetic environment that evaluates these decisions against a deterministic reference policy over explicit evidence, authorization, and constraint dependencies. Eleven dependency families support evaluation on development structures, held-out families, and held-out combinations of structures. Metrics distinguish attempted violations from violations permitted by a simulated execution gate and jointly report policy agreement, task value, withdrawal, resumption, and event reporting. We evaluate three open-weight instruction-tuned models from two families under three prompt conditions on 350 frozen scenarios, yielding 3,150 model-prompt episodes before gate replay. Across the reported conditions, withdrawal recall is at most 0.06, no valid resumption is observed at eligible opportunities, and only one episode matches the complete reference policy. Under the NAQD prompt, Qwen2.5-7B has fewer unsafe-attempt episodes than Qwen2.5-3B and Llama-3.1-8B, but also completes less useful work and preserves unaffected actions less accurately. Exploratory supervised fine-tuning probes increase Qwen2.5-3B decision accuracy from 0.45-0.54 to 0.83-0.92; separate diagnostics reveal inappropriate withdrawal after curriculum omissions and a loss of event reporting. These results motivate evaluating selective withdrawal as a distinct component of agent reliability. The setting measures policy application with trusted structured inputs and does not establish real-world containment or source-verification ability.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.38460 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:14:13.636Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).