Metraipolis › archive › dc_b68ef4ac8277

QuoteBench: How Matched Scores Can Hide Command-Path Failures — arXiv:2608.13547

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.13547
License (as archived)
CC0-1.0
Archived
2026-10-19T04:17:45.264Z
SHA-256 of the archived text
6ad0591c081ed2523846be3807174b481d4dea0764802553a75a761f3c45e69b
Stable URL of this page
https://metraipolis.com/documento/dc_b68ef4ac8277

How to cite

QuoteBench: How Matched Scores Can Hide Command-Path Failures — arXiv:2608.13547 — arXiv (en), https://arxiv.org/abs/2608.13547 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b68ef4ac8277 (Metraipolis archive dc_b68ef4ac8277, sha256 6ad0591c081ed2523846be3807174b481d4dea0764802553a75a761f3c45e69b)
BibTeX
@misc{dcb68ef4ac8277,
  title = {QuoteBench: How Matched Scores Can Hide Command-Path Failures — arXiv:2608.13547},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.13547},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b68ef4ac8277; sha256 6ad0591c081ed2523846be3807174b481d4dea0764802553a75a761f3c45e69b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.13547. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation. Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two. Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins. Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator rather than treat a matched score as an intrinsic model property.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-14 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2608.13547 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T04:13:06.627Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).