Metraipolis › archive › dc_1aba20bd7c8b

How to Speculate about Uncertainty in Agentic Coding? A Draft-Model Gate Method — arXiv:2609.05274

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.05274
License (as archived)
CC0-1.0
Archived
2026-10-16T22:58:45.309Z
SHA-256 of the archived text
2eebec50d2de4177a16223aa121dfa9c46e521afaa137a23d13ea57ff1d17605
Stable URL of this page
https://metraipolis.com/documento/dc_1aba20bd7c8b

How to cite

How to Speculate about Uncertainty in Agentic Coding? A Draft-Model Gate Method — arXiv:2609.05274 — arXiv (en), https://arxiv.org/abs/2609.05274 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1aba20bd7c8b (Metraipolis archive dc_1aba20bd7c8b, sha256 2eebec50d2de4177a16223aa121dfa9c46e521afaa137a23d13ea57ff1d17605)
BibTeX
@misc{dc1aba20bd7c8b,
  title = {How to Speculate about Uncertainty in Agentic Coding? A Draft-Model Gate Method — arXiv:2609.05274},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.05274},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1aba20bd7c8b; sha256 2eebec50d2de4177a16223aa121dfa9c46e521afaa137a23d13ea57ff1d17605},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.05274. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM agents deployed for software engineering fail expensively: they act confidently wrong, and bad actions are recognized only after costly execution and retry. We present Speculative Uncertainty (SU), a method that recovers a predictive failure signal for a black-box agent from its output tokens alone, with no access to logits, weights, activations, or repeated sampling. Inverting speculative decoding, a small open-weight draft model scores the agent's already-generated trajectory in a single forward pass. From these speculative cross-likelihoods we extract phase-aware features by separating the reasoning and action spans, and calibrate them against a verifiable objective. SU produces a failure-likelihood score that any downstream policy, such as routing, human intervention, or extra test-time compute, can consume directly. To show the signal is actionable, we instantiate one such policy, a pre-execution veto gate, on software engineering agents Qwen3-Coder-480B and closed-source Claude 3.5 Sonnet, cutting execution error rate by 6-8 percentage points and token cost by 14-19% in deployment, transferring to out-of-distribution benchmarks without retraining, and generalizing across agent models.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-07 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.05274 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T22:31:32.214Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).