Metraipolis › archive › dc_1b992f2eab99

Pdb or Not to Be: Comparing Debugging Interfaces With Pass$_{X}$@$k$ — arXiv:2611.08666

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.08666
License (as archived)
CC0-1.0
Archived
2026-11-09T22:33:39.442Z
SHA-256 of the archived text
87b584fb8a73908af458e168ef2c55c36c98b0573780e9c3e6134f56d9390354
Stable URL of this page
https://metraipolis.com/documento/dc_1b992f2eab99

How to cite

Pdb or Not to Be: Comparing Debugging Interfaces With Pass$_{X}$@$k$ — arXiv:2611.08666 — arXiv (en), https://arxiv.org/abs/2611.08666 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1b992f2eab99 (Metraipolis archive dc_1b992f2eab99, sha256 87b584fb8a73908af458e168ef2c55c36c98b0573780e9c3e6134f56d9390354)
BibTeX
@misc{dc1b992f2eab99,
  title = {Pdb or Not to Be: Comparing Debugging Interfaces With Pass$_X$@$k$ — arXiv:2611.08666},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.08666},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1b992f2eab99; sha256 87b584fb8a73908af458e168ef2c55c36c98b0573780e9c3e6134f56d9390354},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.08666. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Interactive debuggers are indispensable to human software development, yet coding agents overwhelmingly rely on edit-execute-observe loops, using common bash tools, test scripts, and print statements to debug via repeated shell commands. Stepping through a debugger provides live access to runtime state, which may enable different problem-solving capabilities. However, prior work typically introduces the debugger as an additional tool or through a new subagent, making its individual contribution difficult to isolate from the associated scaffold changes. To address this, we compare a debugger-only agent against a bash-only agent on all 500 SWE-bench Verified instances, holding the rest of the scaffold fixed. We evaluate both architectures on two models, GPT-OSS-120B and Nemotron-3-Ultra-550B, with three independent runs per architecture $\times$ model configuration. Crucially, to distinguish interface specialization from expanded coverage under repeated sampling, we introduce pass$_X$@$k$, a cross-agent pass@$k$ that provides an unbiased estimator of the probability that at least one of $k$ agents is successful. Although the bash and debugger agents solve partially disjoint subsets of the benchmark in any single pair of runs, our pass$_X$@$2$ estimator reveals that this is largely due to run-to-run variance rather than tool complementarity. Across both models, we find that two bash rollouts outperform the combination of one bash and one debugger rollout. Our findings highlight the importance of accounting for stochastic variance in evaluating agent and tool diversity, and we hope that pass$_X$@$k$ will serve as a useful framework for the community.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2611.08666 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-09T22:30:47.251Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).