Metraipolis › archive › dc_e8d202fdb795

Beyond Accuracy: How Quantization Affects Coding Agents — arXiv:2611.02368

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.02368
License (as archived)
CC0-1.0
Archived
2026-11-03T10:28:11.506Z
SHA-256 of the archived text
d4dac9a22cd491f49a9e9a1b9614e7e208ec876fdf1e818c69ed28366847eec3
Stable URL of this page
https://metraipolis.com/documento/dc_e8d202fdb795

How to cite

Beyond Accuracy: How Quantization Affects Coding Agents — arXiv:2611.02368 — arXiv (en), https://arxiv.org/abs/2611.02368 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e8d202fdb795 (Metraipolis archive dc_e8d202fdb795, sha256 d4dac9a22cd491f49a9e9a1b9614e7e208ec876fdf1e818c69ed28366847eec3)
BibTeX
@misc{dce8d202fdb795,
  title = {Beyond Accuracy: How Quantization Affects Coding Agents — arXiv:2611.02368},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02368},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e8d202fdb795; sha256 d4dac9a22cd491f49a9e9a1b9614e7e208ec876fdf1e818c69ed28366847eec3},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02368. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Agents are increasingly used for coding tasks, but the inference cost of large language models (LLMs) limits where they can run. Quantization reduces model size, but its effects on coding agents that execute and revise programs remain underexplored. We compare BF16 with 4-bit AWQ and GPTQ for Qwen2.5-Coder-Instruct (7B, 14B, and 32B) and Devstral-Small-2507 (24B) on OJBench coding problems across three seeds. We detect no statistically significant accuracy loss for any of the coding agents. However, final accuracy can conceal substantial behavioral changes: AWQ reduces Qwen-7B one-shot solves from 17 to 0 while increasing loop-based solves from 10 to 25. All Qwen-32B successes involve tool use, whereas Devstral rarely makes a valid executor call. Failure profiles also differ: Qwen-32B produces far more compilation errors than the smaller Qwen models, rising slightly under quantization, while Devstral's failures shift significantly from runtime errors toward wrong answers and timeouts. These findings indicate that quantization robustness in coding agents should be assessed through solution trajectories, tool use, and failure modes, in addition to final accuracy.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.02368 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:19:50.171Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).