Metraipolis › archive › dc_2ac5138bbe82

HPCAgent-Bench: A Framework for Sound Benchmarking of Code-Optimizing Agents, Harnesses, and Skills — arXiv:2610.20673

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.20673
License (as archived)
CC0-1.0
Archived
2026-10-16T10:25:25.526Z
SHA-256 of the archived text
2d4b3a404918e5dcc1cd90155ba4b1692801f05ac37bb758c8612937478daafb
Stable URL of this page
https://metraipolis.com/documento/dc_2ac5138bbe82

How to cite

HPCAgent-Bench: A Framework for Sound Benchmarking of Code-Optimizing Agents, Harnesses, and Skills — arXiv:2610.20673 — arXiv (en), https://arxiv.org/abs/2610.20673 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2ac5138bbe82 (Metraipolis archive dc_2ac5138bbe82, sha256 2d4b3a404918e5dcc1cd90155ba4b1692801f05ac37bb758c8612937478daafb)
BibTeX
@misc{dc2ac5138bbe82,
  title = {HPCAgent-Bench: A Framework for Sound Benchmarking of Code-Optimizing Agents, Harnesses, and Skills — arXiv:2610.20673},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.20673},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2ac5138bbe82; sha256 2d4b3a404918e5dcc1cd90155ba4b1692801f05ac37bb758c8612937478daafb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.20673. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Coding agents are increasingly used to optimize code, but existing benchmarks fix one accelerator, programming model or harness, so they cannot isolate what each component contributes or show whether added skills and tools repay their cost. We present HPCAgent-Bench with 689 kernels in three tracks: loop-level optimization challenges, kernels extracted from HPC codes, and machine-learning models. The framework specifies each kernel once in NumPy, derives C and Fortran references, accepts submissions through a common C ABI or Python interface, and runs them on CPUs or GPUs, on one or multiple nodes. An external judge checks correctness on hidden fuzzed inputs and times each submission, closing paths agents use to exploit the testing system. We introduce two metrics for reporting results: \textit{intervention efficacy} measures an intervention, such as adding a tool, through paired runs on speedup, success rate and cost, and \textit{weighted token cost} reports input, cached-input and output tokens under explicit weights. Agents reach up to $4.2\times$ speedup over automatic parallelizers, and their optimizations transfer to another machine: $93$--$99\%$ of answers without device-specific code remain correct, and per-kernel speedups keep a rank correlation of $r_s\ge0.84$. Across 40 matched setups, interventions significantly change speedup only twice but token cost by up to $3.0\times$. Because token weighting decides whether a saving is significant, we state the weights and release every token count as a standard for cost reporting. We provide the first comprehensive, broad and hardened benchmark for the sound evaluation of agentic optimization, with transparent cost metrics.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.SE, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2610.20673 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T08:48:17.687Z dalla pagina https://arxiv.org/list/cs.SE/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).