Metraipolis › archive › dc_191edadb7436
OdinEval: A Reproducible Benchmark for LLM-Based Program Repair in the Odin Programming Language — arXiv:2608.18595
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.18595
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T13:54:45.579Z
- SHA-256 of the archived text
276c282cea15952f0ac4f19f66fbb7ae59de84f0f28e487ec8f53b02e86f6d20- Stable URL of this page
https://metraipolis.com/documento/dc_191edadb7436
How to cite
OdinEval: A Reproducible Benchmark for LLM-Based Program Repair in the Odin Programming Language — arXiv:2608.18595 — arXiv (en), https://arxiv.org/abs/2608.18595 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_191edadb7436 (Metraipolis archive dc_191edadb7436, sha256 276c282cea15952f0ac4f19f66fbb7ae59de84f0f28e487ec8f53b02e86f6d20)
BibTeX
@misc{dc191edadb7436,
title = {OdinEval: A Reproducible Benchmark for LLM-Based Program Repair in the Odin Programming Language — arXiv:2608.18595},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.18595},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_191edadb7436; sha256 276c282cea15952f0ac4f19f66fbb7ae59de84f0f28e487ec8f53b02e86f6d20},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.18595. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Repository-level repair benchmarks still center on a few mainstream languages, leaving systems languages such as Odin largely untested. We present OdinEval, a reproducible benchmark built from documented defects in public Odin repositories. Each instance binds an issue to base and fix commits, a gold patch, an issue-specific regression test, a historical toolchain, and execution records. Admission requires the test to fail on the base revision and pass after the gold fix. When no usable developer test exists, a black-box test is reviewed independently by three instances of the same model, executed in both historical states, and revised from recorded feedback under a versioned Test Writing Skill. We evaluate six language models on 168 filtered instances under one shared protocol. Kimi-K3 records the highest Resolved score at 66.7%, while Qwen3.8-Max has the highest Repro score at 96.4%. The release includes frozen data, source archives, containers, validators, model patches, and audit manifests. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-20 in cs.SE). Categorie arXiv: Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2608.18595 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T13:51:24.528Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-08-20?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).