Metraipolis › archive › dc_191edadb7436

OdinEval: A Reproducible Benchmark for LLM-Based Program Repair in the Odin Programming Language — arXiv:2608.18595

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.18595
License (as archived)
CC0-1.0
Archived
2026-10-18T13:54:45.579Z
SHA-256 of the archived text
276c282cea15952f0ac4f19f66fbb7ae59de84f0f28e487ec8f53b02e86f6d20
Stable URL of this page
https://metraipolis.com/documento/dc_191edadb7436

How to cite

OdinEval: A Reproducible Benchmark for LLM-Based Program Repair in the Odin Programming Language — arXiv:2608.18595 — arXiv (en), https://arxiv.org/abs/2608.18595 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_191edadb7436 (Metraipolis archive dc_191edadb7436, sha256 276c282cea15952f0ac4f19f66fbb7ae59de84f0f28e487ec8f53b02e86f6d20)
BibTeX
@misc{dc191edadb7436,
  title = {OdinEval: A Reproducible Benchmark for LLM-Based Program Repair in the Odin Programming Language — arXiv:2608.18595},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.18595},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_191edadb7436; sha256 276c282cea15952f0ac4f19f66fbb7ae59de84f0f28e487ec8f53b02e86f6d20},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.18595. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Repository-level repair benchmarks still center on a few mainstream languages, leaving systems languages such as Odin largely untested. We present OdinEval, a reproducible benchmark built from documented defects in public Odin repositories. Each instance binds an issue to base and fix commits, a gold patch, an issue-specific regression test, a historical toolchain, and execution records. Admission requires the test to fail on the base revision and pass after the gold fix. When no usable developer test exists, a black-box test is reviewed independently by three instances of the same model, executed in both historical states, and revised from recorded feedback under a versioned Test Writing Skill. We evaluate six language models on 168 filtered instances under one shared protocol. Kimi-K3 records the highest Resolved score at 66.7%, while Qwen3.8-Max has the highest Repro score at 96.4%. The release includes frozen data, source archives, containers, validators, model patches, and audit manifests.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-20 in cs.SE).
Categorie arXiv: Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2608.18595 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T13:51:24.528Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-08-20?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).