Metraipolis › archive › dc_dc89fcb7eeff

More Programs or More Rolls? Separating Coverage from Specialization in LLM Harnesses — arXiv:2609.35873

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.35873
License (as archived): CC0-1.0
Archived: 2026-10-07T16:22:05.924Z
SHA-256 of the archived text: 14ef394975c5fcea22358c9b9e60fdc8439478a0360ec3a59b0202ece87d44e4
Stable URL of this page: https://metraipolis.com/documento/dc_dc89fcb7eeff

How to cite

More Programs or More Rolls? Separating Coverage from Specialization in LLM Harnesses — arXiv:2609.35873 — arXiv (en), https://arxiv.org/abs/2609.35873 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dc89fcb7eeff (Metraipolis archive dc_dc89fcb7eeff, sha256 14ef394975c5fcea22358c9b9e60fdc8439478a0360ec3a59b0202ece87d44e4)

BibTeX

@misc{dcdc89fcb7eeff,
  title = {More Programs or More Rolls? Separating Coverage from Specialization in LLM Harnesses — arXiv:2609.35873},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.35873},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dc89fcb7eeff; sha256 14ef394975c5fcea22358c9b9e60fdc8439478a0360ec3a59b0202ece87d44e4},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35873. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Automated generation of LLM harnesses promises to improve inference through task specialization. Yet additional answer coverage can arise from repeated execution of the same program, making specialization difficult to identify. We introduce a controlled evaluation that separates answer coverage, repeatable task advantages, and gains from pre-execution selection. On 386 MATH-500 tasks, we compare eight generated harnesses plus a baseline with nine byte-identical baseline copies, using three executions per member. Identical programs yield 2.16 percentage points of repeat-averaged oracle headroom. Generated programs exhibit substantially more repeatable score patterns, but these chiefly reveal persistent weaknesses: losses relative to the baseline persist across all three repeats on 100 tasks, while persistent wins occur on only one task and are sensitive to answer extraction. The frozen selector gains 0.00 percentage points, and both populations reach 98.70% oracle coverage at 27 harness executions. Stable complementarity remains unresolved at three repeats. Supporting BIRD traces locate failures in mechanism implementation, activation, and output validity. Together, these findings establish why coverage and repeatability alone cannot justify claims of useful specialization. They motivate an evaluation standard for harness diversity: task advantages should persist across executions, guide usable decisions, and improve on additional fixed-program executions under matched inference budgets.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG); Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2609.35873 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T15:54:53.439Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).