Metraipolis › archive › dc_c77deb8490ca

HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation — arXiv:2610.19184

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.19184
License (as archived): CC0-1.0
Archived: 2026-10-15T20:23:34.368Z
SHA-256 of the archived text: f89f3a86e5434d73bee42ee9f27ebb8a4b159e2b9f900422a0eed23f4c2e0ed5
Stable URL of this page: https://metraipolis.com/documento/dc_c77deb8490ca

How to cite

HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation — arXiv:2610.19184 — arXiv (en), https://arxiv.org/abs/2610.19184 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c77deb8490ca (Metraipolis archive dc_c77deb8490ca, sha256 f89f3a86e5434d73bee42ee9f27ebb8a4b159e2b9f900422a0eed23f4c2e0ed5)

BibTeX

@misc{dcc77deb8490ca,
  title = {HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation — arXiv:2610.19184},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.19184},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c77deb8490ca; sha256 f89f3a86e5434d73bee42ee9f27ebb8a4b159e2b9f900422a0eed23f4c2e0ed5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.19184. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Harness optimization improves LLMs by refining reusable prompts and scaffolding, yet requires further adaptation for program generation tasks such as text-to-SQL and text-to-Python. Existing methods often rely on LLM reflectors to propose harness updates, which can fix observed failures but also disrupt previously correct behavior. Text-to-code tasks offer a natural way to mitigate this issue: we can execute generated outputs, allowing us to evaluate and verify the effects of proposed updates directly. Building on this signal, we introduce \textsc{HERO} (\textbf{H}arnessing \textbf{E}vidence for \textbf{R}efinement and \textbf{O}ptimization), which combines offline and test-time harness optimization. At the first stage, \method{} learns reusable prompt updates from execution-verified successes and failures and validates candidate revisions before adoption. At test time, we treat the generated code as an instance-specific harness and refine it using execution feedback. Across four SQL, Cypher, and Python benchmarks, \method{} achieves the best offline performance, improving pass@1 by \(7.6\) points over the initial prompt, while test-time refinement adds another \(2.0\) points and improves execution efficiency.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.19184 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T19:36:18.169Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).