Metraipolis › archive › dc_de7da247f143
Compile as Hypothesis: A Headroom Typology for Prompt Optimization on Real Reasoning Models — arXiv:2611.01135
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.01135
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T10:52:29.919Z
- SHA-256 of the archived text
0432cb1a262e736e4812dbc030d350edf3a6a338ac6b5f2b24b4a11bf5229cd5- Stable URL of this page
https://metraipolis.com/documento/dc_de7da247f143
How to cite
Compile as Hypothesis: A Headroom Typology for Prompt Optimization on Real Reasoning Models — arXiv:2611.01135 — arXiv (en), https://arxiv.org/abs/2611.01135 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_de7da247f143 (Metraipolis archive dc_de7da247f143, sha256 0432cb1a262e736e4812dbc030d350edf3a6a338ac6b5f2b24b4a11bf5229cd5)
BibTeX
@misc{dcde7da247f143,
title = {Compile as Hypothesis: A Headroom Typology for Prompt Optimization on Real Reasoning Models — arXiv:2611.01135},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.01135},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_de7da247f143; sha256 0432cb1a262e736e4812dbc030d350edf3a6a338ac6b5f2b24b4a11bf5229cd5},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01135. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Automated prompt compilers search for or rewrite prompts under the implicit assumption that optimization consistently improves downstream task performance. We evaluate this assumption on frontier reasoning models through rigorous paired, same-day measurements. Across four optimizer families (genome search, profile-guided mutation, GEPA, and ESPO) and eight tasks, structured search yields no statistically significant accuracy gains on strong reasoners. However, a discrete structural genome provides essential architectural stability, preventing catastrophic bimodal seed-collapse (where unconstrained search collapses toward the judge floor in 3/5 runs). Through 190-problem evaluations on knowledge-intensive benchmarks, we establish a three-dimensional headroom typology: prompt optimization cannot compensate for intrinsic knowledge deficits or physical token truncation, but successfully resolves protocol and formatting errors. On gpt-6, while free-text reflective search (GEPA) yields a +.14 gain on a task where structured search remains flat, qualitative ablation reveals that ~.035 stems from judge-rubric reconstruction while ~.105 reflects genuine instruction refinement. Our pre-registered AutoAPC-Select deployment gate matches the oracle policy in 5/7 groups and rescues seed-collapse arms with zero regret. Finally, we expose silent backend-mixing in commercial APIs during outage windows and provide automated per-call audit verification tools to ensure empirical evaluation fidelity. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2611.01135 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T10:49:38.291Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).