Metraipolis › archive › dc_4b241817bd5d

Verifier strength does not buy discovery quality: a pre-registered two-domain test of LLM-based automated research, with post-hoc evidence pointing to proposer-pool… — arXiv:2611.04420

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.04420
License (as archived)
CC0-1.0
Archived
2026-11-04T21:37:59.711Z
SHA-256 of the archived text
8f75b0c2484240ea55ca4fa614a02c69a38383e124985f2fa98578567f277489
Stable URL of this page
https://metraipolis.com/documento/dc_4b241817bd5d

How to cite

Verifier strength does not buy discovery quality: a pre-registered two-domain test of LLM-based automated research, with post-hoc evidence pointing to proposer-pool… — arXiv:2611.04420 — arXiv (en), https://arxiv.org/abs/2611.04420 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4b241817bd5d (Metraipolis archive dc_4b241817bd5d, sha256 8f75b0c2484240ea55ca4fa614a02c69a38383e124985f2fa98578567f277489)
BibTeX
@misc{dc4b241817bd5d,
  title = {Verifier strength does not buy discovery quality: a pre-registered two-domain test of LLM-based automated research, with post-hoc evidence pointing to proposer-pool… — arXiv:2611.04420},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04420},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4b241817bd5d; sha256 8f75b0c2484240ea55ca4fa614a02c69a38383e124985f2fa98578567f277489},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04420. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Automated research pipelines pair an LLM proposer with a verifier on an implicit bet: discovery quality rises with verification strength. Under a pre-registered protocol we test this bet in two real-data domains -- job-shop scheduling and vehicular edge offloading on Beijing T-Drive trajectories -- across 100 matched LLM/random pairs per domain and three verifier tiers. The manipulation binds -- a 429-case mutation battery shows scores rising across tiers -- yet the paired LLM-over-random advantage does not scale with it (A, one-sided p=0.1115; D, p=0.9994), and in D shrinks under the strict tier (margin contrast -0.040, p=0.0013). The two lower gates promote identical candidates in 100/100 and 99/100 pairs, so the strict order is untestable when they coincide. Constructively, LLM proposals carry a real paired advantage over random controls, but a post-hoc union-pool analysis shows the fastest path to better discovery is widening the pool, not tightening the gate: mixing the two pools adds +1.52 holdout-value units in domain A (one-sided Wilcoxon, n=100, p=0.0002) -- evidence that pool composition, not verification strength, is the lever. Criteria, ledgers and scripts are archived on Zenodo (DOI: https://doi.org/10.5281/zenodo.23308866 ).

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2611.04420 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T21:35:49.546Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).