Metraipolis › archive › dc_f2ceeafe843f
Selecting LLM-Generated Hypotheses Raises Truth Beyond What an LLM Panel Expects: A Pre-registered Study on NAS-Bench-201 and HPOBench — arXiv:2611.00643
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.00643
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T10:46:04.040Z
- SHA-256 of the archived text
e9faf69579f208d74e74d54f77e2bb13de3d329c346dec2b4ac7803a8cc0427f- Stable URL of this page
https://metraipolis.com/documento/dc_f2ceeafe843f
How to cite
Selecting LLM-Generated Hypotheses Raises Truth Beyond What an LLM Panel Expects: A Pre-registered Study on NAS-Bench-201 and HPOBench — arXiv:2611.00643 — arXiv (en), https://arxiv.org/abs/2611.00643 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f2ceeafe843f (Metraipolis archive dc_f2ceeafe843f, sha256 e9faf69579f208d74e74d54f77e2bb13de3d329c346dec2b4ac7803a8cc0427f)
BibTeX
@misc{dcf2ceeafe843f,
title = {Selecting LLM-Generated Hypotheses Raises Truth Beyond What an LLM Panel Expects: A Pre-registered Study on NAS-Bench-201 and HPOBench — arXiv:2611.00643},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.00643},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f2ceeafe843f; sha256 e9faf69579f208d74e74d54f77e2bb13de3d329c346dec2b4ac7803a8cc0427f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00643. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
LLM research-ideation systems aim for hypotheses that are both true and non-obvious, yet neither property can normally be checked. We build a testbed in which both can. Generators propose conditional interaction claims about two fully tabulated benchmarks, NAS-Bench-201 and HPOBench; every claim is verified exactly against the tables, and its non-obviousness is scored against a calibrated blind prior elicited from other LLMs. Excess truth measures how much more often a run's claims hold than this panel expects. Across a series of pre-registered stages with Claude and GPT generators, our main positive result is a simple two-stage step found by a harness optimizer: draft a large pool of claims, then select a subset in a separate call. For Sonnet on NAS-Bench-201, this step raised excess truth in every pre-registered test, including against a same-pool ablation that keeps only likely-true claims, without lowering precision. The gain comes both from credit for true claims and from avoiding near-miss false ones. In post hoc checks it held under every calibration, panel and truth rule we tried, and it exceeded random and type-filter selection from the same pools. Because kept claims recur across runs, the evidence weakens when runs and claims are resampled jointly, and it is not significant against Sonnet's own prior. Two secondary findings are also pre-registered. GPT makes real use of the results of cheap probes, beating shuffled and withheld controls, though not more efficiently than asking for more claims. Optimizing the harness for an LLM judge raised judge scores while sharply cutting precision for every generator. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2611.00643 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T10:39:50.422Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).