Metraipolis › archive › dc_67df284f79d8

rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment — arXiv:2610.40488

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.40488
License (as archived)
CC0-1.0
Archived
2026-11-04T21:38:21.020Z
SHA-256 of the archived text
055744feb79ee5540e3f22644c0b6ef42834dee3f7b5ccaaa8c1f0dda181a96d
Stable URL of this page
https://metraipolis.com/documento/dc_67df284f79d8

How to cite

rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment — arXiv:2610.40488 — arXiv (en), https://arxiv.org/abs/2610.40488 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_67df284f79d8 (Metraipolis archive dc_67df284f79d8, sha256 055744feb79ee5540e3f22644c0b6ef42834dee3f7b5ccaaa8c1f0dda181a96d)
BibTeX
@misc{dc67df284f79d8,
  title = {rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment — arXiv:2610.40488},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.40488},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_67df284f79d8; sha256 055744feb79ee5540e3f22644c0b6ef42834dee3f7b5ccaaa8c1f0dda181a96d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40488. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Do research agents have taste, the judgment of which experiments are worth running before their outcomes are known? We give six frontier models seven recent NLP papers. Each model, working as a research agent alone or as a team, proposes minimum viable experiments (MVEs), forecasts whether each will pass a threshold it sets and ranks them. A separate executor then attempts every MVE, including those ranked last, and three model auditors judge the results. Of 415 MVEs, 194 are judged sound, valuable and new, but fewer than half of these (73) are also judged beyond the seed paper and non obvious. The agents' first choices match the auditors' most valued experiment more often than a random pick, but are not shown to improve on choosing the experiment written first. Their forecasts beat a base rate but are not measurably more accurate than those of a fresh instance of the same model given only the paper and registrations. Teams cost more without a clear overall improvement in proposal quality. The agents produce many worthwhile experiments, but the evidence that their rankings improve the choice of which to run remains limited. The evaluation harness and a dataset of all 415 MVEs, with their forecasts and audited outcomes, are available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.40488 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T21:35:49.597Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).