Metraipolis › archive › dc_bbdc0be730ba

Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks — arXiv:2608.30047

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.30047
License (as archived): CC0-1.0
Archived: 2026-10-17T07:44:29.949Z
SHA-256 of the archived text: 1a69f619c9b345b078d81930bfe0c1c9c66f5c31074e2bf19bec9f450045b34b
Stable URL of this page: https://metraipolis.com/documento/dc_bbdc0be730ba

How to cite

Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks — arXiv:2608.30047 — arXiv (en), https://arxiv.org/abs/2608.30047 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bbdc0be730ba (Metraipolis archive dc_bbdc0be730ba, sha256 1a69f619c9b345b078d81930bfe0c1c9c66f5c31074e2bf19bec9f450045b34b)

BibTeX

@misc{dcbbdc0be730ba,
  title = {Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks — arXiv:2608.30047},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.30047},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bbdc0be730ba; sha256 1a69f619c9b345b078d81930bfe0c1c9c66f5c31074e2bf19bec9f450045b34b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.30047. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Recent AI systems promise autonomous scientific discovery, claiming to discover algorithms and produce research papers, yet understanding whether they exhibit creativity, the capacity to produce solutions that are both novel and useful, remains an open question. We present a framework for evaluating multi-turn LLM research agents' creativity using ML engineering tasks as a testbed, through three dimensions: P-Creativity (psychological novelty: novel relative to the agent's own prior solutions within a run), H-Creativity (historical novelty: novel relative to the corpus of human solutions), and Usefulness (task performance). Evaluating two agent frameworks, AIDE and AIRA-Dojo, on 10 Kaggle-style machine learning tasks from MLE-Bench, we develop an LLM-as-a-Judge pipeline and verify its strong correlation with human creativity judgments, providing a reliable automated metric for P-Creativity evaluation at scale. Applying this pipeline to agent trajectories, we find: (1) all agents exhibit declining P-Creativity as they transition from exploration to exploitation; (2) LLMs exhibit greater H-Creativity than medal-winning humans, yet achieve lower performance. Our findings reveal that current agents can explore novel regions of the solution space but lack the capacity to convert this novelty into improved task performance.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.30047 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T07:28:26.580Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).